Self-Referencing agents for unsupervised reinforcement learning
Published in Neural Networks , 2025
Abstract: Current unsupervised reinforcement learning methods often overlook reward nonstationarity during pre-training and the forgetting of exploratory behavior during fine-tuning. Our study introduces Self-Reference (SR), a novel add-on module designed to address both issues. SR stabilizes intrinsic rewards through historical referencing in pre-training, mitigating nonstationarity. During fine-tuning, it preserves exploratory behaviors, retaining valuable skills. Our approach significantly boosts the performance and sample efficiency of existing URL model-free methods on the Unsupervised Reinforcement Learning Benchmark, improving IQM by up to 17% and reducing the Optimality Gap by 31%. This highlights the general applicability and compatibility of our add-on module with existing methods.
Recommended citation: Andrew Zhao, Erle Zhu, Rui Lu, Matthieu Lin, Yong-Jin Liu, Gao Huang, Self-Referencing Agents for Unsupervised Reinforcement Learning, Neural Networks, Volume 188, 2025, 107448, ISSN 0893-6080,https://doi.org/10.1016/j.neunet.2025.107448.