6 research outputs found
How to Fine-tune the Model: Unified Model Shift and Model Bias Policy Optimization
Designing and deriving effective model-based reinforcement learning (MBRL)
algorithms with a performance improvement guarantee is challenging, mainly
attributed to the high coupling between model learning and policy optimization.
Many prior methods that rely on return discrepancy to guide model learning
ignore the impacts of model shift, which can lead to performance deterioration
due to excessive model updates. Other methods use performance difference bound
to explicitly consider model shift. However, these methods rely on a fixed
threshold to constrain model shift, resulting in a heavy dependence on the
threshold and a lack of adaptability during the training process. In this
paper, we theoretically derive an optimization objective that can unify model
shift and model bias and then formulate a fine-tuning process. This process
adaptively adjusts the model updates to get a performance improvement guarantee
while avoiding model overfitting. Based on these, we develop a straightforward
algorithm USB-PO (Unified model Shift and model Bias Policy Optimization).
Empirical results show that USB-PO achieves state-of-the-art performance on
several challenging benchmark tasks