41 research outputs found
Combining Model-Based and Model-Free Updates for Trajectory-Centric Reinforcement Learning
Reinforcement learning (RL) algorithms for real-world robotic applications
need a data-efficient learning process and the ability to handle complex,
unknown dynamical systems. These requirements are handled well by model-based
and model-free RL approaches, respectively. In this work, we aim to combine the
advantages of these two types of methods in a principled manner. By focusing on
time-varying linear-Gaussian policies, we enable a model-based algorithm based
on the linear quadratic regulator (LQR) that can be integrated into the
model-free framework of path integral policy improvement (PI2). We can further
combine our method with guided policy search (GPS) to train arbitrary
parameterized policies such as deep neural networks. Our simulation and
real-world experiments demonstrate that this method can solve challenging
manipulation tasks with comparable or better performance than model-free
methods while maintaining the sample efficiency of model-based methods. A video
presenting our results is available at
https://sites.google.com/site/icml17pilqrComment: Paper accepted to the International Conference on Machine Learning
(ICML) 201
MBMF: Model-Based Priors for Model-Free Reinforcement Learning
Reinforcement Learning is divided in two main paradigms: model-free and
model-based. Each of these two paradigms has strengths and limitations, and has
been successfully applied to real world domains that are appropriate to its
corresponding strengths. In this paper, we present a new approach aimed at
bridging the gap between these two paradigms. We aim to take the best of the
two paradigms and combine them in an approach that is at the same time
data-efficient and cost-savvy. We do so by learning a probabilistic dynamics
model and leveraging it as a prior for the intertwined model-free optimization.
As a result, our approach can exploit the generality and structure of the
dynamics model, but is also capable of ignoring its inevitable inaccuracies, by
directly incorporating the evidence provided by the direct observation of the
cost. Preliminary results demonstrate that our approach outperforms purely
model-based and model-free approaches, as well as the approach of simply
switching from a model-based to a model-free setting.Comment: After we submitted the paper for consideration in CoRL 2017 we found
a paper published in the recent past with a similar method (see related work
for a discussion). Considering the similarities between the two papers, we
have decided to retract our paper from CoRL 201