Model-based Reinforcement Learning with Parametrized Physical Models and Optimism-Driven Exploration

In this paper, we present a robotic model-based reinforcement learning method that combines ideas from model identification and model predictive control. We use a feature-based representation of the dynamics that allows the dynamics model to be fitted with a simple least squares procedure, and the features are identified from a high-level specification of the robot's morphology, consisting of the number and connectivity structure of its links. Model predictive control is then used to choose the actions under an optimistic model of the dynamics, which produces an efficient and goal-directed exploration strategy. We present real time experimental results on standard benchmark problems involving the pendulum, cartpole, and double pendulum systems. Experiments indicate that our method is able to learn a range of benchmark tasks substantially faster than the previous best methods. To evaluate our approach on a realistic robotic control task, we also demonstrate real time control of a simulated 7 degree of freedom arm.Comment: 8 page

Перенос подхода машинного обучения с подкреплением с симуляционной модели на мобильного робота

Обучение с подкреплением, как один из способов машинного обучения, показывает многообещающие результаты при его интеграции в различные робототехнические алгоритмы. Но для того, чтобы добиться оптимального поведения робота, требуется значительное количество времени и ресурсов. Используя виртуальные эксперименты, возможно значительно ускорить и улучшить производительность алгоритмов. Мы внедрили подход обучения с подкреплением для алгоритма локализации и картографирования, применяемого на мобильном роботе. Алгоритм был обучен в симуляционной среде Gazebo и перенесен на реального робота. В публикации показана целесообразность использования симуляции для обучения алгоритмов, применяемых мобильными роботами.201-21

Deterministic Value-Policy Gradients

Reinforcement learning algorithms such as the deep deterministic policy gradient algorithm (DDPG) has been widely used in continuous control tasks. However, the model-free DDPG algorithm suffers from high sample complexity. In this paper we consider the deterministic value gradients to improve the sample efficiency of deep reinforcement learning algorithms. Previous works consider deterministic value gradients with the finite horizon, but it is too myopic compared with infinite horizon. We firstly give a theoretical guarantee of the existence of the value gradients in this infinite setting. Based on this theoretical guarantee, we propose a class of the deterministic value gradient algorithm (DVG) with infinite horizon, and different rollout steps of the analytical gradients by the learned model trade off between the variance of the value gradients and the model bias. Furthermore, to better combine the model-based deterministic value gradient estimators with the model-free deterministic policy gradient estimator, we propose the deterministic value-policy gradient (DVPG) algorithm. We finally conduct extensive experiments comparing DVPG with state-of-the-art methods on several standard continuous control benchmarks. Results demonstrate that DVPG substantially outperforms other baselines