Smooth Exploration for Robotic Reinforcement Learning

Reinforcement learning (RL) enables robots to learn skills from interactions with the real world. In practice, the unstructured step-based exploration used in Deep RL -- often very successful in simulation -- leads to jerky motion patterns on real robots. Consequences of the resulting shaky behavior are poor exploration, or even damage to the robot. We address these issues by adapting state-dependent exploration (SDE) to current Deep RL algorithms. To enable this adaptation, we propose two extensions to the original SDE, using more general features and re-sampling the noise periodically, which leads to a new exploration method generalized state-dependent exploration (gSDE). We evaluate gSDE both in simulation, on PyBullet continuous control tasks, and directly on three different real robots: a tendon-driven elastic robot, a quadruped and an RC car. The noise sampling interval of gSDE permits to have a compromise between performance and smoothness, which allows training directly on the real robots without loss of performance. The code is available at https://github.com/DLR-RM/stable-baselines3.Comment: Code: https://github.com/DLR-RM/stable-baselines3/ Training scripts: https://github.com/DLR-RM/rl-baselines3-zoo

Position Control of an Underactuated Continuum Mechanism using a Reduced Nonlinear Model

The paper treats the model-based control of a nonlinear system composed of a continuum mechanism currently used as a neck of a humanoid robot. The structural flexibility of the continuum mechanism allows for compliant Cartesian motion of the link side, i.e. the head of the humanoid. The position control approach proposed in this work uses a reduced nonlinear model of the system. This reduced model consists of the rigid body dynamics and an experimentally obtained approximation of the nonlinear Cartesian spring characteristics of the continuum with multivariate polynomials. Two model-based controllers are set up, namely a partial feedback linearization and a passivity-based approach. As the system is underactuated, the six generalized coordinates are split up into four task and two remaining coordinates. The two proposed controllers are designed to regulate the task coordinates to desired positions. The proposed control approaches are tested and compared in simulation and experiments on the real system