Representation Abstractions as Incentives for Reinforcement Learning Agents: A Robotic Grasping Case Study

Choosing an appropriate representation of the environment for the underlying decision-making process of the RL agent is not always straightforward. The state representation should be inclusive enough to allow the agent to informatively decide on its actions and compact enough to increase sample efficiency for policy training. Given this outlook, this work examines the effect of various state representations in incentivizing the agent to solve a specific robotic task: antipodal and planar object grasping. A continuum of state representation abstractions is defined, starting from a model-based approach with complete system knowledge, through hand-crafted numerical, to image-based representations with decreasing level of induced task-specific knowledge. We examine the effects of each representation in the ability of the agent to solve the task in simulation and the transferability of the learned policy to the real robot. The results show that RL agents using numerical states can perform on par with non-learning baselines. Furthermore, we find that agents using image-based representations from pre-trained environment embedding vectors perform better than end-to-end trained agents, and hypothesize that task-specific knowledge is necessary for achieving convergence and high success rates in robot control.Comment: 8 pages, 6 figure

Analysis of Randomization Effects on Sim2Real Transfer in Reinforcement Learning for Robotic Manipulation Tasks

Randomization is currently a widely used approach in Sim2Real transfer for data-driven learning algorithms in robotics. Still, most Sim2Real studies report results for a specific randomization technique and often on a highly customized robotic system, making it difficult to evaluate different randomization approaches systematically. To address this problem, we define an easy-to-reproduce experimental setup for a robotic reach-and-balance manipulator task, which can serve as a benchmark for comparison. We compare four randomization strategies with three randomized parameters both in simulation and on a real robot. Our results show that more randomization helps in Sim2Real transfer, yet it can also harm the ability of the algorithm to find a good policy in simulation. Fully randomized simulations and fine-tuning show differentiated results and translate better to the real robot than the other approaches tested.Comment: Accepted to IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 202

A Model Predictive Controller for Minimum Time Cornering

This paper presents a model predictive control approach to drive the vehicle up to its tires adhesion limits. The main focus is optimality of the closed-loop control to follow a certain track in minimum time. A Linear Time Varying Model Predictive Controller (LTV-MPC) is developed to be able to adapt with high non-linearities of the model, but also having low computational complexity in order to work in real-time. In order to have the best reference trajectory and apply it to the LTV-MPC, the problem is first solved in a nonlinear optimal control framework. This solution represents a benchmark as well, to which the LTV-MPC results are compared. The proposed controller is tested in simulation showing promising results