Improving RL Power for On-Line Evolution of Gaits in Modular Robots

This paper addresses the problem of on-line gait learning in modular robots whose shape is not known in advance. The best algorithm for this problem known to us is a reinforcement learning method, called RL PoWER. In this study we revisit the original RL PoWER algorithm and observe that in essence it is a specific evolutionary algorithm. Based on this insight we propose two modifications of the main search operators and compare the quality of the evolved gaits when either or both of these modified operators are employed. The results show that using 2-parent crossover as well as mutation with self- adaptive step-sizes can significantly improve the performance of the original algorithm

Acquiring moving skills in robots with evolvable morphologies: Recent results and outlook

© 2017 ACM. We construct and investigate a strongly embodied evolutionary system, where not only the controllers but also the morphologies undergo evolution in an on-line fashion. In these studies, we have been using various types of robot morphologies and controller architectures in combination with several learning algorithms, e.g. evolutionary algorithms, reinforcement learning, simulated annealing, and HyperNEAT. This hands-on experience provides insights and helps us elaborate on interesting research directions for future development

Deep Reinforcement Learning for Tensegrity Robot Locomotion

Tensegrity robots, composed of rigid rods connected by elastic cables, have a number of unique properties that make them appealing for use as planetary exploration rovers. However, control of tensegrity robots remains a difficult problem due to their unusual structures and complex dynamics. In this work, we show how locomotion gaits can be learned automatically using a novel extension of mirror descent guided policy search (MDGPS) applied to periodic locomotion movements, and we demonstrate the effectiveness of our approach on tensegrity robot locomotion. We evaluate our method with real-world and simulated experiments on the SUPERball tensegrity robot, showing that the learned policies generalize to changes in system parameters, unreliable sensor measurements, and variation in environmental conditions, including varied terrains and a range of different gravities. Our experiments demonstrate that our method not only learns fast, power-efficient feedback policies for rolling gaits, but that these policies can succeed with only the limited onboard sensing provided by SUPERball's accelerometers. We compare the learned feedback policies to learned open-loop policies and hand-engineered controllers, and demonstrate that the learned policy enables the first continuous, reliable locomotion gait for the real SUPERball robot. Our code and other supplementary materials are available from http://rll.berkeley.edu/drl_tensegrityComment: International Conference on Robotics and Automation (ICRA), 2017. Project website link is http://rll.berkeley.edu/drl_tensegrit

Learning directed locomotion in modular robots with evolvable morphologies

The vision behind this paper looks ahead to evolutionary robot systems where morphologies and controllers are evolved together and ‘newborn’ robots undergo a learning process to optimize their inherited brain for the inherited body. The specific problem we address is learning controllers for the task of directed locomotion in evolvable modular robots. To this end, we present a test suite of robots with different shapes and sizes and compare two learning algorithms, Bayesian optimization and HyperNEAT. The experiments in simulation show that both methods obtain good controllers, but Bayesian optimization is more effective and sample efficient. We validate the best learned controllers by constructing three robots from the test suite in the real world and observe their fitness and actual trajectories. The obtained results indicate a reality gap, but overall the trajectories are adequate and follow the target directions successfully

Using Parameterized Black-Box Priors to Scale Up Model-Based Policy Search for Robotics

The most data-efficient algorithms for reinforcement learning in robotics are model-based policy search algorithms, which alternate between learning a dynamical model of the robot and optimizing a policy to maximize the expected return given the model and its uncertainties. Among the few proposed approaches, the recently introduced Black-DROPS algorithm exploits a black-box optimization algorithm to achieve both high data-efficiency and good computation times when several cores are used; nevertheless, like all model-based policy search approaches, Black-DROPS does not scale to high dimensional state/action spaces. In this paper, we introduce a new model learning procedure in Black-DROPS that leverages parameterized black-box priors to (1) scale up to high-dimensional systems, and (2) be robust to large inaccuracies of the prior information. We demonstrate the effectiveness of our approach with the "pendubot" swing-up task in simulation and with a physical hexapod robot (48D state space, 18D action space) that has to walk forward as fast as possible. The results show that our new algorithm is more data-efficient than previous model-based policy search algorithms (with and without priors) and that it can allow a physical 6-legged robot to learn new gaits in only 16 to 30 seconds of interaction time.Comment: Accepted at ICRA 2018; 8 pages, 4 figures, 2 algorithms, 1 table; Video at https://youtu.be/HFkZkhGGzTo ; Spotlight ICRA presentation at https://youtu.be/_MZYDhfWeL

Challenges in the Locomotion of Self-Reconfigurable Modular Robots

Self-Reconfigurable Modular Robots (SRMRs) are assemblies of autonomous robotic units, referred to as modules, joined together using active connection mechanisms. By changing the connectivity of these modules, SRMRs are able to deliberately change their own shape in order to adapt to new environmental circumstances. One of the main motivations for the development of SRMRs is that conventional robots are limited in their capabilities by their morphology. The promise of the field of self-reconfigurable modular robotics is to design robots that are robust, self-healing, versatile, multi-purpose, and inexpensive. Despite significant efforts by numerous research groups worldwide, the potential advantages of SRMRs have yet to be realized. A high number of degrees of freedom and connectors make SRMRs more versatile, but also more complex both in terms of mechanical design and control algorithms. Scalability issues affect these robots in terms of hardware, low-level control, and high-level planning. In this thesis we identify and target three major challenges: (i) Hardware design; (ii) Planning and control; and, (iii) Application challenges. To tackle the hardware challenges we redesigned and manufactured the Self-Reconfigurable Modular Robot Roombots to meet desired requirements and characteristics. We explored in detail and improved two major mechanical components of an SRMR: the actuation and the connection mechanisms. We also analyzed the use of compliant extensions to increase locomotion performance in terms of locomotion speed and power consumption. We contributed to the control challenge by developing new methods that allow an arbitrary SRMR structure to learn to locomote in an efficient way. We defined a novel bio-inspired locomotion-learning framework that allows the quick and reliable optimization of new gaits after a morphological change due to self-reconfiguration or human construction. In order to find new suitable application scenarios for SRMRs we envision the use of Roombots modules to create Self-Reconfigurable Robotic Furniture. As a first step towards this vision, we explored the use and control of Plug-n-Play Robotic Elements that can augment existing pieces of furniture and create new functionalities in a household to improve quality of life

Using evolutionary artificial neural networks to design hierarchical animat nervous systems.

The research presented in this thesis examines the area of control systems for robots or animats (animal-like robots). Existing systems have problems in that they require a great deal of manual design or are limited to performing jobs of a single type. For these reasons, a better solution is desired. The system studied here is an Artificial Nervous System (ANS) which is biologically inspired; it is arranged as a hierarchy of layers containing modules operating in parallel. The ANS model has been developed to be flexible, scalable, extensible and modular. The ANS can be implemented using any suitable technology, for many different environments. The implementation focused on the two lowest layers (the reflex and action layers) of the ANS, which are concerned with control and rhythmic movement. Both layers were realised as Artificial Neural Networks (ANN) which were created using Evolutionary Algorithms (EAs). The task of the reflex layer was to control the position of an actuator (such as linear actuators or D.C. motors). The action layer performed the task of Central Pattern Generators (CPG), which produce rhythmic patterns of activity. In particular, different biped and quadruped gait patterns were created. An original neural model was specifically developed for assisting in the creation of these time-based patterns. It is shown in the thesis that Artificial Reflexes and CPGs can be configured successfully using this technique. The Artificial Reflexes were better at generalising across different actuators, without changes, than traditional controllers. Gaits such as pace, trot, gallop and pronk were successfully created using the CPGs. Experiments were conducted to determine whether modularity in the networks had an impact. It has been demonstrated that the degree of modularization in the network influences its evolvability, with more modular networks evolving more efficiently

Legged Robots for Object Manipulation: A Review

Legged robots can have a unique role in manipulating objects in dynamic, human-centric, or otherwise inaccessible environments. Although most legged robotics research to date typically focuses on traversing these challenging environments, many legged platform demonstrations have also included "moving an object" as a way of doing tangible work. Legged robots can be designed to manipulate a particular type of object (e.g., a cardboard box, a soccer ball, or a larger piece of furniture), by themselves or collaboratively. The objective of this review is to collect and learn from these examples, to both organize the work done so far in the community and highlight interesting open avenues for future work. This review categorizes existing works into four main manipulation methods: object interactions without grasping, manipulation with walking legs, dedicated non-locomotive arms, and legged teams. Each method has different design and autonomy features, which are illustrated by available examples in the literature. Based on a few simplifying assumptions, we further provide quantitative comparisons for the range of possible relative sizes of the manipulated object with respect to the robot. Taken together, these examples suggest new directions for research in legged robot manipulation, such as multifunctional limbs, terrain modeling, or learning-based control, to support a number of new deployments in challenging indoor/outdoor scenarios in warehouses/construction sites, preserved natural areas, and especially for home robotics.Comment: Preprint of the paper submitted to Frontiers in Mechanical Engineerin