Learning and Transfer of Modulated Locomotor Controllers

We study a novel architecture and training procedure for locomotion tasks. A high-frequency, low-level "spinal" network with access to proprioceptive sensors learns sensorimotor primitives by training on simple tasks. This pre-trained module is fixed and connected to a low-frequency, high-level "cortical" network, with access to all sensors, which drives behavior by modulating the inputs to the spinal network. Where a monolithic end-to-end architecture fails completely, learning with a pre-trained spinal module succeeds at multiple high-level tasks, and enables the effective exploration required to learn from sparse rewards. We test our proposed architecture on three simulated bodies: a 16-dimensional swimming snake, a 20-dimensional quadruped, and a 54-dimensional humanoid. Our results are illustrated in the accompanying video at https://youtu.be/sboPYvhpraQComment: Supplemental video available at https://youtu.be/sboPYvhpra

Fast Damage Recovery in Robotics with the T-Resilience Algorithm

Damage recovery is critical for autonomous robots that need to operate for a long time without assistance. Most current methods are complex and costly because they require anticipating each potential damage in order to have a contingency plan ready. As an alternative, we introduce the T-resilience algorithm, a new algorithm that allows robots to quickly and autonomously discover compensatory behaviors in unanticipated situations. This algorithm equips the robot with a self-model and discovers new behaviors by learning to avoid those that perform differently in the self-model and in reality. Our algorithm thus does not identify the damaged parts but it implicitly searches for efficient behaviors that do not use them. We evaluate the T-Resilience algorithm on a hexapod robot that needs to adapt to leg removal, broken legs and motor failures; we compare it to stochastic local search, policy gradient and the self-modeling algorithm proposed by Bongard et al. The behavior of the robot is assessed on-board thanks to a RGB-D sensor and a SLAM algorithm. Using only 25 tests on the robot and an overall running time of 20 minutes, T-Resilience consistently leads to substantially better results than the other approaches

Shape-based compliance control for snake robots

I serpenti robot sono una classe di meccanismi iper-ridondanti che appartiene alla robotica modulare. Grazie alla loro forma snella ed allungata e all'alto grado di ridondanza possono muoversi in ambienti complessi con elevata agilità. L'abilità di spostarsi, manipolare e adattarsi efficientemente ad una grande varietà di terreni li rende ideali per diverse applicazioni, come ad esempio attività di ricerca e soccorso, ispezione o ricognizione. I robot serpenti si muovono nello spazio modificando la propria forma, senza necessità di ulteriori dispositivi quali ruote od arti. Tali deformazioni, che consistono in movimenti ondulatori ciclici che generano uno spostamento dell'intero meccanismo, vengono definiti andature. La maggior parte di esse sono ispirate al mondo naturale, come lo strisciamento, il movimento laterale o il movimento a concertina, mentre altre sono create per applicazioni specifiche, come il rotolamento o l'arrampicamento. Un serpente robot con molti gradi di libertà deve essere capace di coordinare i propri giunti e reagire ad ostacoli in tempo reale per riuscire a muoversi efficacemente in ambienti complessi o non strutturati. Inoltre, aumentare la semplicità e ridurre il numero di controllori necessari alla locomozione alleggerise una struttura di controllo che potrebbe richiedere complessità per ulteriori attività specifiche. L'obiettivo di questa tesi è ottenere un comportamento autonomo cedevole che si adatti alla conformazione dell'ambiente in cui il robot si sta spostando, accrescendo le capacità di locomozione del serpente robot. Sfruttando la cedevolezza intrinseca del serpente robot utilizzato in questo lavoro, il SEA Snake, e utilizzando un controllo che combina cedevolezza attiva ad una struttura di coordinazione che ammette una decentralizzazione variabile del robot, si dimostra come tre andature possano essere modificate per ottenere una locomozione efficiente in ambienti complessi non noti a priori o non modellabili

Learning directed locomotion in modular robots with evolvable morphologies

The vision behind this paper looks ahead to evolutionary robot systems where morphologies and controllers are evolved together and ‘newborn’ robots undergo a learning process to optimize their inherited brain for the inherited body. The specific problem we address is learning controllers for the task of directed locomotion in evolvable modular robots. To this end, we present a test suite of robots with different shapes and sizes and compare two learning algorithms, Bayesian optimization and HyperNEAT. The experiments in simulation show that both methods obtain good controllers, but Bayesian optimization is more effective and sample efficient. We validate the best learned controllers by constructing three robots from the test suite in the real world and observe their fitness and actual trajectories. The obtained results indicate a reality gap, but overall the trajectories are adequate and follow the target directions successfully

Evolutionary Distributed Control of a Biologically Inspired Modular Robot

fals

Towards Informed Exploration for Deep Reinforcement Learning

In this thesis, we discuss various techniques for improving exploration for deep reinforcement learning. We begin with a brief review of reinforcement learning (RL) and the fundamental v.s. exploitation trade-off. Then we review how deep RL has improved upon classical and summarize six categories of the latest exploration methods for deep RL, in the order increasing usage of prior information. We then explore representative works in three categories discuss their strengths and weaknesses. The first category, represented by Soft Q-learning, uses regularization to encourage exploration. The second category, represented by count-based via hashing, maps states to hash codes for counting and assigns higher exploration to less-encountered states. The third category utilizes hierarchy and is represented by modular architecture for RL agents to play StarCraft II. Finally, we conclude that exploration by prior knowledge is a promising research direction and suggest topics of potentially impact

Static and bootstrapped neuro-simulation for complex robots in evolutionary robotics

Evolutionary Robotics (ER) is a field of study focused on the automatic development of controllers and robot morphologies. Evolving controllers on real-world hardware is time-consuming and can damage hardware through wear. Robotic simulators can be used as an alternative to a real-world robot in order to speed up the ER process. Most simulation techniques in practice use physics-based models that rely on an understanding of the robotic system in question. Developing effective physics-based simulators is time consuming and requires a significant level of specialised knowledge. A lengthy simulator development and tuning process is typically required before the ER process can begin. Artificial Neural Networks simulators (SNNs) can be used as an alternative to a physics based simulation approach. SNNs are simple to construct, do not require significant levels of prior knowledge of the robotic system, are computationally efficient and can be highly accurate. Two types of ER approaches utilising SNNs exist. The Static Neuro-Simulation (SNS) approach involves developing SNNs before the ER process where these SNNs are used instead of a physics-based simulator. Alternatively, SNNs can be developed during the ER process, called the Bootstrapped Neuro-Simulation (BNS) approach. Prior work investigating SNNs has largely been limited to simple robots. A complex robot has many degrees of freedom and ifa low-level controller design is used, the solution search space is high-dimensional and difficult to traverse. Prior work investigating the SNS and BNS approaches have mostly relied on simplified controller designs which rely on built-in prior knowledge of intended robot behaviours. This research uses low-level controller designs which in turn rely on low level simulators. Most ER studies are conducted on a single type of robot morphology. This research investigates the SNS and BNS approaches on two significantly different classes of robots. A Hexapod and Snake robot are used to study the SNS and BNS approaches. The Hexapod robot exhibits limbed, walking behaviours. The Snake robot is limbless and generates crawling behaviours. Demonstrating the viability of the SNS and BNS approaches for two different classes of robots provides strong evidence that the tested approaches are likely viable on other classes of robots. Various proposed improvements to the SNS and BNS approaches are investigated. The Results demonstrate that the SNS and BNS approaches are viable when applied to Hexapod and Snake robots without restricting controller designs to those with significant levels of built-in prior knowledge of robot behaviours. SNNs configured in ensembles improve the likely performance outcomes of solutions. The expected benefit of adding simulator noise during the evolutionary process were not as pronounced for problems investigated in this work

A Comparative Analysis of Darwinian Asexual and Sexual Reproduction in Evolutionary Robotics

Evolutionary Robotics systems draw inspiration from natural evolution to solve the problem of robot design. A key moment in the evolutionary process is reproduction, when the genotype of one or more parents is inherited by their offspring. Existent approaches have used both sexual and asexual reproduction but a comparison between the two is still missing. In this work, we study the effects of sexual and asexual reproduction on the controllers of an Evolutionary Robotics system. In our system, both morphologies and controllers are jointly evolved to solve two separate tasks. We adopt the Triangle of Life framework, in which the controllers go through a phase of learning before reproduction. Using extensive simulations we show that sexual reproduction of the robots' brains is preferable over asexual reproduction as it obtains better robots in terms of fitness. Moreover, we show that sexually reproducing robots present different morphologies and behaviors than the asexually reproducing ones, even though the reproduction mechanism only affects their brains. Finally, we study the effects of the reproduction mechanism on the robots' learning capabilities. By measuring the difference between the inherited and the learned brain we find that robots that evolved using sexual reproduction have better inherited brains and are also better learners

Vision-based control of multi-agent systems

Scope and Methodology of Study: Creating systems with multiple autonomous vehicles places severe demands on the design of decision-making supervisors, cooperative control schemes, and communication strategies. In last years, several approaches have been developed in the literature. Most of them solve the vehicle coordination problem assuming some kind of communications between team members. However, communications make the group sensitive to failure and restrict the applicability of the controllers to teams of friendly robots. This dissertation deals with the problem of designing decentralized controllers that use just local sensor information to achieve some group goals.Findings and Conclusions: This dissertation presents a decentralized architecture for vision-based stabilization of unmanned vehicles moving in formation. The architecture consists of two main components: (i) a vision system, and (ii) vision-based control algorithms. The vision system is capable of recognizing and localizing robots. It is a model-based scheme composed of three main components: image acquisition and processing, robot identification, and pose estimation.Using vision information, we address the problem of stabilizing groups of mobile robots in leader- or two leader-follower formations. The strategies use relative pose between a robot and its designated leader or leaders to achieve formation objectives. Several leader-follower formation control algorithms, which ensure asymptotic coordinated motion, are described and compared. Lyapunov's stability theory-based analysis and numerical simulations in a realistic tridimensional environment show the stability properties of the control approaches