Adaptive, fast walking in a biped robot under neuronal control and learning

Human walking is a dynamic, partly self-stabilizing process relying on the interaction of the biomechanical design with its neuronal control. The coordination of this process is a very difficult problem, and it has been suggested that it involves a hierarchy of levels, where the lower ones, e.g., interactions between muscles and the spinal cord, are largely autonomous, and where higher level control (e.g., cortical) arises only pointwise, as needed. This requires an architecture of several nested, sensori–motor loops where the walking process provides feedback signals to the walker's sensory systems, which can be used to coordinate its movements. To complicate the situation, at a maximal walking speed of more than four leg-lengths per second, the cycle period available to coordinate all these loops is rather short. In this study we present a planar biped robot, which uses the design principle of nested loops to combine the self-stabilizing properties of its biomechanical design with several levels of neuronal control. Specifically, we show how to adapt control by including online learning mechanisms based on simulated synaptic plasticity. This robot can walk with a high speed (> 3.0 leg length/s), self-adapting to minor disturbances, and reacting in a robust way to abruptly induced gait changes. At the same time, it can learn walking on different terrains, requiring only few learning experiences. This study shows that the tight coupling of physical with neuronal control, guided by sensory feedback from the walking pattern itself, combined with synaptic learning may be a way forward to better understand and solve coordination problems in other complex motor tasks

Locomotion through morphology, evolution and learning for legged and limbless robots

Mención Internacional en el título de doctorRobot locomotion is concerned with providing autonomous locomotion capabilities to mobile robots. Most current day robots feature some form of locomotion for navigating in their environment. Modalities of robot locomotion includes: (i) aerial locomotion, (ii) terrestrial locomotion, and (iii) aquatic locomotion (on or under water). Three main forms of terrestrial locomotion are, legged locomotion, limbless locomotion and wheel-based locomotion. A Modular Robot (MR), on the other hand, is a robotic system composed of several independent unit modules, where, each module is a robot by itself. The objective in this thesis is to develop legged locomotion in a humanoid robot, as well as, limbless locomotion in modular robotic configurations. Taking inspiration from biology, robot locomotion from the perspective of robot’s morphology, through evolution, and through learning are investigated in this thesis. Locomotion is one of the key distinguishing characteristics of a zoological organism. Almost all animal species, and even some plant species, produce some form of locomotion. In the past few years, robots have been “moving out” of the factory floor and research labs, and are becoming increasingly common in everyday life. So, providing stable and agile locomotion capabilities for robots to navigate a wide range of environments becomes pivotal. Developing locomotion in robots through biologically inspired methods, also facilitates furthering our understanding on how biological processes may function. Connected modules in a configuration, exert force on each other as a result of interaction between each other and their environment. This phenomenon is studied and quantified, and then used as implicit communication between robot modules for producing locomotion coordination in MRs. Through this, a strong link between robot morphology and the gait that emerge in it is established. A variety of locomotion controller, some periodic-function based and some morphology based, are developed for MR locomotion and bipedal gait generation. A hybrid Evolutionary Algorithm (EA) is implemented for evolving gaits, both in simulation as well as in the real-world on a physical modular robotic configuration. Limbless gaits in MRs are also learnt by learning optimal control policies, through Reinforcement Learning (RL).En robótica, la locomoción trata de proporcionar capacidades de locomoción autónoma a robots móviles. La mayoría de los robots actuales tiene alguna forma de locomoción para navegar en su entorno. Los modos de locomoción robótica se pueden repartir entre: (i) locomoción aérea, (ii) locomoción terrestre, y (iii) locomoción acuática (sobre o bajo el agua). Las tres formas básicas de locomoción terrestre son la locomoción mediante piernas, la locomoción sin miembros, y la locomoción basada en ruedas. Un Robot Modular, por otra parte, es un sistema robótico compuesto por varios módulos independientes, donde cada módulo es un robot en sí mismo. El objetivo de esta tesis es el desarrollo de la locomoción mediante piernas para un robot humanoide, así como el de la locomoción sin miembros para varias configuraciones de robots modulares. Inspirándose en la biología, también se investiga en esta tesis el desarrollo de la locomoción del robot según su morfología, gracias a técnicas de evolución y de aprendizaje. La locomoción es una de las características distintivas de un organismo zoológico. Casi todas las especies animales, e incluso algunas especies de plantas, poseen algún tipo de locomoción. En los últimos años, los robots han “migrado” desde las fábricas y los laboratorios de investigación, y se están integrando cada vez más en nuestra vida diaria. Por estas razones, es crucial proporcionar capacidades de locomoción estables y ágiles a los robots para que puedan navegar por todo tipo de entornos. El uso de métodos de inspiración biológica para alcanzar esta meta también nos ayuda a entender mejor cómo pueden funcionar los procesos biológicos equivalentes. En una configuración de módulos conectados, puesto que cada uno interacciona con su entorno, los módulos ejercen fuerza los unos sobre los otros. Este fenómeno se ha estudiado y cuantificado, y luego se ha usado como comunicación implícita entre los módulos para producir la coordinación en la locomoción de este robot. De esta manera, se establece un fuerte vínculo entre la morfología de un robot y el modo de andar que este desarrolla. Se han desarrollado varios controladores de locomoción para robots modulares y robots bípedos, algunos basados en funciones periódicas, otros en la morfología del robot. Un algoritmo evolutivo híbrido se ha implementado para la evolución de locomociones, tanto en simulación como en el mundo real en una configuración física de robot modular. También se pueden generar locomociones sin miembros para robots modulares, determinando las políticas de control óptimo gracias a técnicas de aprendizaje por refuerzo. Se presenta en primer lugar en esta tesis el estado del arte de la robótica modular, enfocándose en la locomoción de robots modulares, los controladores, la locomoción bípeda y la computación morfológica. A continuación se describen cinco configuraciones diferentes de robot modular que se utilizan en esta tesis, seguido de cuatro controladores de locomoción. Estos controladores son el controlador heterogéneo, el controlador basado en funciones periódicas, el controlador homogéneo y el controlador basado en la morfología del robot. Se desarrolla como parte de este trabajo un controlador de locomoción lineal, periódico, basado en features, para la locomoción bípeda de robots humanoides. Los parámetros de control se ajustan primero a mano para reproducir un modelo cart-table, y el controlador se evalúa en un robot humanoide simulado. A continuación, gracias a un algoritmo evolutivo, la optimización de los parámetros de control permite desarrollar una locomoción sin modelo predeterminado. Se desarrolla como parte de esta tesis un enfoque sobre algoritmos de Embodied Evolución, en otras palabras el uso de robots modulares físicos en la fase de evolución. La implementación material, la configuración experimental, y el Algoritmo Evolutivo implementado para Embodied Evolución, se explican detalladamente. El trabajo también incluye una visión general de las técnicas de aprendizaje por refuerzo y de los Procesos de Decisión de Markov. A continuación se presenta un algoritmo popular de aprendizaje por refuerzo, llamado Q-Learning, y su adaptación para aprender locomociones de robots modulares. Se proporcionan una implementación del algoritmo de aprendizaje y la evaluación experimental de la locomoción generada.Programa Oficial de Doctorado en Ingeniería Eléctrica, Electrónica y AutomáticaPresidente: Antonio Barrientos Cruz.- Secretario: Luis Santiago Garrido Bullón.- Vocal: Giuseppe Carbon

Reinforcement Learning of CPG-regulated Locomotion Controller for a Soft Snake Robot

Intelligent control of soft robots is challenging due to the nonlinear and difficult-to-model dynamics. One promising model-free approach for soft robot control is reinforcement learning (RL). However, model-free RL methods tend to be computationally expensive and data-inefficient and may not yield natural and smooth locomotion patterns for soft robots. In this work, we develop a bio-inspired design of a learning-based goal-tracking controller for a soft snake robot. The controller is composed of two modules: An RL module for learning goal-tracking behaviors given the unmodeled and stochastic dynamics of the robot, and a central pattern generator (CPG) with the Matsuoka oscillators for generating stable and diverse locomotion patterns. We theoretically investigate the maneuverability of Matsuoka CPG's oscillation bias, frequency, and amplitude for steering control, velocity control, and sim-to-real adaptation of the soft snake robot. Based on this analysis, we proposed a composition of RL and CPG modules such that the RL module regulates the tonic inputs to the CPG system given state feedback from the robot, and the output of the CPG module is then transformed into pressure inputs to pneumatic actuators of the soft snake robot. This design allows the RL agent to naturally learn to entrain the desired locomotion patterns determined by the CPG maneuverability. We validated the optimality and robustness of the control design in both simulation and real experiments, and performed extensive comparisons with state-of-art RL methods to demonstrate the benefit of our bio-inspired control design.Comment: 20 pages, 17 figures, 4 tables, in IEEE Transactions on Robotic

CPU-less robotics: distributed control of biomorphs

Traditional robotics revolves around the microprocessor. All well-known demonstrations of sensory guided motor control, such as jugglers and mobile robots, require at least one CPU. Recently, the availability of fast CPUs have made real-time sensory-motor control possible, however, problems with high power consumption and lack of autonomy still remain. In fact, the best examples of real-time robotics are usually tethered or require large batteries. We present a new paradigm for robotics control that uses no explicit CPU. We use computational sensors that are directly interfaced with adaptive actuation units. The units perform motor control and have learning capabilities. This architecture distributes computation over the entire body of the robot, in every sensor and actuator. Clearly, this is similar to biological sensory- motor systems. Some researchers have tried to model the latter in software, again using CPUs. We demonstrate this idea in with an adaptive locomotion controller chip. The locomotory controller for walking, running, swimming and flying animals is based on a Central Pattern Generator (CPG). CPGs are modeled as systems of coupled non-linear oscillators that control muscles responsible for movement. Here we describe an adaptive CPG model, implemented in a custom VLSI chip, which is used to control an under-actuated and asymmetric robotic leg

Use of neural oscillators triggered by loading and hip angles to study the activation patterns at the ankle during walking in humans

Spinale Mustergeneratoren (SPG) sind neuronale Netze ohne supraspinales Input, die zyklische Bewegungen steuern. Wir wollten untersuchen, ob sich SPG an die variablen Anforderungen verschiedener Geschwindigkeiten, Störungen und ungewöhnlicher Koordinationsmuster beim Gehen anpassen können. Das SPG-Modell ist ein Oszillator aus zwei Neuronen; eines aktiviert einen Dorsalextensor und das andere einen Plantarflexor. Das Output des Oszillators repräsentiert die jeweilige Muskelaktivierung. Die Modellparameter wurden angepasst, um eine optimale Passung zwischen simulierten und gemessenen elektromyographischen Daten von gesunden Probanden zu erzielen. Eine hohe Korrelation zwischen simulierten und gemessenen Muskelaktivierungen beim normalen Gehen wies darauf hin, dass spinale Kontrolle in Modellen vom Gehen beim Menschen berücksichtigt sollte werden. Unsere experimentellen Ergebnisse zeigen, dass der Soleus vom Rückenmark kontrolliert werden könnte, aber nicht der Tibialis anterior

The Future of Humanoid Robots

This book provides state of the art scientific and engineering research findings and developments in the field of humanoid robotics and its applications. It is expected that humanoids will change the way we interact with machines, and will have the ability to blend perfectly into an environment already designed for humans. The book contains chapters that aim to discover the future abilities of humanoid robots by presenting a variety of integrated research in various scientific and engineering fields, such as locomotion, perception, adaptive behavior, human-robot interaction, neuroscience and machine learning. The book is designed to be accessible and practical, with an emphasis on useful information to those working in the fields of robotics, cognitive science, artificial intelligence, computational methods and other fields of science directly or indirectly related to the development and usage of future humanoid robots. The editor of the book has extensive R&D experience, patents, and publications in the area of humanoid robotics, and his experience is reflected in editing the content of the book

Toward simple control for complex, autonomous robotic applications: combining discrete and rhythmic motor primitives

Vertebrates are able to quickly adapt to new environments in a very robust, seemingly effortless way. To explain both this adaptivity and robustness, a very promising perspective in neurosciences is the modular approach to movement generation: Movements results from combinations of a finite set of stable motor primitives organized at the spinal level. In this article we apply this concept of modular generation of movements to the control of robots with a high number of degrees of freedom, an issue that is challenging notably because planning complex, multidimensional trajectories in time-varying environments is a laborious and costly process. We thus propose to decrease the complexity of the planning phase through the use of a combination of discrete and rhythmic motor primitives, leading to the decoupling of the planning phase (i.e. the choice of behavior) and the actual trajectory generation. Such implementation eases the control of, and the switch between, different behaviors by reducing the dimensionality of the high-level commands. Moreover, since the motor primitives are generated by dynamical systems, the trajectories can be smoothly modulated, either by high-level commands to change the current behavior or by sensory feedback information to adapt to environmental constraints. In order to show the generality of our approach, we apply the framework to interactive drumming and infant crawling in a humanoid robot. These experiments illustrate the simplicity of the control architecture in terms of planning, the integration of different types of feedback (vision and contact) and the capacity of autonomously switching between different behaviors (crawling and simple reaching