Distributed reinforcement learning for self-reconfiguring modular robots

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2007.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Includes bibliographical references (p. 101-106).In this thesis, we study distributed reinforcement learning in the context of automating the design of decentralized control for groups of cooperating, coupled robots. Specifically, we develop a framework and algorithms for automatically generating distributed controllers for self-reconfiguring modular robots using reinforcement learning. The promise of self-reconfiguring modular robots is that of robustness, adaptability and versatility. Yet most state-of-the-art distributed controllers are laboriously handcrafted and task-specific, due to the inherent complexities of distributed, local-only control. In this thesis, we propose and develop a framework for using reinforcement learning for automatic generation of such controllers. The approach is profitable because reinforcement learning methods search for good behaviors during the lifetime of the learning agent, and are therefore applicable to online adaptation as well as automatic controller design. However, we must overcome the challenges due to the fundamental partial observability inherent in a distributed system such as a self reconfiguring modular robot. We use a family of policy search methods that we adapt to our distributed problem. The outcome of a local search is always influenced by the search space dimensionality, its starting point, and the amount and quality of available exploration through experience.(cont) We undertake a systematic study of the effects that certain robot and task parameters, such as the number of modules, presence of exploration constraints, availability of nearest-neighbor communications, and partial behavioral knowledge from previous experience, have on the speed and reliability of learning through policy search in self-reconfiguring modular robots. In the process, we develop novel algorithmic variations and compact search space representations for learning in our domain, which we test experimentally on a number of tasks. This thesis is an empirical study of reinforcement learning in a simulated lattice based self-reconfiguring modular robot domain. However, our results contribute to the broader understanding of automatic generation of group control and design of distributed reinforcement learning algorithms.by Paulina Varshavskaya.Ph.D

Autonomous Task-Based Evolutionary Design of Modular Robots

In an attempt to solve the problem of finding a set of multiple unique modular robotic designs that can be constructed using a given repertoire of modules to perform a specific task, a novel synthesis framework is introduced based on design optimization concepts and evolutionary algorithms to search for the optimal design. Designing modular robotic systems faces two main challenges: the lack of basic rules of thumb and design bias introduced by human designers. The space of possible designs cannot be easily grasped by human designers especially for new tasks or tasks that are not fully understood by designers. Therefore, evolutionary computation is employed to design modular robots autonomously. Evolutionary algorithms can efficiently handle problems with discrete search spaces and solutions of variable sizes as these algorithms offer feasible robustness to local minima in the search space; and they can be parallelized easily to reducing system runtime. Moreover, they do not have to make assumptions about the solution form. This dissertation proposes a novel autonomous system for task-based modular robotic design based on evolutionary algorithms to search for the optimal design. The introduced system offers a flexible synthesis algorithm that can accommodate to different task-based design needs and can be applied to different modular shapes to produce homogenous modular robots. The proposed system uses a new representation for modular robotic assembly configuration based on graph theory and Assembly Incidence Matrix (AIM), in order to enable efficient and extendible task-based design of modular robots that can take input modules of different geometries and Degrees Of Freedom (DOFs). Robotic simulation is a powerful tool for saving time and money when designing robots as it provides an accurate method of assessing robotic adequacy to accomplish a specific task. Furthermore, it is difficult to predict robotic performance without simulation. Thus, simulation is used in this research to evaluate the robotic designs by measuring the fitness of the evolved robots, while incorporating the environmental features and robotic hardware constraints. Results are illustrated for a number of benchmark problems. The results presented a significant advance in robotic design automation state of the art

Addressing Tasks Through Robot Adaptation

Developing flexible, broadly capable systems is essential for robots to move out of factories and into our daily lives, functioning as responsive agents that can handle whatever the world throws at them. This dissertation focuses on two kinds of robot adaptation. Modular self-reconfigurable robots (MSRR) adapt to the requirements of their task and environments by transforming themselves. By rearranging the connective structure of their component robot modules, these systems can assume different morphologies: for example, a cluster of modules might configure themselves into a car to maneuver on flat ground, a snake to climb stairs, or an arm to pick and place objects. Conversely, environment augmentation is a strategy in which the robot transforms its environment to meet its own needs, adding physical structures that allow it to overcome obstacles. In both areas, the presented work includes elements of hardware design, algorithms, and integrated systems, with the common goal of establishing these methods of adaptation as viable strategies to address tasks. The research takes a systems-level view of robotics, placing particular emphasis on experimental validation in hardware

Evolutionary Modular Robotics: Survey and Analysis

This paper surveys various applications of artificial evolution in the field of modular robots. Evolutionary robotics aims to design autonomous adaptive robots automatically that can evolve to accomplish a specific task while adapting to environmental changes. A number of studies have demonstrated the feasibility of evolutionary algorithms for generating robotic control and morphology. However, a huge challenge faced was how to manufacture these robots. Therefore, modular robots were employed to simplify robotic evolution and their implementation in real hardware. Consequently, more research work has emerged on using evolutionary computation to design modular robots rather than using traditional hand design approaches in order to avoid cognition bias. These techniques have the potential of developing adaptive robots that can achieve tasks not fully understood by human designers. Furthermore, evolutionary algorithms were studied to generate global modular robotic behaviors including; self-assembly, self-reconfiguration, self-repair, and self-reproduction. These characteristics allow modular robots to explore unstructured and hazardous environments. In order to accomplish the aforementioned evolutionary modular robotic promises, this paper reviews current research on evolutionary robotics and modular robots. The motivation behind this work is to identify the most promising methods that can lead to developing autonomous adaptive robotic systems that require the minimum task related knowledge on the designer side.https://doi.org/10.1007/s10846-018-0902-

The Propulsion of Reconfigurable Modular Robots in Fluidic Environments

Reconfigurable modular robots promise to transform the way robotic systems are designed and operated. Fluidic or microgravity environments, which can be difficult or dangerous for humans to work in, are ideal domains for the use of modular systems. This thesis proposes that combining effective propulsion, large reconfiguration space and high scalability will increase the utility of modular robots. A novel concept for the propulsion of reconfigurable modular robots is developed. Termed Modular Fluidic Propulsion (MFP), this concept describes a system that propels by routing fluid though itself. This allows MFP robots to self-propel quickly and effectively in any configuration, while featuring a cubic lattice structure. A decentralized occlusion-based motion controller for the system is developed. The simplicity of the controller, which requires neither run-time memory nor computation via logic units, combined with the simple binary sensors and actuators of the robot, gives the system a high level of scalabilty. It is proven formally that 2-D MFP robots are able to complete a directed locomotion task under certain assumptions. Simulations in 3-D show that robots composed of 125 modules in a variety of configurations can complete the task. A hardware prototype that floats on the surface of water is developed. Experiments show that robots composed of four modules can complete the task in any configuration. This thesis also investigates the evo-bots, a self-reconfigurable modular system that floats in 2-D on an air table. The evo-bot system uses a stop-start propulsion mechanism to choose between moving randomly or not moving at all. This is demonstrated experimentally for the first time. In addition, the ability of the modules to detect, harvest and share energy, as well as self-assemble into simple structures, is demonstrated

A modular robot architecture capable of learning to move and be automatically reconfigured

Tackling the problem of making a modular robot automatically learn the movements necessary to locomote in different environments is not an easy task. The ability of modular robots to have an arbitrary morphology provides an advantage over usual monolithic robots when moving in different environments. However, being able to reconfigure also has its problems. Movement control for reconfigurable robots is difficult to design and implement. Morphology can also influence the sensing capabilities of a modular robot. Only a few studies include sensor information when adjusting or optimizing controllers for modular robots. The main contribution of this work is the development of an architecture that includes a locomotion training framework that enables a modular robot to move in different environments taking into account sensor information. The framework is composed of four main parts: a control strategy, a configurable environment approach, an adaptation mechanism and a new modular robot platform: the EMERGE modular robot. The EMERGE modular robot platform is designed to be easy to be assembled and can be quickly reconfigured thanks to the magnetic connectors present in its modules. This in turn enables an external agent, like a robot manipulator to reconfigure the robot. Results show that well coordinated movements turn out to be very important for controllers using sensors to improve when being adapted. The mechanisms inside the controller, for example, decision structures, also play a major part in allowing a robot to adapt to move in different environments and be improved. Evaluating robots in reality is a very expensive task and differences between simulation and reality also make robots behave very differently. The magnetic connector makes the assembly of an EMERGE morphology easier but hinders the disassembly process.Resumen: Resolver el problema de hacer, de forma automática, que un robot modular se mueva en diferentes ambientes no es tarea fácil. La habilidad de los robots modulares de tener morfología arbitraria provee una ventaja sobre robots monolíticos normales al moverse en diferentes ambientes. Sin embargo, ser capaz de auto reconfigurarse tiene sus propios problemas. El control de movimiento para robots modulares es difícil de diseñar e implementar. La morfología de los robots también influencia la capacidad de percibir de los robots modulares. Solo contados estudios incluyen información sensorial al ajustar u optimizar controladores para este tipo de robots. La mayor contribución de este trabajo es el desarrollo de una arquitectura de robot modular que hace que este pueda moverse en diferentes ambientes teniendo en cuenta información sensorial. Esta arquitectura está compuesta por cuatro partes principales: una estrategia de control, un modelo de ambiente configurable, un mecanismo de adaptación y una plataforma de robot modular nueva: el robot EMERGE. El robot modular EMERGE, es diseñado para ser fácil de construir y de reconfigurar gracias a sus conectores magnéticos. Esto también posibilita a un agente externo, como un manipulador robótico, a reconfigurar el robot. Los resultados de los experimentos muestran que la buena coordinación del robot es muy importante para que los controles que usan sensores puedan mejorar. Los mecanismos internos del controlador, por ejemplo, las estructuras de decisión también tienen un rol importante al adaptar el robot a diferentes ambientes. Evaluar robots en la realidad es una tarea muy costosa y las diferencias entre la simulación y la realidad hacen que los robots se comporten muy diferente. Los conectores magnéticos hacen que armar las morfologías de módulos de EMERGE sean fáciles de armar, mas no de desarmar.Doctorad