Accelerating Reinforcement Learning by Composing Solutions of Automatically Identified Subtasks

This paper discusses a system that accelerates reinforcement learning by using transfer from related tasks. Without such transfer, even if two tasks are very similar at some abstract level, an extensive re-learning effort is required. The system achieves much of its power by transferring parts of previously learned solutions rather than a single complete solution. The system exploits strong features in the multi-dimensional function produced by reinforcement learning in solving a particular task. These features are stable and easy to recognize early in the learning process. They generate a partitioning of the state space and thus the function. The partition is represented as a graph. This is used to index and compose functions stored in a case base to form a close approximation to the solution of the new task. Experiments demonstrate that function composition often produces more than an order of magnitude increase in learning rate compared to a basic reinforcement learning algorithm

A mosaic of eyes

Autonomous navigation is a traditional research topic in intelligent robotics and vehicles, which requires a robot to perceive its environment through onboard sensors such as cameras or laser scanners, to enable it to drive to its goal. Most research to date has focused on the development of a large and smart brain to gain autonomous capability for robots. There are three fundamental questions to be answered by an autonomous mobile robot: 1) Where am I going? 2) Where am I? and 3) How do I get there? To answer these basic questions, a robot requires a massive spatial memory and considerable computational resources to accomplish perception, localization, path planning, and control. It is not yet possible to deliver the centralized intelligence required for our real-life applications, such as autonomous ground vehicles and wheelchairs in care centers. In fact, most autonomous robots try to mimic how humans navigate, interpreting images taken by cameras and then taking decisions accordingly. They may encounter the following difficulties

Bio-Inspired Robotics

Modern robotic technologies have enabled robots to operate in a variety of unstructured and dynamically-changing environments, in addition to traditional structured environments. Robots have, thus, become an important element in our everyday lives. One key approach to develop such intelligent and autonomous robots is to draw inspiration from biological systems. Biological structure, mechanisms, and underlying principles have the potential to provide new ideas to support the improvement of conventional robotic designs and control. Such biological principles usually originate from animal or even plant models, for robots, which can sense, think, walk, swim, crawl, jump or even fly. Thus, it is believed that these bio-inspired methods are becoming increasingly important in the face of complex applications. Bio-inspired robotics is leading to the study of innovative structures and computing with sensory–motor coordination and learning to achieve intelligence, flexibility, stability, and adaptation for emergent robotic applications, such as manipulation, learning, and control. This Special Issue invites original papers of innovative ideas and concepts, new discoveries and improvements, and novel applications and business models relevant to the selected topics of ``Bio-Inspired Robotics''. Bio-Inspired Robotics is a broad topic and an ongoing expanding field. This Special Issue collates 30 papers that address some of the important challenges and opportunities in this broad and expanding field

분산된 로터로 구동되는 비행 스켈레톤 시스템의 디자인 상태추정 및 제어

학위논문(박사)--서울대학교 대학원 :공과대학 기계항공공학부,2020. 2. 이동준.In this thesis, we present key theoretical components for realizing flying aerial skeleton system called LASDRA (large-size aerial skeleton with distributed rotor actuation). Aerial skeletons are articulated aerial robots actuated by distributed rotors including both ground connected type and flying type. These systems have recently attracted interest and are being actively researched in several research groups, with the expectation of applying those for aerial manipulation in distant/narrow places, or for the performance with entertaining purpose such as drone shows. Among the aerial skeleton systems, LASDRA system, proposed by our group has some significant advantages over the other skeleton systems that it is capable of free SE(3) motion by omni-directional wrench generation of each link, and also the system can be operated with wide range of configuration because of the 3DOF (degrees of freedom) inter-link rotation enabled by cable connection among the link modules. To realize this LASDRA system, following three components are crucial: 1) a link module that can produce omni-directional force and torque and enough feasible wrench space; 2) pose and posture estimation algorithm for an articulated system with high degrees of freedom; and 3) a motion generation framework that can provide seemingly natural motion while being able to generate desired motion (e.g., linear and angular velocity) for the entire body. The main contributions of this thesis is theoretically developing these three components, and verifying these through outdoor flight experiment with a real LASDRA system. First of all, a link module for the LASDRA system is designed with proposed constrained optimization problem, maximizing the guaranteed feasible force and torque for any direction while also incorporating some constraints (e.g., avoiding inter-rotor air-flow interference) to directly obtain feasible solution. Also, an issue of ESC-induced (electronic speed control) singularity is first introduced in the literature which is inevitably caused by bi-directional thrust generation with sensorless actuators, and handled with a novel control allocation called selective mapping. Then for the state estimation of the entire LASDRA system, constrained Kalman filter based estimation algorithm is proposed that can provide estimation result satisfying kinematic constraint of the system, also along with a semi-distributed version of the algorithm to endow with system scalability. Lastly, CPG-based motion generation framework is presented that can generate natural biomimetic motion, and by exploiting the inverse CPG model obtained with machine learning method, it becomes possible to generate certain desired motion while still making CPG generated natural motion.본 논문에서는 비행 스켈레톤 시스템 LASDRA (large-size aerial skeleton with distributed rotor actuation) 의 구현을 위해 요구되는 핵심 기법들을 제안하며, 이를 실제 LASDRA 시스템의 실외 비행을 통해 검증한다. 제안된 기법은 1) 전방향으로 힘과 토크를 낼 수 있고 충분한 가용 렌치공간을 가진 링크 모듈, 2) 높은 자유도의 다관절구조 시스템을 위한 위치 및 자세 추정 알고리즘, 3) 자연스러운 움직임을 내는 동시에 전체 시스템이 속도, 각속도 등 원하는 움직임을 내도록 할 수 있는 모션 생성 프레임워크로 구성된다. 본 논문에서는 우선 링크 모듈의 디자인을 위해 전방향으로 보장되는 힘과 토크의 크기를 최대화하는 구속 최적화를 사용하고, 실제 적용가능한 해를 얻기 위해 몇가지 구속조건(로터 간 공기 흐름 간섭의 회피 등)을 고려한다. 또한 센서가 없는 액츄에이터로 양방향 추력을 내는 것에서 야기되는 ESC 유발 특이점 (ESC-induced singularity) 이라는 문제를 처음으로 소개하고, 이를 해결하기 위해 선택적 맵핑 (selective mapping) 이라는 기법을 제시한다. 전체 LASDRA 시스템의 상태추정을 위해 시스템의 기구학적 구속조건을 만족하는 결과를 얻을 수 있도록 구속 칼만 필터 기반의 상태추정 기법을 제시하고, 시스템 확장성을 고려하여 반 분산 (semi-distributed) 개념의 알고리즘을 함께 제시한다. 마지막으로 본 논문에서는 자연스러운 움직임의 생성을 위하여 CPG 기반의 모션 생성 프레임워크를 제안하며, 기계 학습 방법을 통해 CPG 역연산 모델을 얻음으로써 전체 시스템이 원하는 움직임을 낼 수 있도록 한다.1 Introduction 1 1.1 Motivation and Background 1 1.2 Research Problems and Approach 3 1.3 Preview of Contributions 5 2 Omni-Directional Aerial Robot 7 2.1 Introduction 7 2.2 Mechanical Design 12 2.2.1 Design Description 12 2.2.2 Wrench-Maximizing Design Optimization 13 2.3 System Modeling and Control Design 20 2.3.1 System Modeling 20 2.3.2 Pose Trajectory Tracking Control 22 2.3.3 Hybrid Pose/Wrench Control 22 2.3.4 PSPM-Based Teleoperation 24 2.4 Control Allocation with Selective Mapping 27 2.4.1 Infinity-Norm Minimization 27 2.4.2 ESC-Induced Singularity and Selective Mapping 29 2.5 Experiment 38 2.5.1 System Setup 38 2.5.2 Experiment Results 41 2.6 Conclusion 49 3 Pose and Posture Estimation of an Aerial Skeleton System 51 3.1 Introduction 51 3.2 Preliminary 53 3.3 Pose and Posture Estimation 55 3.3.1 Estimation Algorithm via SCKF 55 3.3.2 Semi-Distributed Version of Algorithm 59 3.4 Simulation 62 3.5 Experiment 65 3.5.1 System Setup 65 3.5.2 Experiment of SCKF-Based Estimation Algorithm 66 3.6 Conclusion 69 4 CPG-Based Motion Generation 71 4.1 Introduction 71 4.2 Description of Entire Framework 75 4.2.1 LASDRA System 75 4.2.2 Snake-Like Robot & Pivotboard 77 4.3 CPG Model 79 4.3.1 LASDRA System 79 4.3.2 Snake-Like Robot 80 4.3.3 Pivotboard 83 4.4 Target Pose Calculation with Expected Physics 84 4.5 Inverse Model Learning 86 4.5.1 LASDRA System 86 4.5.2 Snake-Like Robot 89 4.5.3 Pivotboard 90 4.6 CPG Parameter Adaptation 93 4.7 Simulation 94 4.7.1 LASDRA System 94 4.7.2 Snake-Like Robot & Pivotboard 97 4.8 Conclusion 101 5 Outdoor Flight Experiment of the F-LASDRA System 103 5.1 System Setup 103 5.2 Experiment Results 104 6 Conclusion 111 6.1 Summary 111 6.2 Future Works 112Docto

Shape-based compliance control for snake robots

I serpenti robot sono una classe di meccanismi iper-ridondanti che appartiene alla robotica modulare. Grazie alla loro forma snella ed allungata e all'alto grado di ridondanza possono muoversi in ambienti complessi con elevata agilità. L'abilità di spostarsi, manipolare e adattarsi efficientemente ad una grande varietà di terreni li rende ideali per diverse applicazioni, come ad esempio attività di ricerca e soccorso, ispezione o ricognizione. I robot serpenti si muovono nello spazio modificando la propria forma, senza necessità di ulteriori dispositivi quali ruote od arti. Tali deformazioni, che consistono in movimenti ondulatori ciclici che generano uno spostamento dell'intero meccanismo, vengono definiti andature. La maggior parte di esse sono ispirate al mondo naturale, come lo strisciamento, il movimento laterale o il movimento a concertina, mentre altre sono create per applicazioni specifiche, come il rotolamento o l'arrampicamento. Un serpente robot con molti gradi di libertà deve essere capace di coordinare i propri giunti e reagire ad ostacoli in tempo reale per riuscire a muoversi efficacemente in ambienti complessi o non strutturati. Inoltre, aumentare la semplicità e ridurre il numero di controllori necessari alla locomozione alleggerise una struttura di controllo che potrebbe richiedere complessità per ulteriori attività specifiche. L'obiettivo di questa tesi è ottenere un comportamento autonomo cedevole che si adatti alla conformazione dell'ambiente in cui il robot si sta spostando, accrescendo le capacità di locomozione del serpente robot. Sfruttando la cedevolezza intrinseca del serpente robot utilizzato in questo lavoro, il SEA Snake, e utilizzando un controllo che combina cedevolezza attiva ad una struttura di coordinazione che ammette una decentralizzazione variabile del robot, si dimostra come tre andature possano essere modificate per ottenere una locomozione efficiente in ambienti complessi non noti a priori o non modellabili