Master of Science

thesisThis thesis details the development of the Algorithmic Robotics Laboratory, its experimental software environment, and a case study featuring a novel hardware validation of optimal reciprocal collision avoidance. We constructed a robotics laboratory in both software and hardware in which to perform our experiments. This lab features a netted flying volume with motion capture and two custom quadrotors. Also, two experimental software architectures are developed for actuating both ground and aerial robots within a Linux Robot Operating System environment. The first of the frameworks is based upon a single finite state machine program which managed each aspect of the experiment. Concerns about the complexity and reconfigurability of the finite state machine prompted the development of a second framework. This final framework is a multimodal structure featuring programs which focus on these specific functions: State Estimation, Robot Drivers, Experimental Controllers, Inputs, Human Robot Interaction, and a program tailored to the specifics of the algorithm tested in the experiment. These modular frameworks were used to fulfill the mission of the Algorithmic Robotics Lab, in that they were developed to validate robotics algorithms in experiments that were previously only shown in simulation. A case study into collision avoidance was used to mark the foundation of the laboratory through the proving of an optimal reciprocal collision avoidance algorithm for the first time in hardware. In the case study, two human-controlled quadrotors were maliciously flown in colliding trajectories. Optimal reciprocal collision avoidance was demonstrated for the first time on completely independent agents with local sensing. The algorithm was shown to be robust to violations of its inherent assumptions about the dynamics of agents and the ability for those agents to sense imminent collisions. These experiments, in addition to the mathematical foundation of exponential convergence, submits th a t optimal reciprocal collision avoidance is a viable method for holonomic robots in both 2-D and 3-D with noisy sensing. A basis for the idea of reciprocal dance, a motion often seen in human collision avoidance, is also suggested in demonstration to be a product of uncertainty about the state of incoming agents. In the more than one hundred tests conducted in multiple environments, no midair collisions were ever produced

충돌 학습을 통한 지역 경로 계획 방법

학위논문 (석사)-- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2019. 2. 이범희.본 논문에서는 강화 학습 기반의 충돌 회피 방법을 제안한다. 충돌 회피란 로봇이 다른 로봇 또는 장애물과 충돌 없이 목표 지점에 도달하는 것을 목적으로 한다. 이 문제는 단일 로봇 충돌 회피와 다개체 로봇 충돌 회피, 이렇게 두 가지로 나눌 수 있다. 단일 로봇 충돌 회피 문제는 하나의 중심 로봇과 여러 개의 움직이는 장애물로 구성되어 있다. 중심 로봇은 랜덤하게 움직이는 장애물을 피해 목표 지점에 도달하는 것을 목적으로 한다. 다개체 로봇 충돌 회피 문제는 여러 대의 중심 로봇으로 구성되어 있다. 이 문제에도 역시 장애물을 포함시킬 수 있다. 중심 로봇들은 서로 충돌을 회피하면서 각자의 목표 지점에 도달하는 것을 목적으로 한다. 만약 환경에 예상치 못한 장애물이 등장하더라도, 로봇들은 그것들을 피해야 한다. 이 문제를 해결하기 위하여 본 논문에서는 충돌 회피를 위한 충돌 학습 방법 (CALC) 을 제안한다. CALC는 강화 학습 개념을 이용해 문제를 해결한다. 제안하는 방법은 학습 그리고 계획 이렇게 두 가지 환경으로 구성 된다. 학습 환경은 하나의 중심 로봇과 하나의 장애물 그리고 학습 영역으로 구성되어 있다. 학습 환경에서 중심 로봇은 장애물과 충돌하는 법을 학습하고 그에 대한 정책을 도출해 낸다. 즉, 중심 로봇이 장애물과 충돌하게 되면 그것은 양의 보상을 받는다. 그리고 만약 중심 로봇이 장애물과 충돌 하지 않고 학습 영역을 빠져나가면, 그것은 음의 보상을 받는다. 계획 환경은 여러 개의 장애물 또는 로봇들과 하나의 목표 지점으로 구성되어 있다. 학습 환경에서 학습한 정책을 통해 중심 로봇은 여러 대의 장애물 또는 로봇들과의 충돌을 피할 수 있다. 본 방법은 충돌을 학습 했기 때문에, 충돌을 회피하기 위해서는 도출된 정책을 뒤집어야 한다. 하지만, 목표 지점과는 일종의 `충돌'을 해야하기 때문에, 목표 지점에 대해서는 도출된 정책을 그대로 적용해야 한다. 이 두 가지 종류의 정책들을 융합하게 되면, 중심 로봇은 장애물 또는 로봇들과의 충돌을 회피하면서 동시에 목표 지점에 도달할 수 있다. 학습 환경에서 로봇은 홀로노믹 로봇을 가정한다. 학습된 정책이 홀로노믹 로봇을 기반으로 하더라도, 제안하는 방법은 홀로노믹 로봇과 비홀로노믹 로봇 모두에 적용이 가능하다. CALC는 다음의 세 가지 문제에 적용할 수 있다. 1) 홀로노믹 단일 로봇의 충돌 회피. 2) 비홀로노믹 단일 로봇의 충돌 회피. 3) 비홀로노믹 다개체 로봇의 충돌 회피. 제안된 방법은 시뮬레이션과 실제 로봇 환경에서 실험 되었다. 시뮬레이션은 로봇 운영체제 (ROS) 기반의 시뮬레이터인 가제보와 게임 라이브러리의 한 종류인 PyGame을 사용하였다. 시뮬레이션에서는 홀로노믹과 비홀로노믹 로봇을 모두 사용하여 실험을 진행하였다. 실제 로봇 환경 실험에서는 비홀로노믹 로봇의 한 종류인 e-puck 로봇을 사용하였다. 또한, 시뮬레이션에서 학습된 정책은 실제 로봇 환경 실험에서 재학습 또는 별도의 수정과정 없이 바로 적용이 가능하였다. 이러한 실험들의 결과를 통해 제안된 방법은 Reciprocal Velocity Obstacle (RVO) 또는 Optimal Reciprocal Collision Avoidance (ORCA)와 같은 기존의 방법들과 비교하였을 때 향상된 성능을 보였다. 게다가, 학습의 효율성 또한 기존의 학습 기반의 방법들에 비해 높은 결과를 보였다.This thesis proposes a reinforcement learning based collision avoidance method. The problem can be defined as an ability of a robot to reach its goal point without colliding with other robots and obstacles. There are two kinds of collision avoidance problem, single robot and multi-robot collision avoidance. Single robot collision avoidance problem contains multiple dynamic obstacles and one agent robot. The objective of the agent robot is to reach its goal point and avoid obstacles with random dynamics. Multi-robot collision avoidance problem contains multiple agent robots. It is also possible to include unknown dynamic obstacles to the problem. The agents should reach their own goal points without colliding with each other. If the environment contains unknown obstacles, the agents should avoid them also. To solve the problems, Collision Avoidance by Learning Collision (CALC) is proposed. CALC adopts the concept of reinforcement learning. The method is divided into two environments, training and planning. The training environment consists of one agent, one obstacle, and a training range. In the training environment, the agent learns how to collide with the obstacle and generates a colliding policy. In other words, when the agent collides with the obstacle, it receives positive reward. On the other hand, when the agent escapes the training range without collision, it receives negative reward. The planning environment contains multiple obstacles or robots and a single goal point. With the trained policy, the agent can solve the collision avoidance problem in the planning environment regardless of its dimension. Since the method learned collision, the generated policy should be inverted in the planning environment to avoid obstacles or robots. However, the policy should be applied directly for the goal point so that the agent can `collide' with the goal. With the combination of both policies, the agent can avoid the obstacles or robots and reach to the goal point simultaneously. In the training algorithm, the robot is assumed to be a holonomic robot. Even though the trained policy is generated from the holonomic robot, the method can be applied to both holonomic and non-holonomic robots by holonomic to non-holonomic converting method. CALC is applied to three problems, single holonomic robot, single non-holonomic robot, and multiple non-holonomic robot collision avoidance. The proposed method is validated both in the robot simulation and real-world experiment. For simulation, Robot Operating System (ROS) based simulator called Gazebo and simple game library PyGame are used. The method is tested with both holonomic and non-holonomic robots in the simulation experiment. For real-world planning experiment, non-holonomic mobile robot named e-puck is used. The learned policy from the simulation can be directly applied to the real-world robot without any calibration or retraining. The result shows that the proposed method outperforms the existing methods such as Reciprocal Velocity Obstacle (RVO), PrEference Appraisal Reinforcement Learning (PEARL), and Optimal Reciprocal Collision Avoidance (ORCA). In addition, it is shown that the proposed method is more efficient in terms of learning than existing learning-based method.1. Introduction 1 1.1 Motivations 1 1.2 Contributions 6 1.3 Organizations 7 2 Related Work 8 2.1 Reinforcement Learning 8 2.2 Classical Navigation Methods 11 2.3 Learning-Based Navigation Methods 13 3. Learning Collision 17 3.1 Introduction 17 3.2 Learning Collision 18 3.2.1 Markov Decision Process Setup 18 3.2.2 Training Algorithm 19 3.2.3 Experimental Results 22 4. Single Robot Collision Avoidance 25 4.1 Introduction 25 4.2 Holonomic Robot Obstacle Avoidance 26 4.2.1 Approach 26 4.2.2 Experimental Results 29 4.3 Non-Holonomic Robot Obstacle Avoidance 31 4.3.1 Approach 31 4.3.2 Experimental Results 33 5. Multi-Robot Collision Avoidance 36 5.1 Introduction 36 5.2 Approach 37 5.3 Experimental Results 40 5.3.1 Simulated Experiment 40 5.3.2 Real-World Experiment 44 5.3.3 Holonomic to Non-Holonomic Conversion Experiment 49 6. Conclusion 52 Bibliography 55 초록 62 감사의 글 64Maste

Testing Method for Multi-UAV Conflict Resolution Using Agent-Based Simulation and Multi-Objective Search

A new approach to testing multi-UAV conflict resolution algorithms is presented. The problem is formulated as a multi-objective search problem with two objectives: finding air traffic encounters that 1) are able to reveal faults in conflict resolution algorithms and 2) are likely to happen in the real world. The method uses agent-based simulation and multi-objective search to automatically find encounters satisfying these objectives. It describes pairwise encounters in three-dimensional space using a parameterized geometry representation, which allows encounters involving multiple UAVs to be generated by combining several pairwise encounters. The consequences of the encounters, given the conflict resolution algorithm, are explored using a fast-time agent-based simulator. To find encounters meeting the two objectives, a genetic algorithm approach is used. The method is applied to test ORCA-3D, a widely cited open-source multi-UAV conflict resolution algorithm, and the method’s performance is compared with a plausible random testing approach. The results show that the method can find the required encounters more efficiently than the random search. The identified safety incidents are then the starting points for understanding limitations of the conflict resolution algorithm

A control architecture and human interface for agile, reconfigurable micro aerial vehicle formations

This thesis considers the problem of controlling a group of micro aerial vehicles for agile maneuvering cooperatively, or distributively. We first introduce the background and motivation for micro aerial vehicles, especially for the popular multi-rotor aerial vehicle platform. Then, we discuss the dynamics of quadrotor helicopters. A quadrotor is a specific kind of multi-rotor aerial vehicle with a special property called differential flatness, which simplifies the algorithm of trajectory planning, such that, instead of planning a trajectory in a 12-dimensional state space and 4-dimensional input space, we only need to plan the trajectory in 4-dimensional, so called, flat output space, while the 12-dimensional state and 4-dimensional input can be recovered from a mapping called endogenous transformation. We propose a series of approaches to achieve agile maneuvering of a dynamic quadrotor formation, from controlling a single quadrotor in an artificial vector field, to controlling a group of quadrotors in a Virtual Rigid Body (VRB) framework, to balancing the effect between the human control and autonomy for collision avoidance, and to fast on-line distributed collision avoidance with Buffered Voronoi Cells (BVC). In the vector field method, we generate velocity, acceleration, jerk and snap fields, depending on the tasks, or the positions of obstacles, such that a single quadrotor can easily find its required state and input from the endogenous transformation in order to track the artificial vector field. Next, with a Virtual Rigid Body framework, we let a group of quadrotors follow a single control command while also keeping a required formation, or even reconfigure from one formation to another. The Virtual Rigid Body framework decouples the trajectory planning problem into two sub-problems. Then we consider the problem of collision avoidance of the quadrotor formation when it is meanwhile tele-operated by a single human operator. The autonomy with collision avoidance algorithm, based on the vector field methods for a single quadrotor, is an assistive portion of the quadrotor formation controller, such that the human operator can focus on his/her high-level tasks, leaving the low-level collision avoidance task be handled automatically. We also consider the full autonomy problem of quadrotor formations when reconfiguring from one formation to another by developing a fast, on-line distributed collision avoidance algorithm using Buffered Voronoi Cells (BVCs). Our BVC based collision avoidance algorithm only requires sensed relative position, rather than relative position and velocity, while the computational complexity is comparable to other methods like velocity obstacles. At last, we introduce our experimental quadrotor platform which is built from PixHawk flight controller and Odroid-XU4 single-board computer. The hardware and software architecture of this multiple-quadrotor platform is described in detail so that our platform can easily be adopted and extended with different purposes. Our conclusion remark and discussion of future work are also given in this thesi

Learning Multi-Agent Navigation from Human Crowd Data

The task of safely steering agents amidst static and dynamic obstacles has many applications in robotics, graphics, and traffic engineering. While decentralized solutions are essential for scalability and robustness, achieving globally efficient motions for the entire system of agents is equally important. In a traditional decentralized setting, each agent relies on an underlying local planning algorithm that takes as input a preferred velocity and the current state of the agent\u27s neighborhood and then computes a new velocity for the next time-step that is collision-free and as close as possible to the preferred one. Typically, each agent promotes a goal-oriented preferred velocity, which can result in myopic behaviors as actions that are locally optimal for one agent is not necessarily optimal for the global system of agents. In this thesis, we explore a human-inspired approach for efficient multi-agent navigation that allows each agent to intelligently adapt its preferred velocity based on feedback from the environment. Using supervised learning, we investigate different egocentric representations of the local conditions that the agents face and train various deep neural network architectures on extensive collections of human trajectory datasets to learn corresponding life-like velocities. During simulation, we use the learned velocities as high-level, preferred velocities signals passed as input to the underlying local planning algorithm of the agents. We evaluate our proposed framework using two state-of-the-art local methods, the ORCA method, and the PowerLaw method. Qualitative and quantitative results on a range of scenarios show that adapting the preferred velocity results in more time- and energy-efficient navigation policies, allowing agents to reach their destinations faster as compared to agents simulated with vanilla ORCA and PowerLaw