Combining Subgoal Graphs with Reinforcement Learning to Build a Rational Pathfinder

In this paper, we present a hierarchical path planning framework called SG-RL (subgoal graphs-reinforcement learning), to plan rational paths for agents maneuvering in continuous and uncertain environments. By "rational", we mean (1) efficient path planning to eliminate first-move lags; (2) collision-free and smooth for agents with kinematic constraints satisfied. SG-RL works in a two-level manner. At the first level, SG-RL uses a geometric path-planning method, i.e., Simple Subgoal Graphs (SSG), to efficiently find optimal abstract paths, also called subgoal sequences. At the second level, SG-RL uses an RL method, i.e., Least-Squares Policy Iteration (LSPI), to learn near-optimal motion-planning policies which can generate kinematically feasible and collision-free trajectories between adjacent subgoals. The first advantage of the proposed method is that SSG can solve the limitations of sparse reward and local minima trap for RL agents; thus, LSPI can be used to generate paths in complex environments. The second advantage is that, when the environment changes slightly (i.e., unexpected obstacles appearing), SG-RL does not need to reconstruct subgoal graphs and replan subgoal sequences using SSG, since LSPI can deal with uncertainties by exploiting its generalization ability to handle changes in environments. Simulation experiments in representative scenarios demonstrate that, compared with existing methods, SG-RL can work well on large-scale maps with relatively low action-switching frequencies and shorter path lengths, and SG-RL can deal with small changes in environments. We further demonstrate that the design of reward functions and the types of training environments are important factors for learning feasible policies.Comment: 20 page

Maximizing Seaweed Growth on Autonomous Farms: A Dynamic Programming Approach for Underactuated Systems Navigating on Uncertain Ocean Currents

Seaweed biomass offers significant potential for climate mitigation, but large-scale, autonomous open-ocean farms are required to fully exploit it. Such farms typically have low propulsion and are heavily influenced by ocean currents. We want to design a controller that maximizes seaweed growth over months by taking advantage of the non-linear time-varying ocean currents for reaching high-growth regions. The complex dynamics and underactuation make this challenging even when the currents are known. This is even harder when only short-term imperfect forecasts with increasing uncertainty are available. We propose a dynamic programming-based method to efficiently solve for the optimal growth value function when true currents are known. We additionally present three extensions when as in reality only forecasts are known: (1) our methods resulting value function can be used as feedback policy to obtain the growth-optimal control for all states and times, allowing closed-loop control equivalent to re-planning at every time step hence mitigating forecast errors, (2) a feedback policy for long-term optimal growth beyond forecast horizons using seasonal average current data as terminal reward, and (3) a discounted finite-time Dynamic Programming (DP) formulation to account for increasing ocean current estimate uncertainty. We evaluate our approach through 30-day simulations of floating seaweed farms in realistic Pacific Ocean current scenarios. Our method demonstrates an achievement of 95.8% of the best possible growth using only 5-day forecasts. This confirms the feasibility of using low-power propulsion and optimal control for enhanced seaweed growth on floating farms under real-world conditions.Comment: 8 pages, submitted to 2023 IEEE 62th Annual Conference on Decision and Control (CDC) Matthias Killer and Marius Wiggert contributed equally to this wor

통합형 무인 수상선-케이블-수중선 시스템의 다물체동역학 거동 및 제어

Underwater exploration is becoming more and more important, since a vast range of unknown resources in the deep ocean remain undeveloped. This dissertation thus presents a modeling of the coupled dynamics of an Unmanned Surface Vehicle (USV) system with an Underwater Vehicles (UV) connected by an underwater cable (UC). The complexity of this multi-body dynamics system and ocean environments is very difficult to model. First, for modeling this, dynamics analysis was performed on each subsystem and further total coupled system dynamics were studied. The UV which is towed by a UC is modeled with 6-DOF equations of motion that reflects its hydrodynamic characteristic was studied. The 4th-order Runge–Kutta numerical method was used to analyze the motion of the USV with its hydrodynamic coefficients which were obtained through experiments and from the literature. To analyze the effect of the UC, the complicated nonlinear and coupled UC dynamics under currents forces, the governing equations of the UC dynamics are established based on the catenary equation method, then it is solved by applying the shooting method. The new formulation and solution of the UC dynamics yields the three dimensional position and forces of the UC end point under the current forces. Also, the advantage of the proposed method is that the catenary equations using shooting method can be solved in real time such that the calculated position and forces of UC according to time can be directly utilized to calculate the UV motion. The proposed method offers advantages of simple formulation, convenient use, and fast calculation time with exact result. Some simple numerical simulations were conducted to observe the dynamic behaviors of AUV with cable effects. The simulations results clearly reveal that the UC can greatly influence the motions of the vehicles, especially on the UV motions. Based on both the numerical model and simulation results developed in the dissertation, we may offer some valuable information for the operation of the UV and USV. Secondly, for the design controller, a PD controller and its application to automatic berthing control of USV are also studied. For this, a nonlinear mathematical model for the maneuvering of USV in the presence of environmental forces was firstly established. Then, in order to control rudder and propeller during automatic berthing process, a PD control algorithm is applied. The algorithm consists of two parts, the forward velocity control and heading angle control. The control algorithm was designed based on the longitudinal and yaw dynamic models of USV. The desired heading angle was obtained by the so-called “Line of Sight” method. To support the validity of the proposed method, the computer simulations of automatic USV berthing are carried out. The results of simulation showed good performance of the developed berthing control system. Also, a hovering-type AUV equipped with multiple thrusters should maintain the specified position and orientation in order to perform given tasks by applying a dynamic positioning (DP) system. Besides, the control allocation algorithm based on a scaling factor is presented for distributing the forces required by the control law onto the available set of actuators in the most effective and energy efficient way. Thus, it is necessary for the robust control algorithm to conduct successfully given missions in spite of a model uncertainty and a disturbance. In this dissertation, the robust DP control algorithm based on a sliding mode theory is also addressed to guarantee the stability and better performance despite the model uncertainty and disturbance of current and cable effects. Finally, a series of simulations are conducted to verify the availability of the generated trajectories and performance of the designed robust controller. Thirdly, for the navigation of UV, a method for designing the path tracking controller using a Rapidly-exploring Random Trees (RRT) algorithm is proposed. The RRT algorithm is firstly used for the generation of collision free waypoints. Next, the unnecessary waypoints are removed by a simple path pruning algorithm generating a piecewise linear path. After that, a path smoothing algorithm utilizing cubic Bezier spiral curves to generate a continuous curvature path that satisfies the minimum radius of curvature constraint of underwater is implemented. The angle between two waypoints is the only information required for the generation of the continuous curvature path. In order to underwater vehicle follow the reference path, the path tracking controller using the global Sliding Mode Control (SMC) approach is designed. To verify the performance of the proposed algorithm, some simulation results are performed. Simulation results showed that the RRT algorithm could be applied to generate an optimal path in a complex ocean environment with multiple obstacles.Acknowledgement .................................................................................................. vi Abstract……. ....................................................................................... ………….viii Nomenclature ....................................................................................................... xvi List of Abbreviations ........................................................................................... xxi List of Tables ...................................................................................................... xxiii List of Figures ..................................................................................................... xxiv Chapter 1: Introduction ......................................................................................... 1 1.1 Background .................................................................................................. 1 1.1.1 Unmanned Surface Vehicles (USVs) ...................................................... 1 1.1.2 Umbilical Cable ....................................................................................... 4 1.1.3 Unmanned Underwater Vehicles (UUVs) ............................................... 5 1.1.4 Literature on Modeling of Marine Vehicles ............................................ 9 1.1.5 Literature on Control and Guidance of Marine Vehicles ...................... 11 1.2 Our System Architecture ........................................................................... 12 1.3 Motivation ................................................................................................. 13 1.4 Contribution ............................................................................................... 16 1.5 Publications Associated to the Dissertation .............................................. 17 1.6 Structure of the Dissertation ...................................................................... 18 Chapter 2: Mathematical Model of Unmanned Surface Vehicle (USV) ......... 20 2.1 Basic Assumptions .................................................................................... 20 2.2 Three Coordinate Systems ......................................................................... 20 2.3 Variable Notation ...................................................................................... 22 2.4 Kinematics ................................................................................................. 23 2.5 Kinetics ...................................................................................................... 26 2.5.1 Rigid Body Equations of Motion ........................................................... 26 2.5.2 Hydrodynamic Forces and Moments ..................................................... 28 2.5.3 Restoring Forces and Moments ............................................................. 31 2.5.4 Environmental Disturbances .................................................................. 32 2.5.5 Propulsion Forces and Moments ........................................................... 35 2.6 Nonlinear 6DOF Dynamics ....................................................................... 35 2.7 Mathematical Model of USV in 3 DOF .................................................... 36 2.7.1 Planar Kinematics .................................................................................. 36 2.7.2 Planar Nonlinear 3 DOF Dynamics ....................................................... 38 2.8 Configuration of Thrusters ........................................................................ 40 2.9 General Structure and Model Parameters .................................................. 41 2.9.1 Structure of USV ................................................................................... 41 2.9.2 Control System of USV ......................................................................... 42 2.9.3 Winch Control System ........................................................................... 43 Chapter 3: Mathematical Model of the Umbilical Cable (UC) ........................ 45 3.1 Basic Assumptions for UC ........................................................................ 45 3.2 Analysis on Forces of UV ......................................................................... 47 3.3 Relation for UC Equilibrium ..................................................................... 50 3.4 Catenary Equation in the Space Case ........................................................ 51 3.5 Shooting Method ....................................................................................... 55 3.6 Boundary Conditions ................................................................................. 57 3.7 Cable Effects ............................................................................................. 58 3.8 Model Parameters and Simulation ............................................................. 59 Chapter 4: Mathematical Model of Underwater Vehicle (UV) ........................ 63 4.1 Background ................................................................................................ 63 4.1.1 Basic Assumptions................................................................................. 63 4.1.2 Reference Frames .................................................................................. 64 4.1.3 Notations ................................................................................................ 65 4.2 Kinematics Equations ................................................................................ 66 4.3 Kinetic Equations ...................................................................................... 67 4.3.1 Rigid-Body Kinetics .............................................................................. 67 4.3.2 Hydrostatic Terms ................................................................................. 69 4.3.3 Hydrodynamic Terms ............................................................................ 70 4.3.4 Actuator Modeling ................................................................................. 75 4.3.5 Umbilical Cable Forces ......................................................................... 75 4.4 Nonlinear Equations of Motion (6DOF) ................................................... 76 4.5 Simplification of UV Dynamic Model ...................................................... 77 4.5.1 Simplifying the Mass and Inertia Matrix ............................................... 78 4.5.2 Simplifying the Hydrodynamic Damping Matrix.................................. 79 4.5.3 Simplifying the Gravitational and Buoyancy Vector ............................ 80 4.6 Thruster Modeling ..................................................................................... 80 4.7 Current Modeling ...................................................................................... 83 4.8 Dynamic Model Including Ocean Currents ............................................... 84 4.9 Complete Motion Equations of AUV (6DOF) .......................................... 89 4.10 Dynamics Model Parameter Identification ................................................ 91 4.11 Numerical Solution for Equations of Motion ............................................ 93 4.12 General Structure and Model Parameters .................................................. 94 4.12.1 Structure of AUV ............................................................................... 94 4.12.2 Control System of AUV ..................................................................... 96 Chapter 5: Guidance Theory ............................................................................... 97 5.1 Configuration of GNC System .................................................................. 97 5.1.1 Guidance ................................................................................................ 98 5.1.2 Navigation .............................................................................................. 98 5.1.3 Control ................................................................................................... 98 5.2 Maneuvering Problem Statement .............................................................. 99 5.3 Guidance Objectives ................................................................................ 100 5.3.1 Target Tracking ................................................................................... 100 5.3.2 Trajectory Tracking ............................................................................. 100 5.4 Waypoint Representation ........................................................................ 101 5.5 Path Following ......................................................................................... 102 5.6 Line of Sight (LOS) Waypoint Guidance ................................................ 102 5.6.1 Enclosure-Based Steering .................................................................... 104 5.6.2 Look-ahead Based Steering ................................................................. 105 5.6.3 LOS Control......................................................................................... 106 5.7 Cubic Polynomial for Path-Following ..................................................... 107 Chapter 6: Control Algorithm Design and Analysis ....................................... 110 6.1 Proportional Integral Differential (PID) Controller ................................ 110 6.1.1 General Theory .................................................................................... 110 6.1.2 Stability of General PID Controller ..................................................... 112 6.1.3 PID Tuning .......................................................................................... 114 6.1.4 Nonlinear PID for Marine Vehicles ..................................................... 116 6.1.5 Nonlinear PD for Marine Vehicles ...................................................... 117 6.1.6 Stability of Designed PD Controller .................................................... 117 6.2 Sliding Mode Controller .......................................................................... 118 6.2.1 Tracking Error and Sliding Surface ..................................................... 119 6.2.2 Chattering Situation ............................................................................. 120 6.2.3 Control Law and Stability .................................................................... 121 6.3 Allocation Control ................................................................................... 124 6.3.1 Linear Quadratic Unconstrained Control Allocation Using Lagrange Multipliers ................................................................................................ 125 6.3.2 Thruster Allocation with a Constrained Linear Model ........................ 127 6.4 Simulation Results and Discussion ......................................................... 131 6.4.1 Berthing (parking) Control of USV ..................................................... 133 6.4.2 Motion Control of UV ......................................................................... 136 Chapter 7: Obstacle Avoidance and Path Planning for Vehicle Using Rapidly-Exploring Random Trees Algorithm.................................................................. 168 7.1 Path Planning and Guidance: Two Interrelated Problems ....................... 168 7.2 RRT Algorithm for Exploration .............................................................. 171 7.2.1 Random Node Selection ...................................................................... 172 7.2.2 Nearest Neighbor Node Selection ....................................................... 173 7.2.3 RRT Exploration with Obstacles ......................................................... 174 7.3 RRT Algorithm for Navigation of AUV ................................................. 176 7.3.1 Basic RRT Algorithm .......................................................................... 176 7.3.2 Biased-Greedy RRT Algorithm ........................................................... 178 7.3.3 Synchronized Biased-Greedy RRT Algorithm .................................... 179 7.4 Path Pruning ............................................................................................ 182 7.4.1 Path Pruning Using LOS ..................................................................... 182 7.4.2 Global Path Pruning ............................................................................. 183 7.5 Summarize the Proposed RRT Algorithm ............................................... 185 7.6 Simulation for Path Following of AUV .................................................. 187 Chapter 8: Simulation of Complete USV-UC-UV Systems ............................ 196 8.1 Simulation Procedure .............................................................................. 196 8.2 Simulation Results and Discussion ......................................................... 201 8.2.1 Dynamic Behaviors of Complete USV (Stable)-Cable- AUV (Turning Motion) ..................................................................................................... 201 8.2.2 Dynamic Behaviors of Complete USV (Forward motion)-Cable- AUV (Turning Motion) ...................................................................................... 207 8.2.3 Applied Controller to Complete USV -Cable- AUV ........................... 215 Chapter 9: Conclusions and Future Works ..................................................... 238 9.1 Modeling of Complete USV-Cable-AUV System .................................. 238 9.2 Motion Control ........................................................................................ 239 9.3 Cable Force and Moment at the Tow Points ........................................... 239 9.4 Path Planning ........................................................................................... 239 9.5 Future Works ........................................................................................... 240Docto

Adaptive low-level control of autonomous underwater vehicles using deep reinforcement learning

Low-level control of autonomous underwater vehicles (AUVs) has been extensively addressed by classical control techniques. However, the variable operating conditions and hostile environments faced by AUVs have driven researchers towards the formulation of adaptive control approaches. The reinforcement learning (RL) paradigm is a powerful framework which has been applied in different formulations of adaptive control strategies for AUVs. However, the limitations of RL approaches have lead towards the emergence of deep reinforcement learning which has become an attractive and promising framework for developing real adaptive control strategies to solve complex control problems for autonomous systems. However, most of the existing applications of deep RL use video images to train the decision making artificial agent but obtaining camera images only for an AUV control purpose could be costly in terms of energy consumption. Moreover, the rewards are not easily obtained directly from the video frames. In this work we develop a deep RL framework for adaptive control applications of AUVs based on an actor-critic goal-oriented deep RL architecture, which takes the available raw sensory information as input and as output the continuous control actions which are the low-level commands for the AUV's thrusters. Experiments on a real AUV demonstrate the applicability of the stated deep RL approach for an autonomous robot control problem.Fil: Carlucho, Ignacio. Universidad Nacional del Centro de la Provincia de Buenos Aires. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires. - Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Tandil. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires. - Provincia de Buenos Aires. Gobernación. Comisión de Investigaciones Científicas. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires; ArgentinaFil: de Paula, Mariano. Universidad Nacional del Centro de la Provincia de Buenos Aires. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires. - Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Tandil. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires. - Provincia de Buenos Aires. Gobernación. Comisión de Investigaciones Científicas. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires; ArgentinaFil: Wang, Sen. Heriot-Watt University; Reino UnidoFil: Petillot, Yvan. Heriot-Watt University; Reino UnidoFil: Acosta, Gerardo Gabriel. Universidad Nacional del Centro de la Provincia de Buenos Aires. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires. - Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Tandil. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires. - Provincia de Buenos Aires. Gobernación. Comisión de Investigaciones Científicas. Centro de Investigaciones en Física e Ingeniería del Centro de la Provincia de Buenos Aires; Argentin

Robotics 2010

Without a doubt, robotics has made an incredible progress over the last decades. The vision of developing, designing and creating technical systems that help humans to achieve hard and complex tasks, has intelligently led to an incredible variety of solutions. There are barely technical fields that could exhibit more interdisciplinary interconnections like robotics. This fact is generated by highly complex challenges imposed by robotic systems, especially the requirement on intelligent and autonomous operation. This book tries to give an insight into the evolutionary process that takes place in robotics. It provides articles covering a wide range of this exciting area. The progress of technical challenges and concepts may illuminate the relationship between developments that seem to be completely different at first sight. The robotics remains an exciting scientific and engineering field. The community looks optimistically ahead and also looks forward for the future challenges and new development

해양 작업 지원선의 자율 운항 및 설치 작업 지원을 위한 시뮬레이션 방법

학위논문 (박사)-- 서울대학교 대학원 : 공과대학 조선해양공학과, 2019. 2. 노명일.Autonomous ships have gained a huge amount of interest in recent years, like their counterparts on land{autonomous cars, because of their potential to significantly lower the cost of operation, attract seagoing professionals and increase transportation safety. Technologies developed for the autonomous ships have potential to notably reduce maritime accidents where 75% cases can be attributed to human error and a significant proportion of these are caused by fatigue and attention deficit. However, developing a high-level autonomous system which can operate in an unstructured and unpredictable environment is still a challenging task. When the autonomous ships are operating in the congested waterway with other manned or unmanned vessels, the collision avoidance algorithm is the crucial point in keeping the safety of both the own ship and any encountered ships. Instead of developing new traffic rules for the autonomous ships to avoid collisions with each other, autonomous ships are expected to follow the existing guidelines based on the International Regulations for Preventing Collisions at Sea (COLREGs). Furthermore, when using the crane on the autonomous ship to transfer and install subsea equipment to the seabed, the heave and swaying phenomenon of the subsea equipment at the end of flexible wire ropes makes its positioning at an exact position is very difficult. As a result, an Anti-Motion Control (AMC) system for the crane is necessary to ensure the successful installation operation. The autonomous ship is highly relying on the effectiveness of autonomous systems such as autonomous path following system, collision avoidance system, crane control system and so on. During the previous two decades, considerable attention has been paid to develop robust autonomous systems. However, several are facing challenges and it is worthwhile devoting much effort to this. First of all, the development and testing of the proposed control algorithms should be adapted across a variety of environmental conditions including wave, wind, and current. This is one of the challenges of this work aimed at creating an autonomous path following and collision avoidance system in the ship. Secondly, the collision avoidance system has to comply with the regulations and rules in developing an autonomous ship. Thirdly, AMC system with anti-sway abilities for a knuckle boom crane remains problems regarding its under-actuated mechanism. At last, the performance of the control system should be evaluated in advance of the operation to perform its function successfully. In particular, such performance analysis is often very costly and time-consuming, and realistic conditions are typically impossible to establish in a testing environment. Consequently, to address these issues, we proposed a simulation framework with the following scenarios, which including the autonomous navigation scenario and crane operation scenario. The research object of this study is an autonomous offshore support vessel (OSV), which provides support services to offshore oil and gas field development such as offshore drilling, pipe laying, and oil producing assets (production platforms and FPSOs) utilized in EP (Exploration Production) activities. Assume that the autonomous OSV confronts an urgent mission under the harsh environmental conditions: on the way to an imperative offshore construction site, the autonomous OSV has to avoid target ships while following a predefined path. When arriving at the construction site, it starts to install a piece of subsea equipment on the seabed. So what technologies are needed, what should be invested for ensuring the autonomous OSV could robustly kilometers from shore, and how can an autonomous OSV be made at least as safe as the conventional ship. In this dissertation, we focus on the above critical activities for answering the above questions. In the general context of the autonomous navigation and crane control problem, the objective of this dissertation is thus fivefold: • Developing a COLREGs-compliant collision avoidance system. • Building a robust path following and collision avoidance system which can handle the unknown and complicated environment. • Investigating an efficient multi-ship collision avoidance method enable it easy to extend. • Proposing a hardware-in-the-loop simulation environment for the AHC system. • Solving the anti-sway problem of the knuckle boom crane on an autonomous OSV. First of all, we propose a novel deep reinforcement learning (RL) algorithm to achieve effective and efficient capabilities of the path following and collision avoidance system. To perform and verify the proposed algorithm, we conducted simulations for an autonomous ship under unknown environmental disturbance iiito adjust its heading in real-time. A three-degree-of-freedom dynamic model of the autonomous ship was developed, and the Line-of-sight (LOS) guidance system was used to converge the autonomous ship to follow the predefined path. Then, a proximal policy optimization (PPO) algorithm was implemented on the problem. By applying the advanced deep RL method, in which the autonomous OSV learns the best behavior through repeated trials to determine a safe and economical avoidance behavior in various circumstances. The simulation results showed that the proposed algorithm has the capabilities to guarantee collision avoidance of moving encountered ships while ensuring following a predefined path. Also, the algorithm demonstrated that it could manage complex scenarios with various encountered ships in compliance with COLREGs and have the excellent adaptability to the unknown, sophisticated environment. Next, the AMC system includes Anti-Heave Control (AHC) and Anti-Sway Control (ASC), which is applied to install subsea equipment in regular and irregular for performance analysis. We used the proportional-integral-derivative (PID) control method and the sliding mode control method respectively to achieve the control objective. The simulation results show that heave and sway motion could be significantly reduced by the proposed control methods during the construction. Moreover, to evaluate the proposed control system, we have constructed the HILS environment for the AHC system, then conducted a performance analysis of it. The simulation results show the AHC system could be evaluated effectively within the HILS environment. We can conclude that the proposed or adopted methods solve the problems issued in autonomous system design.해양 작업 지원선 (Offshore Support Vessel: OSV)의 경우 극한의 환경에도 불구하고 출항하여 해상에서 작업을 수행해야 하는 경우가 있다. 이러한 위험에의 노출을 최소화하기 위해 자율 운항에 대한 요구가 증가하고 있다. 여기서의 자율 운항은 선박이 출발지에서 목적지까지 사람의 도움 없이 이동함을 의미한다. 자율 운항 방법은 경로 추종 방법과 충돌 회피 방법을 포함한다. 우선, 운항 및 작업 중 환경 하중 (바람, 파도, 조류 등)에 대한 고려를 해야 하고, 국제 해상 충돌 예방 규칙 (Convention of the International Regulations for Preventing Collisions at Sea, COLREGs)에 의한 선박간의 항법 규정을 고려하여 충돌 회피 규칙을 준수해야 한다. 특히 연근해의 복잡한 해역에서는 많은 선박을 자동으로 회피할 필요가 있다. 기존의 해석적인 방법을 사용하기 위해서는 선박들에 대한 정확한 시스템 모델링이 되어야 하며, 그 과정에서 경험 (experience)에 의존하는 파라미터 튜닝이 필수적이다. 또한, 회피해야 할 선박 수가 많아질 경우 시스템 모델이 커지게 되고 계산 양과 계산 시간이 늘어나 실시간 적용이 어렵다는 단점이 있다. 또한, 경로 추종 및 충돌 회피를 포함하여 자율 운항 방법을 적용하기가 어렵다. 따라서 본 연구에서는 강화 학습 (Reinforcement Learning: RL) 기법을 이용하여 기존 해석적인 방법의 문제점을 극복할 수 있는 방법을 제안하였다. 경로를 추종하는 선박 (agent)은 외부 환경 (environment)과 상호작용하면서 학습을 진행한다. State S_0 (선박의 움직임과 관련된 각종 상태) 가지는 agent는 policy (현재 위치에서 어떤 움직임을 선택할 것인가)에 따라 action A_0 (움직일 방향) 취한다. 이에 environment는 agent의 다음 state S_1 을 계산하고, 그에 따른 보상 R_0 (해당 움직임의 적합성)을 결정하여 agent에게 전달한다. 이러한 작업을 반복하면서 보상이 최대가 되도록 policy를 학습하게 된다. 한편, 해상에서 크레인을 이용한 장비의 이동이나 설치 작업 시 위험을 줄이기 위해 크레인의 거동 제어에 대한 요구가 증가하고 있다. 특히 해상에서는 선박의 운동에 의해 크레인에 매달린 물체가 상하 동요 (heave)와 크레인을 기준으로 좌우 동요 (sway)가 발생하는데, 이러한 운동은 작업을 지연시키고, 정확한 위치에 물체를 놓지 못하게 하며, 자칫 주변 구조물과의 충돌을 야기할 수 있다. 이와 같은 동요를 최소화하는 Anti-Motion Control (AMC) 시스템은 Anti-Heave Control (AHC)과 Anti-Sway Control (ASC)을 포함한다. 본 연구에서는 해양 작업 지원선에 적합한 AMC 시스템의 설계 및 검증 방법을 연구하였다. 먼저 상하 동요를 최소화하기 위해 크레인의 와이어 길이를 능동적으로 조정하는 AHC 시스템을 설계하였다. 또한, 기존의 제어 시스템의 검증 방법은 실제 선박이나 해양 구조물에 해당 제어 시스템을 직접 설치하기 전에는 그 성능을 테스트하기가 힘들었다. 이를 해결하기 위해 본 연구에서는 Hardware-In-the-Loop Simulation (HILS) 기법을 활용하여 AHC 시스템의 검증 방법을 연구하였다. 또한, ASC 시스템을 설계할 때 제어 대상이 under-actuated 시스템이기 때문에 제어하기가 매우 어렵다. 따라서 본 연구에서는 sliding mode control 알고리즘을 이용하며 다관절 크레인 (knuckle boom crane)의 관절 (joint) 각도를 제어하여 좌우 동요를 줄일 수 있는 ASC 시스템을 설계하였다.Chapter 1 Introduction 1 1.1 Background and Motivation . . . . . . . . . . . . . . . . . . . . . 1 1.2 Requirements for Autonomous Operation . . . . . . . . . . . . . 5 1.2.1 Path Following for Autonomous Ship . . . . . . . . . . . . 5 1.2.2 Collision Avoidance for Autonomous Ship . . . . . . . . . 5 1.2.3 Anti-Motion Control System for Autonomous Ship . . . . 6 1.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3.1 Related Work for Path Following System . . . . . . . . . 9 1.3.2 Related Work for Collision Avoidance System . . . . . . . 9 1.3.3 Related Work for Anti-Heave Control System . . . . . . . 13 1.3.4 Related Work for Anti-Sway Control System . . . . . . . 14 1.4 Configuration of Simulation Framework . . . . . . . . . . . . . . 16 1.4.1 Application Layer . . . . . . . . . . . . . . . . . . . . . . 16 1.4.2 Autonomous Ship Design Layer . . . . . . . . . . . . . . . 17 1.4.3 General Technique Layer . . . . . . . . . . . . . . . . . . 17 1.5 Contributions (Originality) . . . . . . . . . . . . . . . . . . . . . 19 Chapter 2 Theoretical Backgrounds 20 2.1 Maneuvering Model for Autonomous Ship . . . . . . . . . . . . . 20 2.1.1 Kinematic Equation for Autonomous Ship . . . . . . . . . 20 2.1.2 Kinetic Equation for Autonomous Ship . . . . . . . . . . 21 2.2 Multibody Dynamics Model for Knuckle Boom Crane of Autonomous Ship. . . 25 2.2.1 Embedding Techniques . . . . . . . . . . . . . . . . . . . . 25 2.3 Control System Design . . . . . . . . . . . . . . . . . . . . . . . . 31 2.3.1 Proportional-Integral-Derivative (PID) Control . . . . . . 31 2.3.2 Sliding Mode Control . . . . . . . . . . . . . . . . . . . . 31 2.4 Deep Reinforcement Learning Algorithm . . . . . . . . . . . . . . 34 2.4.1 Value Based Learning Method . . . . . . . . . . . . . . . 36 2.4.2 Policy Based Learning Method . . . . . . . . . . . . . . . 37 2.4.3 Actor-Critic Method . . . . . . . . . . . . . . . . . . . . . 41 2.5 Hardware-in-the-Loop Simulation . . . . . . . . . . . . . . . . . . 43 2.5.1 Integrated Simulation Method . . . . . . . . . . . . . . . 43 Chapter 3 Path Following Method for Autonomous OSV 46 3.1 Guidance System . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.1.1 Line-of-sight Guidance System . . . . . . . . . . . . . . . 46 3.2 Deep Reinforcement Learning for Path Following System . . . . . 50 3.2.1 Deep Reinforcement Learning Setup . . . . . . . . . . . . 50 3.2.2 Neural Network Architecture . . . . . . . . . . . . . . . . 56 3.2.3 Training Process . . . . . . . . . . . . . . . . . . . . . . . 58 3.3 Implementation and Simulation Result . . . . . . . . . . . . . . . 62 3.3.1 Implementation for Path Following System . . . . . . . . 62 3.3.2 Simulation Result . . . . . . . . . . . . . . . . . . . . . . 65 3.4 Comparison Results . . . . . . . . . . . . . . . . . . . . . . . . . 83 3.4.1 Comparison Result of PPO with PID . . . . . . . . . . . 83 3.4.2 Comparison Result of PPO with Deep Q-Network (DQN) 87 Chapter 4 Collision Avoidance Method for Autonomous OSV 89 4.1 Deep Reinforcement Learning for Collision Avoidance System . . 89 4.1.1 Deep Reinforcement Learning Setup . . . . . . . . . . . . 89 4.1.2 Neural Network Architecture . . . . . . . . . . . . . . . . 93 4.1.3 Training Process . . . . . . . . . . . . . . . . . . . . . . . 94 4.2 Implementation and Simulation Result . . . . . . . . . . . . . . . 95 4.2.1 Implementation for Collision Avoidance System . . . . . . 95 4.2.2 Simulation Result . . . . . . . . . . . . . . . . . . . . . . 100 4.3 Implementation and Simulation Result for Multi-ship Collision Avoidance Method . . . . . . . . . . . . . . . . . . . . . . . . . . 107 4.3.1 Limitations of Multi-ship Collision Avoidance Method - 1 107 4.3.2 Limitations of Multi-ship Collision Avoidance Method - 2 108 4.3.3 Implementation of Multi-ship Collision Avoidance Method 110 4.3.4 Simulation Result of Multi-ship Collision Avoidance Method 118 Chapter 5 Anti-Motion Control Method for Knuckle Boom Crane 129 5.1 Configuration of HILS for Anti-Heave Control System . . . . . . 129 5.1.1 Virtual Mechanical System . . . . . . . . . . . . . . . . . 132 5.1.2 Virtual Sensor and Actuator . . . . . . . . . . . . . . . . 138 5.1.3 Control System Design . . . . . . . . . . . . . . . . . . . . 141 5.1.4 Integrated Simulation Interface . . . . . . . . . . . . . . . 142 5.2 Implementation and Simulation Result of HILS for Anti-Heave Control System . . . . . . . . 145 5.2.1 Implementation of HILS for Anti-Heave Control System . 145 5.2.2 Simulation Result of HILS for Anti-Heave Control System 146 5.3 Validation of HILS for Anti-Heave Control System . . . . . . . . 159 5.3.1 Hardware Setup . . . . . . . . . . . . . . . . . . . . . . . 159 5.3.2 Comparison Result . . . . . . . . . . . . . . . . . . . . . . 161 5.4 Configuration of Anti-Sway Control System . . . . . . . . . . . . 162 5.4.1 Mechanical System for Knuckle Boom Crane . . . . . . . 162 5.4.2 Anti-Sway Control System Design . . . . . . . . . . . . . 165 5.4.3 Implementation and Simulation Result of Anti-Sway Control . . . . . . . . . . . . . . 168 Chapter 6 Conclusions and Future Works 176 Bibliography 178 Chapter A Appendix 186 국문초록 188Docto