Controlling a cargo ship without human experience based on deep Q-network

Human experience is regarded as an indispensable part of artificial intelligence in the process of controlling or decision making for autonomous cargo ships. In this paper, a novel Deep Q-Network-based (DQN) approach is proposed, which performs satisfactorily in controlling a cargo ship automatically without any human experience. At the very beginning, we use the model of KRISO Very Large Crude Carrier (KVLCC2) to describe a cargo ship. To manipulate this ship has to conquer great inertia and relatively insufficient driving force. Subsequently, customary waterways, regulations, conventions are described with Artificial Potential Field and value-functions in DQN. Based on this, the artificial intelligence of planning and controlling a cargo ship can be obtained by undertaking sufficient training, which can control the ship directly, while avoiding collisions, keeping its position in the middle of the route as much as possible. In simulation experiments, it is demonstrated that such an approach performs better than manual works and other traditional methods in most conditions, which makes the proposed method a promising solution in improving the autonomy level of cargo ships

COLREG-Compliant Collision Avoidance for Unmanned Surface Vehicle using Deep Reinforcement Learning

Path Following and Collision Avoidance, be it for unmanned surface vessels or other autonomous vehicles, are two fundamental guidance problems in robotics. For many decades, they have been subject to academic study, leading to a vast number of proposed approaches. However, they have mostly been treated as separate problems, and have typically relied on non-linear first-principles models with parameters that can only be determined experimentally. The rise of Deep Reinforcement Learning (DRL) in recent years suggests an alternative approach: end-to-end learning of the optimal guidance policy from scratch by means of a trial-and-error based approach. In this article, we explore the potential of Proximal Policy Optimization (PPO), a DRL algorithm with demonstrated state-of-the-art performance on Continuous Control tasks, when applied to the dual-objective problem of controlling an underactuated Autonomous Surface Vehicle in a COLREGs compliant manner such that it follows an a priori known desired path while avoiding collisions with other vessels along the way. Based on high-fidelity elevation and AIS tracking data from the Trondheim Fjord, an inlet of the Norwegian sea, we evaluate the trained agent's performance in challenging, dynamic real-world scenarios where the ultimate success of the agent rests upon its ability to navigate non-uniform marine terrain while handling challenging, but realistic vessel encounters

Experimental Validation Of An Integrated Guidance And Control System For Marine Surface Vessels

Autonomous operation of marine surface vessels is vital for minimizing human errors and providing efficient operations of ships under varying sea states and environmental conditions which is complicated by the highly nonlinear dynamics of marine surface vessels. To deal with modelling imprecision and unpredictable disturbances, the sliding mode methodology has been employed to devise a heading and a surge displacement controller. The implementation of such a controller necessitates the availability of all state variables of the vessel. However, the measured signals in the current study are limited to the global X and Y positioning coordinates of the boat that are generated by a GPS system. Thus, a nonlinear observer, based on the sliding mode methodology, has been implemented to yield accurate estimates of the state variables in the presence of both structured and unstructured uncertainties. Successful autonomous operation of a marine surface vessel requires a holistic approach encompassing a navigation system, robust nonlinear controllers and observers. Since the overwhelming majority of the experimental work on autonomous marine surface vessels was not conducted in truly uncontrolled real-world environments. The first goal of this work was to experimentally validate a fully-integrated LOS guidance system with a sliding mode controller and observer using a 16’ Tracker Pro Guide V-16 aluminium boat with a 60 hp. Mercury outboard motor operating in the uncontrolled open-water environment of Lake St. Clair, Michigan. The fully integrated guidance and controller-observer system was tested in a model-less configuration, whereby all information provided from the vessel’s nominal model have been ignored. The experimental data serves to demonstrate the robustness and good tracking characteristics of the fully-integrated guidance and controller/observer system by overcoming the large errors induced at the beginning of each segment and converging the boat to the desired trajectory in spite of the presence of environmental disturbances. The second focus of this work was to combine a collision avoidance method with the guidance system that accounted for “International Regulations for Prevention of Collisions at Sea” abbreviated as COLREGS. This new system then needed to be added into the existing architecture. The velocity obstacles method was selected as the base to build upon and additional restrictions were incorporated to account for these additional rules. This completed system was then validated with a software in the loop simulation

A Survey of Recent Machine Learning Solutions for Ship Collision Avoidance and Mission Planning

Machine Learning (ML) techniques have gained significant traction as a means of improving the autonomy of marine vehicles over the last few years. This article surveys the recent ML approaches utilised for ship collision avoidance (COLAV) and mission planning. Following an overview of the ever-expanding ML exploitation for maritime vehicles, key topics in the mission planning of ships are outlined. Notable papers with direct and indirect applications to the COLAV subject are technically reviewed and compared. Critiques, challenges, and future directions are also identified. The outcome clearly demonstrates the thriving research in this field, even though commercial marine ships incorporating machine intelligence able to perform autonomously under all operating conditions are still a long way off

A Survey of Recent Machine Learning Solutions for Ship Collision Avoidance and Mission Planning

Machine Learning (ML) techniques have gained significant traction as a means of improving the autonomy of marine vehicles over the last few years. This article surveys the recent ML approaches utilised for ship collision avoidance (COLAV) and mission planning. Following an overview of the ever-expanding ML exploitation for maritime vehicles, key topics in the mission planning of ships are outlined. Notable papers with direct and indirect applications to the COLAV subject are technically reviewed and compared. Critiques, challenges, and future directions are also identified. The outcome clearly demonstrates the thriving research in this field, even though commercial marine ships incorporating machine intelligence able to perform autonomously under all operating conditions are still a long way off.Peer reviewe

Collision Avoidance for Autonomous Surface Vessels using Novel Artificial Potential Fields

As the demand for transportation through waterways continues to rise, the number of vessels plying the waters has correspondingly increased. This has resulted in a greater number of accidents and collisions between ships, some of which lead to significant loss of life and financial losses. Research has shown that human error is a major factor responsible for such incidents. The maritime industry is constantly exploring newer approaches to autonomy to mitigate this issue. This study presents the use of novel Artificial Potential Fields (APFs) to perform obstacle and collision avoidance in marine environments. This study highlights the advantage of harmonic functions over traditional functions in modeling potential fields. With a modification, the method is extended to effectively avoid dynamic obstacles while adhering to COLREGs. Improved performance is observed as compared to the traditional potential fields and also against the popular velocity obstacle approach. A comprehensive statistical analysis is also performed through Monte Carlo simulations in different congested environments that emulate real traffic conditions to demonstrate robustness of the approach.Comment: 28 pages, 30 figure

aColor: Mechatronics, Machine Learning, and Communications in an Unmanned Surface Vehicle

The aim of this work is to offer an overview of the research questions, solutions, and challenges faced by the project aColor ("Autonomous and Collaborative Offshore Robotics"). This initiative incorporates three different research areas, namely, mechatronics, machine learning, and communications. It is implemented in an autonomous offshore multicomponent robotic system having an Unmanned Surface Vehicle (USV) as its main subsystem. Our results across the three areas of work are systematically outlined in this paper by demonstrating the advantages and capabilities of the proposed system for different Guidance, Navigation, and Control missions, as well as for the high-speed and long-range bidirectional connectivity purposes across all autonomous subsystems. Challenges for the future are also identified by this study, thus offering an outline for the next steps of the aColor project.Comment: Paper was originally submitted to and presented in the 8th Transport Research Arena TRA 2020, April 27-30, 2020, Helsinki, Finlan

Robust trajectory tracking control for unmanned surface vessels under motion constraints and environmental disturbances

To achieve a fully autonomous navigation for unmanned surface vessels (USVs), a robust control capability is essential. The control of USVs in complex maritime environments is rather challenging as numerous system uncertainties and environmental influences affect the control performance. This paper therefore investigates the trajectory tracking control problem for USVs with motion constraints and environmental disturbances. Two different controllers are proposed to achieve the task. The first approach is mainly based on the backstepping technique augmented by a virtual system to compensate for the disturbance and an auxiliary system to bound the input in the saturation limit. The second control scheme is mainly based on the normalisation technique, with which the bound of the input can be limited in the constraints by tuning the control parameters. The stability of the two control schemes is demonstrated by the Lyapunov theory. Finally, simulations are conducted to verify the effectiveness of the proposed controllers. The introduced solutions enable USVs to follow complex trajectories in an adverse environment with varying ocean currents

해양 작업 지원선의 자율 운항 및 설치 작업 지원을 위한 시뮬레이션 방법

학위논문 (박사)-- 서울대학교 대학원 : 공과대학 조선해양공학과, 2019. 2. 노명일.Autonomous ships have gained a huge amount of interest in recent years, like their counterparts on land{autonomous cars, because of their potential to significantly lower the cost of operation, attract seagoing professionals and increase transportation safety. Technologies developed for the autonomous ships have potential to notably reduce maritime accidents where 75% cases can be attributed to human error and a significant proportion of these are caused by fatigue and attention deficit. However, developing a high-level autonomous system which can operate in an unstructured and unpredictable environment is still a challenging task. When the autonomous ships are operating in the congested waterway with other manned or unmanned vessels, the collision avoidance algorithm is the crucial point in keeping the safety of both the own ship and any encountered ships. Instead of developing new traffic rules for the autonomous ships to avoid collisions with each other, autonomous ships are expected to follow the existing guidelines based on the International Regulations for Preventing Collisions at Sea (COLREGs). Furthermore, when using the crane on the autonomous ship to transfer and install subsea equipment to the seabed, the heave and swaying phenomenon of the subsea equipment at the end of flexible wire ropes makes its positioning at an exact position is very difficult. As a result, an Anti-Motion Control (AMC) system for the crane is necessary to ensure the successful installation operation. The autonomous ship is highly relying on the effectiveness of autonomous systems such as autonomous path following system, collision avoidance system, crane control system and so on. During the previous two decades, considerable attention has been paid to develop robust autonomous systems. However, several are facing challenges and it is worthwhile devoting much effort to this. First of all, the development and testing of the proposed control algorithms should be adapted across a variety of environmental conditions including wave, wind, and current. This is one of the challenges of this work aimed at creating an autonomous path following and collision avoidance system in the ship. Secondly, the collision avoidance system has to comply with the regulations and rules in developing an autonomous ship. Thirdly, AMC system with anti-sway abilities for a knuckle boom crane remains problems regarding its under-actuated mechanism. At last, the performance of the control system should be evaluated in advance of the operation to perform its function successfully. In particular, such performance analysis is often very costly and time-consuming, and realistic conditions are typically impossible to establish in a testing environment. Consequently, to address these issues, we proposed a simulation framework with the following scenarios, which including the autonomous navigation scenario and crane operation scenario. The research object of this study is an autonomous offshore support vessel (OSV), which provides support services to offshore oil and gas field development such as offshore drilling, pipe laying, and oil producing assets (production platforms and FPSOs) utilized in EP (Exploration Production) activities. Assume that the autonomous OSV confronts an urgent mission under the harsh environmental conditions: on the way to an imperative offshore construction site, the autonomous OSV has to avoid target ships while following a predefined path. When arriving at the construction site, it starts to install a piece of subsea equipment on the seabed. So what technologies are needed, what should be invested for ensuring the autonomous OSV could robustly kilometers from shore, and how can an autonomous OSV be made at least as safe as the conventional ship. In this dissertation, we focus on the above critical activities for answering the above questions. In the general context of the autonomous navigation and crane control problem, the objective of this dissertation is thus fivefold: • Developing a COLREGs-compliant collision avoidance system. • Building a robust path following and collision avoidance system which can handle the unknown and complicated environment. • Investigating an efficient multi-ship collision avoidance method enable it easy to extend. • Proposing a hardware-in-the-loop simulation environment for the AHC system. • Solving the anti-sway problem of the knuckle boom crane on an autonomous OSV. First of all, we propose a novel deep reinforcement learning (RL) algorithm to achieve effective and efficient capabilities of the path following and collision avoidance system. To perform and verify the proposed algorithm, we conducted simulations for an autonomous ship under unknown environmental disturbance iiito adjust its heading in real-time. A three-degree-of-freedom dynamic model of the autonomous ship was developed, and the Line-of-sight (LOS) guidance system was used to converge the autonomous ship to follow the predefined path. Then, a proximal policy optimization (PPO) algorithm was implemented on the problem. By applying the advanced deep RL method, in which the autonomous OSV learns the best behavior through repeated trials to determine a safe and economical avoidance behavior in various circumstances. The simulation results showed that the proposed algorithm has the capabilities to guarantee collision avoidance of moving encountered ships while ensuring following a predefined path. Also, the algorithm demonstrated that it could manage complex scenarios with various encountered ships in compliance with COLREGs and have the excellent adaptability to the unknown, sophisticated environment. Next, the AMC system includes Anti-Heave Control (AHC) and Anti-Sway Control (ASC), which is applied to install subsea equipment in regular and irregular for performance analysis. We used the proportional-integral-derivative (PID) control method and the sliding mode control method respectively to achieve the control objective. The simulation results show that heave and sway motion could be significantly reduced by the proposed control methods during the construction. Moreover, to evaluate the proposed control system, we have constructed the HILS environment for the AHC system, then conducted a performance analysis of it. The simulation results show the AHC system could be evaluated effectively within the HILS environment. We can conclude that the proposed or adopted methods solve the problems issued in autonomous system design.해양 작업 지원선 (Offshore Support Vessel: OSV)의 경우 극한의 환경에도 불구하고 출항하여 해상에서 작업을 수행해야 하는 경우가 있다. 이러한 위험에의 노출을 최소화하기 위해 자율 운항에 대한 요구가 증가하고 있다. 여기서의 자율 운항은 선박이 출발지에서 목적지까지 사람의 도움 없이 이동함을 의미한다. 자율 운항 방법은 경로 추종 방법과 충돌 회피 방법을 포함한다. 우선, 운항 및 작업 중 환경 하중 (바람, 파도, 조류 등)에 대한 고려를 해야 하고, 국제 해상 충돌 예방 규칙 (Convention of the International Regulations for Preventing Collisions at Sea, COLREGs)에 의한 선박간의 항법 규정을 고려하여 충돌 회피 규칙을 준수해야 한다. 특히 연근해의 복잡한 해역에서는 많은 선박을 자동으로 회피할 필요가 있다. 기존의 해석적인 방법을 사용하기 위해서는 선박들에 대한 정확한 시스템 모델링이 되어야 하며, 그 과정에서 경험 (experience)에 의존하는 파라미터 튜닝이 필수적이다. 또한, 회피해야 할 선박 수가 많아질 경우 시스템 모델이 커지게 되고 계산 양과 계산 시간이 늘어나 실시간 적용이 어렵다는 단점이 있다. 또한, 경로 추종 및 충돌 회피를 포함하여 자율 운항 방법을 적용하기가 어렵다. 따라서 본 연구에서는 강화 학습 (Reinforcement Learning: RL) 기법을 이용하여 기존 해석적인 방법의 문제점을 극복할 수 있는 방법을 제안하였다. 경로를 추종하는 선박 (agent)은 외부 환경 (environment)과 상호작용하면서 학습을 진행한다. State S_0 (선박의 움직임과 관련된 각종 상태) 가지는 agent는 policy (현재 위치에서 어떤 움직임을 선택할 것인가)에 따라 action A_0 (움직일 방향) 취한다. 이에 environment는 agent의 다음 state S_1 을 계산하고, 그에 따른 보상 R_0 (해당 움직임의 적합성)을 결정하여 agent에게 전달한다. 이러한 작업을 반복하면서 보상이 최대가 되도록 policy를 학습하게 된다. 한편, 해상에서 크레인을 이용한 장비의 이동이나 설치 작업 시 위험을 줄이기 위해 크레인의 거동 제어에 대한 요구가 증가하고 있다. 특히 해상에서는 선박의 운동에 의해 크레인에 매달린 물체가 상하 동요 (heave)와 크레인을 기준으로 좌우 동요 (sway)가 발생하는데, 이러한 운동은 작업을 지연시키고, 정확한 위치에 물체를 놓지 못하게 하며, 자칫 주변 구조물과의 충돌을 야기할 수 있다. 이와 같은 동요를 최소화하는 Anti-Motion Control (AMC) 시스템은 Anti-Heave Control (AHC)과 Anti-Sway Control (ASC)을 포함한다. 본 연구에서는 해양 작업 지원선에 적합한 AMC 시스템의 설계 및 검증 방법을 연구하였다. 먼저 상하 동요를 최소화하기 위해 크레인의 와이어 길이를 능동적으로 조정하는 AHC 시스템을 설계하였다. 또한, 기존의 제어 시스템의 검증 방법은 실제 선박이나 해양 구조물에 해당 제어 시스템을 직접 설치하기 전에는 그 성능을 테스트하기가 힘들었다. 이를 해결하기 위해 본 연구에서는 Hardware-In-the-Loop Simulation (HILS) 기법을 활용하여 AHC 시스템의 검증 방법을 연구하였다. 또한, ASC 시스템을 설계할 때 제어 대상이 under-actuated 시스템이기 때문에 제어하기가 매우 어렵다. 따라서 본 연구에서는 sliding mode control 알고리즘을 이용하며 다관절 크레인 (knuckle boom crane)의 관절 (joint) 각도를 제어하여 좌우 동요를 줄일 수 있는 ASC 시스템을 설계하였다.Chapter 1 Introduction 1 1.1 Background and Motivation . . . . . . . . . . . . . . . . . . . . . 1 1.2 Requirements for Autonomous Operation . . . . . . . . . . . . . 5 1.2.1 Path Following for Autonomous Ship . . . . . . . . . . . . 5 1.2.2 Collision Avoidance for Autonomous Ship . . . . . . . . . 5 1.2.3 Anti-Motion Control System for Autonomous Ship . . . . 6 1.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3.1 Related Work for Path Following System . . . . . . . . . 9 1.3.2 Related Work for Collision Avoidance System . . . . . . . 9 1.3.3 Related Work for Anti-Heave Control System . . . . . . . 13 1.3.4 Related Work for Anti-Sway Control System . . . . . . . 14 1.4 Configuration of Simulation Framework . . . . . . . . . . . . . . 16 1.4.1 Application Layer . . . . . . . . . . . . . . . . . . . . . . 16 1.4.2 Autonomous Ship Design Layer . . . . . . . . . . . . . . . 17 1.4.3 General Technique Layer . . . . . . . . . . . . . . . . . . 17 1.5 Contributions (Originality) . . . . . . . . . . . . . . . . . . . . . 19 Chapter 2 Theoretical Backgrounds 20 2.1 Maneuvering Model for Autonomous Ship . . . . . . . . . . . . . 20 2.1.1 Kinematic Equation for Autonomous Ship . . . . . . . . . 20 2.1.2 Kinetic Equation for Autonomous Ship . . . . . . . . . . 21 2.2 Multibody Dynamics Model for Knuckle Boom Crane of Autonomous Ship. . . 25 2.2.1 Embedding Techniques . . . . . . . . . . . . . . . . . . . . 25 2.3 Control System Design . . . . . . . . . . . . . . . . . . . . . . . . 31 2.3.1 Proportional-Integral-Derivative (PID) Control . . . . . . 31 2.3.2 Sliding Mode Control . . . . . . . . . . . . . . . . . . . . 31 2.4 Deep Reinforcement Learning Algorithm . . . . . . . . . . . . . . 34 2.4.1 Value Based Learning Method . . . . . . . . . . . . . . . 36 2.4.2 Policy Based Learning Method . . . . . . . . . . . . . . . 37 2.4.3 Actor-Critic Method . . . . . . . . . . . . . . . . . . . . . 41 2.5 Hardware-in-the-Loop Simulation . . . . . . . . . . . . . . . . . . 43 2.5.1 Integrated Simulation Method . . . . . . . . . . . . . . . 43 Chapter 3 Path Following Method for Autonomous OSV 46 3.1 Guidance System . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.1.1 Line-of-sight Guidance System . . . . . . . . . . . . . . . 46 3.2 Deep Reinforcement Learning for Path Following System . . . . . 50 3.2.1 Deep Reinforcement Learning Setup . . . . . . . . . . . . 50 3.2.2 Neural Network Architecture . . . . . . . . . . . . . . . . 56 3.2.3 Training Process . . . . . . . . . . . . . . . . . . . . . . . 58 3.3 Implementation and Simulation Result . . . . . . . . . . . . . . . 62 3.3.1 Implementation for Path Following System . . . . . . . . 62 3.3.2 Simulation Result . . . . . . . . . . . . . . . . . . . . . . 65 3.4 Comparison Results . . . . . . . . . . . . . . . . . . . . . . . . . 83 3.4.1 Comparison Result of PPO with PID . . . . . . . . . . . 83 3.4.2 Comparison Result of PPO with Deep Q-Network (DQN) 87 Chapter 4 Collision Avoidance Method for Autonomous OSV 89 4.1 Deep Reinforcement Learning for Collision Avoidance System . . 89 4.1.1 Deep Reinforcement Learning Setup . . . . . . . . . . . . 89 4.1.2 Neural Network Architecture . . . . . . . . . . . . . . . . 93 4.1.3 Training Process . . . . . . . . . . . . . . . . . . . . . . . 94 4.2 Implementation and Simulation Result . . . . . . . . . . . . . . . 95 4.2.1 Implementation for Collision Avoidance System . . . . . . 95 4.2.2 Simulation Result . . . . . . . . . . . . . . . . . . . . . . 100 4.3 Implementation and Simulation Result for Multi-ship Collision Avoidance Method . . . . . . . . . . . . . . . . . . . . . . . . . . 107 4.3.1 Limitations of Multi-ship Collision Avoidance Method - 1 107 4.3.2 Limitations of Multi-ship Collision Avoidance Method - 2 108 4.3.3 Implementation of Multi-ship Collision Avoidance Method 110 4.3.4 Simulation Result of Multi-ship Collision Avoidance Method 118 Chapter 5 Anti-Motion Control Method for Knuckle Boom Crane 129 5.1 Configuration of HILS for Anti-Heave Control System . . . . . . 129 5.1.1 Virtual Mechanical System . . . . . . . . . . . . . . . . . 132 5.1.2 Virtual Sensor and Actuator . . . . . . . . . . . . . . . . 138 5.1.3 Control System Design . . . . . . . . . . . . . . . . . . . . 141 5.1.4 Integrated Simulation Interface . . . . . . . . . . . . . . . 142 5.2 Implementation and Simulation Result of HILS for Anti-Heave Control System . . . . . . . . 145 5.2.1 Implementation of HILS for Anti-Heave Control System . 145 5.2.2 Simulation Result of HILS for Anti-Heave Control System 146 5.3 Validation of HILS for Anti-Heave Control System . . . . . . . . 159 5.3.1 Hardware Setup . . . . . . . . . . . . . . . . . . . . . . . 159 5.3.2 Comparison Result . . . . . . . . . . . . . . . . . . . . . . 161 5.4 Configuration of Anti-Sway Control System . . . . . . . . . . . . 162 5.4.1 Mechanical System for Knuckle Boom Crane . . . . . . . 162 5.4.2 Anti-Sway Control System Design . . . . . . . . . . . . . 165 5.4.3 Implementation and Simulation Result of Anti-Sway Control . . . . . . . . . . . . . . 168 Chapter 6 Conclusions and Future Works 176 Bibliography 178 Chapter A Appendix 186 국문초록 188Docto

Path planning and collision avoidance for autonomous surface vehicles I: a review

Autonomous surface vehicles are gaining increasing attention worldwide due to the potential benefits of improving safety and efficiency. This has raised the interest in developing methods for path planning that can reduce the risk of collisions, groundings, and stranding accidents at sea, as well as costs and time expenditure. In this paper, we review guidance, and more specifically, path planning algorithms of autonomous surface vehicles and their classification. In particular, we highlight vessel autonomy, regulatory framework, guidance, navigation and control components, advances in the industry, and previous reviews in the field. In addition, we analyse the terminology used in the literature and attempt to clarify ambiguities in commonly used terms related to path planning. Finally, we summarise and discuss our findings and highlight the potential need for new regulations for autonomous surface vehicles