복잡한 해상 상황에서의 강화 학습 기반 선박 충돌 회피 방법

학위논문(박사) -- 서울대학교대학원 : 공과대학 조선해양공학과, 2024. 2. 노명일.A Method for Collision Avoidance of a Ship Based on Reinforcement Learning in Complex Maritime Situations As shipping routes become increasingly complex, the number of human-caused ship accidents (such as by navigator inattention) is also increasing. Especially where many ships operate in small areas, there is a growing need for an autonomous navigation system for a maritime autonomous surface ship (MASS) that accurately predicts potential danger to own ship and performs safe navigation without collision with maritime obstacles such as target ships. The autonomous navigation system must accurately identify the position, speed, and direction of target ships in real-time, and assess the collision risk of each target ship to the own ship. Also, the autonomous navigation system must perform collision avoidance at the appropriate time based on the evaluation of the current situation around own ship. For collision risk assessment to quantify the potential danger between the own ship and the target ship, many studies have used the ship domain method and the closest point of approach (CPA) based method. The ship domain method is a method of creating a ship domain around the own ship based on the maneuvering performance. The ship domain is considered that own ship and target ship collide if the target ship invades the ship domain. Although it provides a clear standard for collisions, it has the disadvantage of not providing a quantitative value of collision risk. The CPA-based method is a method of assessing the value of collision risk based on various indicators calculated by the closest point of approach when the own ship and the target ship maintain their respective speeds and directions. Although it provides quantitative value, it has the disadvantage of not providing a clear standard for collision. Therefore, in this study, a collision risk assessment is proposed combining the ship domain method and the CPA-based method. However, the input data of the target ship that the own ship identifies may be subject to errors, such as false detection and noise of sensor data. As a result, it can be difficult to assess the accurate collision risk of the target ship. To solve this problem, this study proposes a collision risk assessment by reflecting the uncertainty of input data of the target ship based on multivariate normal distribution. The errors in the x, and y position, speed, and direction of the target ship are assumed to be multivariate normal distributions. The probability distribution of the area where the TS exists can be calculated by considering the uncertainty of input data. Then, the largest value of the collision risk assessed within the area is set as the representative value for each uncertainty. Meanwhile, all ships are required to perform collision avoidance by complying with the Convention on the International Regulations for Preventing Collision at Sea (CORREGs, 1972) to safely avoid collisions between ships when operating. In this study, a collision avoidance method is proposed that complies with COLREGs and can flexibly handle sudden changes in the situation. The proposed method is based on deep reinforcement learning, which receives increasing attention in the field of artificial intelligence and machine control, to derive the path and speed of own ship by considering multiple target ships simultaneously. The proposed collision avoidance method based on deep reinforcement learning recognizes multiple target ships around the own ship simultaneously and performs collision avoidance based on all target ships. In addition, the rudder angle and the propeller RPM of the own ship are controlled together to derive the most efficient path and speed. At this time, own ship receives a path following reward if the own ship follows the desired path and speed. If the own ship safely avoids target ships, own ship receives a collision avoidance reward. In this way, own ship is trained to follow an efficient and safe path to arrive at the destination. In this study, an autonomous navigation system is proposed that includes the three methods described above. To verify the proposed methods, the collision risk assessment and collision avoidance results for various ship encounter situations were confirmed. First, to verify the effectiveness of the collision risk assessment, the collision risk assessment was applied to various ship-to-ship encounter situations. Also, the collision risk assessment considering uncertainty was applied by generating an imaginary error that simulates the error of the actual sensor data. As a result, it was confirmed that the collision risk assessment considering uncertainty proposed in this study performs safe navigation considering potential threats even when the accurate collision risk cannot be calculated due to errors in target ship information. Next, to verify the effectiveness of the deep reinforcement learning-based collision avoidance method, we applied the proposed method to a benchmarking test problem for the collision avoidance of a ship. The results show that even when multiple target ships are operating around the own ship, the own ship avoids all target ships and arrives at the destination safely. Finally, the autonomous navigation system proposed in this study was applied when a large number of target ships were approaching the own ship and there were continuous changes and errors in the movement and information of target ships. As a result, it was confirmed that the proposed autonomous navigation system enables safe navigation even in an environment where the error of target ship information is large and many target ships are densely concentrated. Keywords: Maritime autonomous surface ship; Autonomous Navigation System; Collision risk assessment; Ship collision avoidance; Multivariate normal distribution; Deep reinforcement learning; Ship control Student number: 2017-27337항로가 점점 복잡해짐에 따라, 항해사의 부주의 등과 같은 사람에 의한 선박 사고가 점점 증가하고 있다. 특히 다수의 선박이 좁은 지역에서 복잡하게 운항하는 연안에서는 자선의 잠재적인 위험을 정확하게 예측하고, 이를 기반으로 타선들과 충돌하지 않고 안전한 운항을 수행하는 자율 운항 시스템에 대한 필요성이 증가하고 있다. 자율 운항 시스템은 타선 등 해상 장애물의 위치, 속력, 진행 방향 등을 실시간으로 정확하게 파악해야 하며, 이를 기반으로 자선에 대한 각 타선의 상대적인 위험을 평가하고 적절한 시점에 충돌 회피를 수행해야 한다. 자선과 타선 사이의 잠재적인 위험을 정량적으로 평가하기 위해 다수의 연구에서는 선박 안전 영역 방법 및 최근접점 방법을 활용하였다. 선박 안전 영역 방법은 자선의 주위에 선박의 운동 성능을 기반으로 해당 영역을 침범할 경우 충돌한 것으로 간주하는 선박 안전 영역을 생성하는 방법으로, 충돌에 대한 확실한 기준을 제공하지만 정량적인 수치를 제공하지 않는다는 단점을 가진다. 최근접점 방법은 자선과 타선이 각각의 속력 및 진행 방향을 유지할 경우 가장 가까워지는 점인 최근접점 (Closest Point of Approach)에 기반한 각종 지표를 기반으로 충돌 위험도를 평가하는 방법으로, 타선의 잠재적인 위협에 대한 정량적인 수치는 제공하지만 충돌에 대한 확실한 기준을 제공하지 못한다는 단점을 가진다. 따라서 본 연구에서는 선박 안전 영역 방법과 최근접점 방법을 결합한 충돌 위험도 평가 방법에 대한 연구를 수행하였다. 하지만 이 때, 자선이 인지하는 타선의 정보에는 센서 데이터 오탐지 등에 의한 오차가 발생할 수 있으며, 그로 인해 타선의 정확한 충돌 위험도를 평가하기 어려울 수 있다. 이를 해결하기 위해, 본 연구에서는 다변수 정규 분포(multivariate normal distribution)에 기반한 타선 정보의 불확실성을 반영하여 충돌 위험도를 평가하는 방법을 제안하였다. 타선의 x, y 위치, 속력, 진행 방향의 오차를 다변수 정규 분포 형태로 가정하였으며, 이를 기반으로 타선 정보의 존재 확률에 대한 불확실성 영역을 생성하였다. 그 후, 해당 영역 내에서 계산된 충돌 위험도 중 가장 큰 값을 각 존재 확률에 대한 대푯값으로 설정하였다. 한편, 모든 선박들은 운항 시 선박 간의 충돌을 안전하게 회피하기 위해 국제 해상 충돌 방지 협약 COLREGs (Convention on the International Regulations for Preventing Collision at Sea, 1972)를 준수하여 충돌 회피를 수행해야 한다. 본 연구에서는 CORLEGs 를 준수하고 갑작스러운 상황 변화에도 유연하게 잘 대처할 수 있는 자율 운항 시스템을 위해 인공 지능 및 기계 제어 분야에서 각광받는 심층 강화 학습 (deep reinforcement learning)을 기반으로 다수의 타선을 동시에 고려해 자선의 경로 및 속력을 도출하는 충돌 회피 방법을 제안하였다. 본 연구에서 제안된 심층 강화 학습 기반 충돌 회피 방법은 자선 주변의 다수의 타선을 동시에 인지하고 이를 기반으로 충돌 회피를 수행한다. 또한, 자선의 조타각 (rudder angle) 및 프로펠러의 분당 회전수 (propeller RPM)을 함께 제어하여 가장 효율적인 자선의 경로와 속력을 도출한다. 이 때, 자선이 계획 경로 및 속력을 준수할 경우 경로 보상 (path following reward)를, 자선이 타선을 안전하게 회피할 경우 회피 보상 (collision avoidance reward)를 부여하여 자선이 효율적이고 안전한 경로를 따라 목적지에 도달하도록 학습되었다. 본 연구에서는 위에 설명한 세가지 방법을 포함하는 자율 운항 시스템을 제안하였다. 제안된 방법들을 검증하기 위해 다양한 선박 조우 상황에 대한 충돌 위험도 평가 및 충돌 회피 결과를 확인하였다. 먼저, 불확실성 기반 충돌 위험도 평가 방법의 효용성을 검증하기 위하여 여러가지 선박 간 조우 상황에 대해 충돌 위험도 평가 방법을 적용하였으며, 실제 센서 데이터의 오차를 모사한 가상의 오차를 생성하여 불확실성 기반 충돌 위험도 평가 방법을 적용하였다. 그 결과, 본 연구에서 제안한 불확실성 기반 충돌 위험도 평가에 의해 타선 정보에 오차가 발생하여 정확한 충돌 위험도를 계산할 수 없을 때도 잠재적인 위협을 고려하여 안전한 운항을 수행하는 것을 확인하였다. 다음으로, 심층 강화 학습 기반 충돌 회피 방법의 효용성을 검증하기 위하여 선박의 충돌 회피 벤치 마킹 테스트 문제에 적용하였다. 그 결과, 다수의 타선이 자선 주변에서 운항하는 상황에서도 자선이 타선을 회피하여 안전하게 목적지에 도달하는 것을 확인하였다. 마지막으로, 다수의 타선이 자선을 향해 다가오고 타선의 움직임 및 정보에 지속적인 변화와 오차가 발생한 경우 본 연구에서 제안된 자율 운항 시스템을 적용하였다. 그 결과, 제안된 자율 운항 시스템을 적용하면 타선 정보의 오차가 크고 많은 타선이 밀집한 환경에서도 안전한 운항이 가능함을 확인하였다.Abstract i Nomenclature . x Introduction 1 1.1. Research Backgrounds 1 1.2. Related Works 5 1.2.1. Related Works for Collision Risk Assessment . 5 1.2.2. Related Works for Collision Avoidance Method 7 1.3. Research Objectives and Work Scope 12 Theoretical Backgrounds 13 2.1. Collision Risk Assessment 14 2.1.1. Regulations for Preventing Collisions in the Sea: COLREGs 15 2.1.2. Calculation of Collision Risk 17 2.2. Uncertainty of Input Data for Collision Avoidance Method 27 2.2.1. Uncertainty of Input Data to Collision Risk Assessment 27 2.2.2. Multivariate Normal Distribution for Collision Risk Assessment ..30 2.3. Collision Avoidance Method for MASS 34 2.3.1. Motion and Maneuver System for MASS .34 2.3.2. Collision Avoidance Method with Deep Reinforcement Learning 38 2.3.3. Formulation of the State 43 2.3.4. Formulation of the Action47 2.3.5. Formulation of the Rewards 50 (1) Goal reward 52 (2) Cross reward 54 (3) Speed reward 56 (4) Risk reward 57 (5) COLREGs reward 58 Verification 60 3.1. Verification of the Collision Risk Assessment .63 3.2. Verification of the Uncertainty of Input Data for Collision Avoidance Method 72 3.3. Verification of the Collision Avoidance Method for MASS 79 (1) Difference by continuous control 88 (2) Difference in propeller RPM control 96 (3) Difference by complexity of state 102 (4) Difference by collision avoidance method . 108 Applications 121 4.1. SyDLabs Collision Avoidance Program 121 4.2. Comparison of Collision Risk Assessment with Other Methods 123 (1) Head-on situation 125 (2) Crossing situation 126 (3) Overtaking situation 127 4.3. Case Study of Collision Risk Assessment with Uncertainty of Input Data . 129 4.4. Case Study of Collision Avoidance Method 142 Conclusions and Future Works . 167 5.1. Summary. 167 5.2. Contributions (Originality) . 169 5.2.1. Theoretical Contributions 169 5.2.2. Contributions for Applications 170 5.2.3. Other Contributions 170 5.3. Future Works 171 References 172 APPENDICES 177 A. Approximation of Maneuverability of Ship 178 국문 초록 182 Figures Figure 1-1. Autonomous navigation system for MASS 2 Figure 1-2. Necessity of the intermediate navigation system 4 Figure 2-1. Framework for autonomous navigation system for MASS . 13 Figure 2-2. Collision avoidance according to the COLREGs for each situation 17 Figure 2-3. Concepts of CPA, DCPA, and TCPA 19 Figure 2-4. Ship domain used in this study 20 Figure 2-5. Calculation of DCPA coefficient of CR. 24 Figure 2-6. Calculation of TCPA coefficient of CR . 25 Figure 2-7. Difference by considering uncertainty of input data 29 Figure 2-8. Difference by considering the uncertainty of input data 33 Figure 2-9. The entire DRL learning process 39 Figure 2-10. Definition of the CNN state 46 Figure 2-11. Definition of the policy with multi-input state and multi-continuous action 50 Figure 2-12. Procedure for calculating the total reward . 51 Figure 2-13. Definition of goal reward 53 Figure 2-14. Definition of cross reward . 55 Figure 3-1. Summary of verification cases 60 Figure 3-2. The hull of the model ship of KVLCC2 62 Figure 3-3. Thrust coefficient of the propeller of own ship . 63 Figure 3-4. Path of the OS and TS in a head-on situation . 65 Figure 3-5. Collision risk of TS over time and distance in a head-on situation . 65 Figure 3-6. Path of the OS and TS in a starboard boundary situation . 66 Figure 3-7. Collision risk of TS over time and distance in a starboard boundary situation 67 Figure 3-8. Path of the OS and TS in a starboard situation 68 Figure 3-9. Collision risk of TS over time and distance in a starboard situation 68 Figure 3-10. Path of the OS and TS in a crossing situation . 69 Figure 3-11. Collision risk of TS over time and distance in a crossing situation 70 Figure 3-12. Path of the OS and TS in an overtaking situation 71 Figure 3-13. Collision risk of TS over time and distance in an overtaking situation 71 Figure 3-14. Input data and covariance matrix of TS 73 Figure 3-15. Real and noise value of the position of TS 74 Figure 3-16. Real and noise value of speed and direction of TS 75 Figure 3-17. Real and noise value of collision risk of TS . 76 Figure 3-18. Collision risk of TS with various conditions 78 Figure 3-19. Neural network architecture of the policy . 81 Figure 3-20. The total reward of the scenario for each stage 82 Figure 3-21. benchmark scenarios of the Imazu problem . 83 Figure 3-22. Effect of the variation in each parameter 87 Figure 3-23. Path of the OS and TSs with Model 2 and Model 4 . 89 Figure 3-24. Speed of the OS and TSs with Model 2 and Model 4 . 90 Figure 3-25. Collision risk of TSs with Model 2 and Model 4 92 Figure 3-26. Path of the OS and TSs with Model 6 and Model 8 . 93 Figure 3-27. Speed of the OS and TSs with Model 6 and Model 8 . 94 Figure 3-28. Collision risk of TSs with Model 6 and Model 8 95 Figure 3-29. Path of the OS and TSs with Model 1 and Model 2 . 97 Figure 3-30. Speed of the OS with Model 1 and Model 2 97 Figure 3-31. Collision risk of TSs with Model 1 and Model 2 98 Figure 3-32. Path of the OS and TSs with Model 7 and Model 8 . 99 Figure 3-33. Speed of the OS with Model 7 and Model 8 . 100 Figure 3-34. Collision risk of TSs with Model 7 and Model 8 101 Figure 3-35. Path of the OS and TSs with Model 2 and Model 6 102 Figure 3-36. Speed of the OS with Model 2 and Model 6 . 103 Figure 3-37. Collision risk of TSs with Model 2 and Model 6 104 Figure 3-38. Path of the OS and TSs with Model 4 and Model 8 105 Figure 3-39. Speed of the OS with Model 4 and Model 8 . 106 Figure 3-40. Collision risk of TSs with Model 4 and Model 8 107 Figure 3-41. Path of the OS and TSs with DRL and RVO . 109 Figure 3-42. Speed of the OS with DRL and RVO 109 Figure 3-43. Collision risk of TSs with DRL and RVO 110 Figure 3-44. Path of the OS and TSs of LNG carrier with DRL and RVO for Imazu #10 . 113 Figure 3-45. Speed of the OS of LNG carrier with DRL and RVO for Imazu #10 113 Figure 3-46. Collision risk of TSs of LNG carrier with DRL and RVO for Imazu #10 .. 115 Figure 3-47. Path of the OS and TSs of LNG carrier with DRL and RVO for Imazu #15 . 117 Figure 3-48. Speed of the OS of LNG carrier with DRL and RVO for Imazu #15 118 Figure 3-49. Collision risk of TSs of LNG carrier with DRL and RVO for Imazu #15 .. 119 Figure 4-1. Overview of SyDLabs Collision Avoidance Program 122 Figure 4-2. Overview of collision risk assessment with uncertainty of input data 123 Figure 4-3. Collision risk with each method in a head-on situation . 126 Figure 4-4. Collision risk with each method in a crossing situation . 127 Figure 4-5. Collision risk with each method in an overtaking situation . 128 Figure 4-6. Randomly generated TSs for collision risk assessment 130 Figure 4-7. Path of the OS with CR based on the real value of input data 132 Figure 4-8. Speed of the OS with CR based on the real value of input data . 132 Figure 4-9. CR of the OS with CR based on the real value of input data 133 Figure 4-10. Path of the OS with CR based on the noise value of input data 135 Figure 4-11. Speed of the OS with CR based on the noise value of input data 135 Figure 4-12. CR of the OS with CR based on the noise value of input data . 137 Figure 4-13. Path of the OS with CR considering the uncertainty of input data . 139 Figure 4-14. Speed of the OS with CR considering the uncertainty of input data . 140 Figure 4-15. CR of the OS with CR considering the uncertainty of input data 141 Figure 4-16. Path of the OS with RVO-based collision avoidance method 144 Figure 4-17. Speed of the OS with RVO-based collision avoidance method 145 Figure 4-18. CR of the OS with RVO-based collision avoidance method . 146 Figure 4-19. Path of the OS with DRL-based collision avoidance method 148 Figure 4-20. Speed of the OS with DRL-based collision avoidance method 148 Figure 4-21. CR of the OS with DRL-based collision avoidance method . 149 Figure 4-22. All TSs used in the example for the collision avoidance method 151 Figure 4-23. Path of the OS when all TSs maintain their own path 152 Figure 4-24. Speed of the OS when all TSs maintain their own path 153 Figure 4-25. Maximum CR of the OS at each time when all TSs maintain their own path . 154 Figure 4-26. Path of the OS when all TSs perform RVO-based collision avoidance method . 155 Figure 4-27. Speed of the OS when all TSs perform RVO-based collision avoidance method . 156 Figure 4-28. Maximum CR of the OS at each time when all TSs perform the RVO-based collision avoidance method 157 Figure 4-29. Path of all TSs when all TSs perform RVO-based collision avoidance method . 158 Figure 4-30. Path of the OS when all TSs perform the DRL-based collision avoidance method . 159 Figure 4-31. Speed of the OS when all TSs perform the DRL-based collision avoidance method . 160 Figure 4-32. Maximum CR of the OS at each time when all TSs perform the DRL-based collision avoidance method 161 Figure 4-33. Path of all TSs when all TSs perform DRL-based collision avoidance method . 162 Figure 4-34. Path of the OS when all TSs change the movements randomly 164 Figure 4-35. Speed of the OS when all TSs change the movements randomly. 164 Figure 4-36. Maximum CR of the OS at each time when all TSs change the movements randomly. 165 Figure 4-37. Path of all TSs when all TSs change the movements randomly 166 viii박

Operational Analysis of Container Ships by Using Maritime Big Data

The shipping company or the operator determines the mode of operation of a ship. In the case of container ships, there may be various operating patterns employed to arrive at the destination within the stipulated time. In addition, depending on the influence of the ocean's environmental conditions, the speed and the route can be changed. As the ship's fuel oil consumption is closely related to its operational pattern, it is possible to identify the most economical operations by analyzing the operational patterns of the ships. The operational records of each shipping company are not usually disclosed, so it is necessary to estimate the operational characteristics from publicly available data such as the automatic identification system (AIS) data and ocean environment data. In this study, we developed a visualization program to analyze the AIS data and ocean environmental conditions together and propose two categories of applications for the operational analysis of container ships using maritime big data. The first category applications are the past operation analysis by tracking previous trajectories, and the second category applications are the speed pattern analysis by shipping companies and shipyards under harsh environmental conditions. Thus, the operational characteristics of container ships were evaluated using maritime big data