Human Motion Trajectory Prediction: A Survey

With growing numbers of intelligent autonomous systems in human environments, the ability of such systems to perceive, understand and anticipate human behavior becomes increasingly important. Specifically, predicting future positions of dynamic agents and planning considering such predictions are key tasks for self-driving vehicles, service robots and advanced surveillance systems. This paper provides a survey of human motion trajectory prediction. We review, analyze and structure a large selection of work from different communities and propose a taxonomy that categorizes existing methods based on the motion modeling approach and level of contextual information used. We provide an overview of the existing datasets and performance metrics. We discuss limitations of the state of the art and outline directions for further research.Comment: Submitted to the International Journal of Robotics Research (IJRR), 37 page

Nonlinear Modeling and Control of Driving Interfaces and Continuum Robots for System Performance Gains

With the rise of (semi)autonomous vehicles and continuum robotics technology and applications, there has been an increasing interest in controller and haptic interface designs. The presence of nonlinearities in the vehicle dynamics is the main challenge in the selection of control algorithms for real-time regulation and tracking of (semi)autonomous vehicles. Moreover, control of continuum structures with infinite dimensions proves to be difficult due to their complex dynamics plus the soft and flexible nature of the manipulator body. The trajectory tracking and control of automobile and robotic systems requires control algorithms that can effectively deal with the nonlinearities of the system without the need for approximation, modeling uncertainties, and input disturbances. Control strategies based on a linearized model are often inadequate in meeting precise performance requirements. To cope with these challenges, one must consider nonlinear techniques. Nonlinear control systems provide tools and methodologies for enabling the design and realization of (semi)autonomous vehicle and continuum robots with extended specifications based on the operational mission profiles. This dissertation provides an insight into various nonlinear controllers developed for (semi)autonomous vehicles and continuum robots as a guideline for future applications in the automobile and soft robotics field. A comprehensive assessment of the approaches and control strategies, as well as insight into the future areas of research in this field, are presented.First, two vehicle haptic interfaces, including a robotic grip and a joystick, both of which are accompanied by nonlinear sliding mode control, have been developed and studied on a steer-by-wire platform integrated with a virtual reality driving environment. An operator-in-the-loop evaluation that included 30 human test subjects was used to investigate these haptic steering interfaces over a prescribed series of driving maneuvers through real time data logging and post-test questionnaires. A conventional steering wheel with a robust sliding mode controller was used for all the driving events for comparison. Test subjects operated these interfaces for a given track comprised of a double lane-change maneuver and a country road driving event. Subjective and objective results demonstrate that the driver’s experience can be enhanced up to 75.3% with a robotic steering input when compared to the traditional steering wheel during extreme maneuvers such as high-speed driving and sharp turn (e.g., hairpin turn) passing. Second, a cellphone-inspired portable human-machine-interface (HMI) that incorporated the directional control of the vehicle as well as the brake and throttle functionality into a single holistic device will be presented. A nonlinear adaptive control technique and an optimal control approach based on driver intent were also proposed to accompany the mechatronic system for combined longitudinal and lateral vehicle guidance. Assisting the disabled drivers by excluding extensive arm and leg movements ergonomically, the device has been tested in a driving simulator platform. Human test subjects evaluated the mechatronic system with various control configurations through obstacle avoidance and city road driving test, and a conventional set of steering wheel and pedals were also utilized for comparison. Subjective and objective results from the tests demonstrate that the mobile driving interface with the proposed control scheme can enhance the driver’s performance by up to 55.8% when compared to the traditional driving system during aggressive maneuvers. The system’s superior performance during certain vehicle maneuvers and approval received from the participants demonstrated its potential as an alternative driving adaptation for disabled drivers. Third, a novel strategy is designed for trajectory control of a multi-section continuum robot in three-dimensional space to achieve accurate orientation, curvature, and section length tracking. The formulation connects the continuum manipulator dynamic behavior to a virtual discrete-jointed robot whose degrees of freedom are directly mapped to those of a continuum robot section under the hypothesis of constant curvature. Based on this connection, a computed torque control architecture is developed for the virtual robot, for which inverse kinematics and dynamic equations are constructed and exploited, with appropriate transformations developed for implementation on the continuum robot. The control algorithm is validated in a realistic simulation and implemented on a six degree-of-freedom two-section OctArm continuum manipulator. Both simulation and experimental results show that the proposed method could manage simultaneous extension/contraction, bending, and torsion actions on multi-section continuum robots with decent tracking performance (e.g. steady state arc length and curvature tracking error of 3.3mm and 130mm-1, respectively). Last, semi-autonomous vehicles equipped with assistive control systems may experience degraded lateral behaviors when aggressive driver steering commands compete with high levels of autonomy. This challenge can be mitigated with effective operator intent recognition, which can configure automated systems in context-specific situations where the driver intends to perform a steering maneuver. In this article, an ensemble learning-based driver intent recognition strategy has been developed. A nonlinear model predictive control algorithm has been designed and implemented to generate haptic feedback for lateral vehicle guidance, assisting the drivers in accomplishing their intended action. To validate the framework, operator-in-the-loop testing with 30 human subjects was conducted on a steer-by-wire platform with a virtual reality driving environment. The roadway scenarios included lane change, obstacle avoidance, intersection turns, and highway exit. The automated system with learning-based driver intent recognition was compared to both the automated system with a finite state machine-based driver intent estimator and the automated system without any driver intent prediction for all driving events. Test results demonstrate that semi-autonomous vehicle performance can be enhanced by up to 74.1% with a learning-based intent predictor. The proposed holistic framework that integrates human intelligence, machine learning algorithms, and vehicle control can help solve the driver-system conflict problem leading to safer vehicle operations

도심 교차로에서의 자율주행을 위한 주변 차량 경로 예측 및 거동 계획 알고리즘

학위논문(박사)--서울대학교 대학원 :공과대학 기계항공공학부,2020. 2. 이경수.차랑용 센싱 및 처리기술이 발달함에 따라 자동차 기술 연구가 수동 안전 기술에서 능동 안전 기술로 초점이 확장되고 있다. 최근, 주요 자동차 제작사들은 능동형 차간거리 제어, 차선 유지 보조, 그리고 긴급 자동 제동과 같은 능동 안전 기술이 이미 상업화하고 있다. 이러한 기술적 진보는 사상률 제로를 달성하기 위하여 기술 연구 분야를 능동 안전 기술을 넘어서 자율주행 시스템으로 확장시키고 있다. 특히, 도심 도로는 인도, 사각지대, 주차차량, 이륜차, 보행자 등과 같은 교통 위험 요소를 많이 갖고 있기 때문에 고속도로보다 사고 발생률과 사상률이 높으며, 이는 도심 도로에서의 자율주행은 핵심 이슈가 되고 있다. 많은 프로젝트들이 자율주행의 환경적, 인구학적, 사회적, 그리고 경제적 측면에서의 자율주행의 효과를 평가하기 위해 수행되었거나 수행 중에 있다. 예를 들어, 유럽의 AdaptIVE는 다양한 자율주행 기능을 개발하였으며, 구체적인 평가 방법론을 개발하였다. 또한, CityMobil2는 유럽 전역의 9개의 다른 환경에서 무인 지능형 차량을 성공적으로 통합하였다. 일본에서는 2014년 5월에 시작된 Automated Driving System Research Project는 자율주행 시스템과 차세대 도심 교통 수단의 개발 및 검증에 초점을 맞추었다. 기존 연구들에 대한 조사를 통해 자율주행 시스템은 교통 참여자들의 안전도를 향상시키고, 교통 혼잡을 감소시키며, 운전자 편의성을 증진시키는 것이 증명되었다. 다양한 방법론들이 인지, 거동 계획, 그리고 제어와 같은 도심 도로 자율주행차의 핵심 기술들을 개발하기 위하여 사용되었다. 하지만 많은 최신의 자율주행 연구들은 각 기술의 개발을 별개로 고려하여 진행해왔다. 결과적으로 통합적인 관점에서의 자율주행 기술 설계는 아직 충분히 고려되어 않았다. 따라서, 본 논문은 복잡한 도심 도로 환경에서 라이다, 카메라, GPS, 그리고 간단한 경로 맵에 기반한 완전 자율주행 알고리즘을 개발하는 것을 목표로 하였다. 제안된 자율주행 알고리즘은 비통제 교차로를 포함한 도심 도로 상황을 차량 거동 예측기와 모델 예측 제어 기법에 기반하여 설계되었다. 본 논문은 동적, 정적 환경 표현 및 종횡방향 거동 계획을 중점적으로 다루었다. 본 논문은 도심 도로 자율주행을 위한 거동 계획 알고리즘의 개요를 제시하였으며, 실제 교통 상황에서의 실험 결과는 제안된 알고리즘의 효과성과 운전자 거동과의 유사성을 보여주었다. 실차 실험 결과는 비통제 교차로를 포함한 도심 시나리오에서의 강건한 성능을 보여주었다.The foci of automotive researches have been expanding from passive safety systems to active safety systems with advances in sensing and processing technologies. Recently, the majority of automotive makers have already commercialized active safety systems, such as adaptive cruise control (ACC), lane keeping assistance (LKA), and autonomous emergency braking (AEB). Such advances have extended the research field beyond active safety systems to automated driving systems to achieve zero fatalities. Especially, automated driving on urban roads has become a key issue because urban roads possess numerous risk factors for traffic accidents, such as sidewalks, blind spots, on-street parking, motorcycles, and pedestrians, which cause higher accident rates and fatalities than motorways. Several projects have been conducted, and many others are still underway to evaluate the effects of automated driving in environmental, demographic, social, and economic aspects. For example, the European project AdaptIVe, develops various automated driving functions and defines specific evaluation methodologies. In addition, CityMobil2 successfully integrates driverless intelligent vehicles in nine other environments throughout Europe. In Japan, the Automated Driving System Research Project began on May 2014, which focuses on the development and verification of automated driving systems and next-generation urban transportation. From a careful review of a considerable amount of literature, automated driving systems have been proven to increase the safety of traffic users, reduce traffic congestion, and improve driver convenience. Various methodologies have been employed to develop the core technology of automated vehicles on urban roads, such as perception, motion planning, and control. However, the current state-of-the-art automated driving algorithms focus on the development of each technology separately. Consequently, designing automated driving systems from an integrated perspective is not yet sufficiently considered. Therefore, this dissertation focused on developing a fully autonomous driving algorithm in urban complex scenarios using LiDAR, vision, GPS, and a simple path map. The proposed autonomous driving algorithm covered the urban road scenarios with uncontrolled intersections based on vehicle motion prediction and model predictive control approach. Mainly, four research issues are considered: dynamic/static environment representation, and longitudinal/lateral motion planning. In the remainder of this thesis, we will provide an overview of the proposed motion planning algorithm for urban autonomous driving and the experimental results in real traffic, which showed the effectiveness and human-like behaviors of the proposed algorithm. The proposed algorithm has been tested and evaluated using both simulation and vehicle tests. The test results show the robust performance of urban scenarios, including uncontrolled intersections.Chapter 1 Introduction 1 1.1. Background and Motivation 1 1.2. Previous Researches 4 1.3. Thesis Objectives 9 1.4. Thesis Outline 10 Chapter 2 Overview of Motion Planning for Automated Driving System 11 Chapter 3 Dynamic Environment Representation with Motion Prediction 15 3.1. Moving Object Classification 17 3.2. Vehicle State based Direct Motion Prediction 20 3.2.1. Data Collection Vehicle 22 3.2.2. Target Roads 23 3.2.3. Dataset Selection 24 3.2.4. Network Architecture 25 3.2.5. Input and Output Features 33 3.2.6. Encoder and Decoder 33 3.2.7. Sequence Length 34 3.3. Road Structure based Interactive Motion Prediction 36 3.3.1. Maneuver Definition 38 3.3.2. Network Architecture 39 3.3.3. Path Following Model based State Predictor 47 3.3.4. Estimation of predictor uncertainty 50 3.3.5. Motion Parameter Estimation 53 3.3.6. Interactive Maneuver Prediction 56 3.4. Intersection Approaching Vehicle Motion Prediction 59 3.4.1. Driver Behavior Model at Intersections 59 3.4.2. Intention Inference based State Prediction 63 Chapter 4 Static Environment Representation 67 4.1. Static Obstacle Map Construction 69 4.2. Free Space Boundary Decision 74 4.3. Drivable Corridor Decision 76 Chapter 5 Longitudinal Motion Planning 81 5.1. In-Lane Target Following 82 5.2. Proactive Motion Planning for Narrow Road Driving 85 5.2.1. Motivation for Collision Preventive Velocity Planning 85 5.2.2. Desired Acceleration Decision 86 5.3. Uncontrolled Intersection 90 5.3.1. Driving Phase and Mode Definition 91 5.3.2. State Machine for Driving Mode Decision 92 5.3.3. Motion Planner for Approach Mode 95 5.3.4. Motion Planner for Risk Management Phase 98 Chapter 6 Lateral Motion Planning 105 6.1. Vehicle Model 107 6.2. Cost Function and Constraints 109 Chapter 7 Performance Evaluation 115 7.1. Motion Prediction 115 7.1.1. Prediction Accuracy Analysis of Vehicle State based Direct Motion Predictor 115 7.1.2. Prediction Accuracy and Effect Analysis of Road Structure based Interactive Motion Predictor 122 7.2. Prediction based Distance Control at Urban Roads 132 7.2.1. Driving Data Analysis of Direct Motion Predictor Application at Urban Roads 133 7.2.2. Case Study of Vehicle Test at Urban Roads 138 7.2.3. Analysis of Vehicle Test Results on Urban Roads 147 7.3. Complex Urban Roads 153 7.3.1. Case Study of Vehicle Test at Complex Urban Roads 154 7.3.2. Closed-loop Simulation based Safety Analysis 162 7.4. Uncontrolled Intersections 164 7.4.1. Simulation based Algorithm Comparison of Motion Planner 164 7.4.2. Monte-Carlo Simulation based Safety Analysis 166 7.4.3. Vehicle Tests Results in Real Traffic Conditions 172 7.4.4. Similarity Analysis between Human and Automated Vehicle 194 7.5. Multi-Lane Turn Intersections 197 7.5.1. Case Study of a Multi-Lane Left Turn Scenario 197 7.5.2. Analysis of Motion Planning Application Results 203 Chapter 8 Conclusion & Future Works 207 8.1. Conclusion 207 8.2. Future Works 209 Bibliography 210 Abstract in Korean 219Docto

Experimental Validation Of An Integrated Guidance And Control System For Marine Surface Vessels

Autonomous operation of marine surface vessels is vital for minimizing human errors and providing efficient operations of ships under varying sea states and environmental conditions which is complicated by the highly nonlinear dynamics of marine surface vessels. To deal with modelling imprecision and unpredictable disturbances, the sliding mode methodology has been employed to devise a heading and a surge displacement controller. The implementation of such a controller necessitates the availability of all state variables of the vessel. However, the measured signals in the current study are limited to the global X and Y positioning coordinates of the boat that are generated by a GPS system. Thus, a nonlinear observer, based on the sliding mode methodology, has been implemented to yield accurate estimates of the state variables in the presence of both structured and unstructured uncertainties. Successful autonomous operation of a marine surface vessel requires a holistic approach encompassing a navigation system, robust nonlinear controllers and observers. Since the overwhelming majority of the experimental work on autonomous marine surface vessels was not conducted in truly uncontrolled real-world environments. The first goal of this work was to experimentally validate a fully-integrated LOS guidance system with a sliding mode controller and observer using a 16’ Tracker Pro Guide V-16 aluminium boat with a 60 hp. Mercury outboard motor operating in the uncontrolled open-water environment of Lake St. Clair, Michigan. The fully integrated guidance and controller-observer system was tested in a model-less configuration, whereby all information provided from the vessel’s nominal model have been ignored. The experimental data serves to demonstrate the robustness and good tracking characteristics of the fully-integrated guidance and controller/observer system by overcoming the large errors induced at the beginning of each segment and converging the boat to the desired trajectory in spite of the presence of environmental disturbances. The second focus of this work was to combine a collision avoidance method with the guidance system that accounted for “International Regulations for Prevention of Collisions at Sea” abbreviated as COLREGS. This new system then needed to be added into the existing architecture. The velocity obstacles method was selected as the base to build upon and additional restrictions were incorporated to account for these additional rules. This completed system was then validated with a software in the loop simulation