자율 주행을 위한 학습 기반의 다중 교통 참여자 경로예측 방법

학위논문 (석사)-- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2019. 2. 서승우.자율주행차량이 안전하면서도, 교통 흐름을 방해하지 않는 인간 수준의 interactive한 주행을 실현하기 위해서는 주변 운전자와 보행자를 포함한 교통 참여자들의 의도를 파악하는 것이 필수적이다. 동일한 도로 환경에서도 주변의 교통 참여자들이 어떤 의도를 가지고 이동하고 있는가에 따라서 적합한 주행 전략은 매번 달라진다. 특히 좁은 공간을 많은 에이전트들이 공유하고 있는 골목길 상황에서는 각 에이전트들의 선택은 제한되지만, 각 에이전트들의 의사결정과정은 그들간의 상호작용이 고려되어 매우 복잡하다.이런 상황에 맞는 경로예측은 과거의 궤적, 현재 인식하고 있는 도로 환경, 주변 교통 참여자의 상태 등을 고려하여 수행되어야 한다. 또한 일반화된 대부분의 환경에서 수행되려면, 빠른 러닝 타임과 동시에 고려하는 교통 참여자의 숫자에 대한 제약이 적어야 한다. 이 논문에서는 각 교통 참여자들의 과거 궤적과 현재 위치 상황, 도로 상황을 동시에 고려하여 모든 참여자들의 상호작용이 고려된 경로예측 방법을 제안한다. 각 물체들의 과거 궤적과 동시에 자율주행 차량이 인식하고 있는 도로와 주변 물체를 입력으로하고, 이 두개의 임베딩 결과를 혼합하여 모든 물체들에 대한 경로 예측을 동시에 수행한다. 이 과정에서 네트워크 구조 내부에서 각각 물체들의 임베딩 결과를 위치 정보에 매칭시킴으로써 효과적으로 주변 상황과 과거의 궤적을 동시에 고려하는 경로 예측을 학습할 수 있는 구조를 제안한다.몇가지 네트워크 구조에 따른 성능 비교와 함께, 다양한 주행 환경에서 정량적인 평가와 정성적인 평가로 유효성을 입증한다.또한 보행자 경로예측 데이터에 테스트를 진행함으로써 타 알고리즘과의 성능을 비교한다.Recent autonomous driving research has shown remarkable and promising results. However, safe, sociable driving in an urban environment still has many challenges ahead. For realizing safe, interactive driving in complex alley scenario which shares a narrow area among traffic participants, It is essential to grasp each other's intention. Even in the same road environment, safe, and sociable driving policy may differ depending on the intention of the traffic participant agents around the ego vehicle. But understanding others intention and predicting their trajectories are complicated because each one basically considers multiple factorsroad environment, state of their surrounding traffic participants at the same time which realized as interaction. In this thesis dissertation, we propose a new trajectory prediction algorithm that considers all the information that each of the traffic participants would consider when they make a decision. By combining both each of history trajectories and grid map of surroundings as a latent vector representation, it predicts all the future trajectories of traffic participant agents around ego vehicle at once. This dissertation suggests two main module that fuses spatial and temporal information effectively. We verify the effectiveness of network structure by testing on the various driving scenario comparing with some network variants through quantitative and qualitative evaluation. Also, the proposed network is verified by applying it to public pedestrian trajectory prediction dataset to verify usability as a generalized methodology and to compare it with other SOTA algorithms.Abstract Contents List of Tables List of Figures 1 Introduction 1.1 Background and Motivation 2 Related Work 2.1 Contributions of the Dissertation 3 Conditional Neural Process 3.1 Conditional Neural Process(CNP) Overview 3.2 Trajectory Prediction with Scene Information as CNP 3.2.1 Formulation 3.2.2 Loss and Training Algorithm 4 Efficient Network Architecture for Intention Prediction 4.1 Network Overview 4.2 Trajectory Encoder 4.2.1 Spatio-Temporal Representation 4.3 Scene Feature Extraction 4.3.1 Side Spatial Extraction 4.4 Trajectory Decoder 5 Experiment 5.1 Driving environment dataset 5.1.1 Data acquisition method 5.1.2 Overview 5.1.3 Alley scenario 5.1.4 Urban Scenario 5.2 Public Pedestrian Dataset 6 ConclusionMaste

SOTIF Entropy: Online SOTIF Risk Quantification and Mitigation for Autonomous Driving

Autonomous driving confronts great challenges in complex traffic scenarios, where the risk of Safety of the Intended Functionality (SOTIF) can be triggered by the dynamic operational environment and system insufficiencies. The SOTIF risk is reflected not only intuitively in the collision risk with objects outside the autonomous vehicles (AVs), but also inherently in the performance limitation risk of the implemented algorithms themselves. How to minimize the SOTIF risk for autonomous driving is currently a critical, difficult, and unresolved issue. Therefore, this paper proposes the "Self-Surveillance and Self-Adaption System" as a systematic approach to online minimize the SOTIF risk, which aims to provide a systematic solution for monitoring, quantification, and mitigation of inherent and external risks. The core of this system is the risk monitoring of the implemented artificial intelligence algorithms within the AV. As a demonstration of the Self-Surveillance and Self-Adaption System, the risk monitoring of the perception algorithm, i.e., YOLOv5 is highlighted. Moreover, the inherent perception algorithm risk and external collision risk are jointly quantified via SOTIF entropy, which is then propagated downstream to the decision-making module and mitigated. Finally, several challenging scenarios are demonstrated, and the Hardware-in-the-Loop experiments are conducted to verify the efficiency and effectiveness of the system. The results demonstrate that the Self-Surveillance and Self-Adaption System enables dependable online monitoring, quantification, and mitigation of SOTIF risk in real-time critical traffic environments.Comment: 16 pages, 10 figures, 2 tables, submitted to IEEE TIT

Developing an advanced collision risk model for autonomous vehicles

Aiming at improving road safety, car manufacturers and researchers are verging upon autonomous vehicles. In recent years, collision prediction methods of autonomous vehicles have begun incorporating contextual information such as information about the traffic environment and the relative motion of other traffic participants but still fail to anticipate traffic scenarios of high complexity. During the past two decades, the problem of real-time collision prediction has also been investigated by traffic engineers. In the traffic engineering approach, a collision occurrence can potentially be predicted in real-time based on available data on traffic dynamics such as the average speed and flow of vehicles on a road segment. This thesis attempts to integrate vehicle-level collision prediction approaches for autonomous vehicles with network-level collision prediction, as studied by traffic engineers. [Continues.

Cooperative Perception for Social Driving in Connected Vehicle Traffic

The development of autonomous vehicle technology has moved to the center of automotive research in recent decades. In the foreseeable future, road vehicles at all levels of automation and connectivity will be required to operate safely in a hybrid traffic where human operated vehicles (HOVs) and fully and semi-autonomous vehicles (AVs) coexist. Having an accurate and reliable perception of the road is an important requirement for achieving this objective. This dissertation addresses some of the associated challenges via developing a human-like social driver model and devising a decentralized cooperative perception framework. A human-like driver model can aid the development of AVs by building an understanding of interactions among human drivers and AVs in a hybrid traffic, therefore facilitating an efficient and safe integration. The presented social driver model categorizes and defines the driver\u27s psychological decision factors in mathematical representations (target force, object force, and lane force). A model predictive control (MPC) is then employed for the motion planning by evaluating the prevailing social forces and considering the kinematics of the controlled vehicle as well as other operating constraints to ensure a safe maneuver in a way that mimics the predictive nature of the human driver\u27s decision making process. A hierarchical model predictive control structure is also proposed, where an additional upper level controller aggregates the social forces over a longer prediction horizon upon the availability of an extended perception of the upcoming traffic via vehicular networking. Based on the prediction of the upper level controller, a sequence of reference lanes is passed to a lower level controller to track while avoiding local obstacles. This hierarchical scheme helps reduce unnecessary lane changes resulting in smoother maneuvers. The dynamic vehicular communication environment requires a robust framework that must consistently evaluate and exploit the set of communicated information for the purpose of improving the perception of a participating vehicle beyond the limitations. This dissertation presents a decentralized cooperative perception framework that considers uncertainties in traffic measurements and allows scalability (for various settings of traffic density, participation rate, etc.). The framework utilizes a Bhattacharyya distance filter (BDF) for data association and a fast covariance intersection fusion scheme (FCI) for the data fusion processes. The conservatism of the covariance intersection fusion scheme is investigated in comparison to the traditional Kalman filter (KF), and two different fusion architectures: sensor-to-sensor and sensor-to-system track fusion are evaluated. The performance of the overall proposed framework is demonstrated via Monte Carlo simulations with a set of empirical communications models and traffic microsimulations where each connected vehicle asynchronously broadcasts its local perception consisting of estimates of the motion states of self and neighboring vehicles along with the corresponding uncertainty measures of the estimates. The evaluated framework includes a vehicle-to-vehicle (V2V) communication model that considers intermittent communications as well as a model that takes into account dynamic changes in an individual vehicle’s sensors’ FoV in accordance with the prevailing traffic conditions. The results show the presence of optimality in participation rate, where increasing participation rate beyond a certain level adversely affects the delay in packet delivery and the computational complexity in data association and fusion processes increase without a significant improvement in the achieved accuracy via the cooperative perception. In a highly dense traffic environment, the vehicular network can often be congested leading to limited bandwidth availability at high participation rates of the connected vehicles in the cooperative perception scheme. To alleviate the bandwidth utilization issues, an information-value discriminating networking scheme is proposed, where each sender broadcasts selectively chosen perception data based on the novelty-value of information. The potential benefits of these approaches include, but are not limited to, the reduction of bandwidth bottle-necking and the minimization of the computational cost of data association and fusion post processing of the shared perception data at receiving nodes. It is argued that the proposed information-value discriminating communication scheme can alleviate these adverse effects without sacrificing the fidelity of the perception

Probabilistic Framework for Behavior Characterization of Traffic Participants Enabling Long Term Prediction

This research aims at developing new methods that predict the behaviors of the human driven traffic participants to enable safe operation of autonomous vehicles in complex traffic environments. Autonomous vehicles are expected to operate amongst human driven conventional vehicles in the traffic at least for the next few decades. For safe navigation they will need to infer the intents as well as the behaviors of the human traffic participants using extrinsically observable information, so that their trajectories can be predicted for a time horizon long enough to do a predictive risk analysis and gracefully avert any risky situation. This research approaches this challenge by recognizing that any maneuver performed by a human driver can be divided into four stages that depend on the surrounding context: intent determination, maneuver preparation, gap acceptance and maneuver execution. It builds on the hypothesis that for a given driver, the behavior not only spans across these four maneuver stages, but across multiple maneuvers. As a result, identifying the driver behavior in any of these stages can help characterize the nature of all the subsequent maneuvers that the driver is likely to perform, thus resulting in a more accurate prediction for a longer time horizon. To enable this, a novel probabilistic framework is proposed that couples the different maneuver stages of the observed traffic participant together and associates them to a driving style. To realize this framework two candidate Multiple Model Adaptive Estimation approaches were compared: Autonomous Multiple Model (AMM) and Interacting Multiple Model(IMM) filtering approach. The IMM approach proved superior to the AMM approach and was eventually validated using a trajectory extracted from a real world dataset for efficacy. The proposed framework was then implemented by extending the validated IMM approach with contextual information of the observed traffic participant. The classification of the driving style of the traffic participant (behavior characterization) was then demonstrated for two use case scenarios. The proposed contextual IMM (CIMM) framework also showed improvements in the performance of the behavior classification of the traffic participants compared to the IMM for the identified use case scenarios. This outcome warrants further exploration of this framework for different traffic scenarios. Further, it contributes towards the ongoing endeavors for safe deployment of autonomous vehicles on public roads

Fusion of Data from Heterogeneous Sensors with Distributed Fields of View and Situation Evaluation for Advanced Driver Assistance Systems

In order to develop a driver assistance system for pedestrian protection, pedestrians in the environment of a truck are detected by radars and a camera and are tracked across distributed fields of view using a Joint Integrated Probabilistic Data Association filter. A robust approach for prediction of the system vehicles trajectory is presented. It serves the computation of a probabilistic collision risk based on reachable sets where different sources of uncertainty are taken into account

Intelligent and High-Performance Behavior Design of Autonomous Systems via Learning, Optimization and Control

Nowadays, great societal demands have rapidly boosted the development of autonomous systems that densely interact with humans in many application domains, from manufacturing to transportation and from workplaces to daily lives. The shift from isolated working environments to human-dominated space requires autonomous systems to be empowered to handle not only environmental uncertainties such as external vibrations but also interaction uncertainties arising from human behavior which is in nature probabilistic, causal but not strictly rational, internally hierarchical and socially compliant.This dissertation is concerned with the design of intelligent and high-performance behavior of such autonomous systems, leveraging the strength from control, optimization, learning, and cognitive science. The work consists of two parts. In Part I, the problem of high-level hybrid human-machine behavior design is addressed. The goal is to achieve safe, efficient and human-like interaction with people. A framework based on the theory of mind, utility theories and imitation learning is proposed to efficiently represent and learn the complicated behavior of humans. Built upon that, machine behaviors at three different levels - the perceptual level, the reasoning level, and the action level - are designed via imitation learning, optimization, and online adaptation, allowing the system to interpret, reason and behave as human, particularly when a variety of uncertainties exist. Applications to autonomous driving are considered throughout Part I. Part II is concerned with the design of high-performance low-level individual machine behavior in the presence of model uncertainties and external disturbances. Advanced control laws based on adaptation, iterative learning and the internal structures of uncertainties/disturbances are developed to assure that the high-level interactive behaviors can be reliably executed. Applications on robot manipulators and high-precision motion systems are discussed in this part

AI-based framework for automatically extracting high-low features from NDS data to understand driver behavior

Our ability to detect and characterize unsafe driving behaviors in naturalistic driving environments and associate them with road crashes will be a significant step toward developing effective crash countermeasures. Due to some limitations, researchers have not yet fully achieved the stated goal of characterizing unsafe driving behaviors. These limitations include, but are not limited to, the high cost of data collection and the manual processes required to extract information from NDS data. In light of this limitations, the primary objective of this study is to develop an artificial intelligence (AI) framework for automatically extracting high-low features from the NDS dataset to explain driver behavior using a low-cost data collection method. The author proposed three novel objectives for achieving the study's objective in light of the identified research gaps. Initially, the study develops a low-cost data acquisition system for gathering data on naturalistic driving. Second, the study develops a framework that automatically extracts high- to low-level features, such as vehicle density, turning movements, and lane changes, from the data collected by the developed data acquisition system. Thirdly, the study extracted information from the NDS data to gain a better understanding of people's car-following behavior and other driving behaviors in order to develop countermeasures for traffic safety through data collection and analysis. The first objective of this study is to develop a multifunctional smartphone application for collecting NDS data. Three major modules comprised the designed app: a front-end user interface module, a sensor module, and a backend module. The front-end, which is also the application's user interface, was created to provide a streamlined view that exposed the application's key features via a tab bar controller. This allows us to compartmentalize the application's critical components into separate views. The backend module provides computational resources that can be used to accelerate front-end query responses. Google Firebase powered the backend of the developed application. The sensor modules included CoreMotion, CoreLocation, and AVKit. CoreMotion collects motion and environmental data from the onboard hardware of iOS devices, including accelerometers, gyroscopes, pedometers, magnetometers, and barometers. In contrast, CoreLocation determines the altitude, orientation, and geographical location of a device, as well as its position relative to an adjacent iBeacon device. The AVKit finally provides a high-level interface for video content playback. To achieve objective two, we formulated the problem as both a classification and time-series segmentation problem. This is due to the fact that the majority of existing driver maneuver detection methods formulate the problem as a pure classification problem, assuming a discretized input signal with known start and end locations for each event or segment. In practice, however, vehicle telemetry data used for detecting driver maneuvers are continuous; thus, a fully automated driver maneuver detection system should incorporate solutions for both time series segmentation and classification. The five stages of our proposed methodology are as follows: 1) data preprocessing, 2) segmentation of events, 3) machine learning classification, 4) heuristics classification, and 5) frame-by-frame video annotation. The result of the study indicates that the gyroscope reading is an exceptional parameter for extracting driving events, as its accuracy was consistent across all four models developed. The study reveals that the Energy Maximization Algorithm's accuracy ranges from 56.80 percent (left lane change) to 85.20 percent (right lane change) (lane-keeping) All four models developed had comparable accuracies to studies that used similar models. The 1D-CNN model had the highest accuracy (98.99 percent), followed by the LSTM model (97.75 percent), the RF model (97.71 percent), and the SVM model (97.65 percent). To serve as a ground truth, continuous signal data was annotated. In addition, the proposed method outperformed the fixed time window approach. The study analyzed the overall pipeline's accuracy by penalizing the F1 scores of the ML models with the EMA's duration score. The pipeline's accuracy ranged between 56.8 percent and 85.0 percent overall. The ultimate goal of this study was to extract variables from naturalistic driving videos that would facilitate an understanding of driver behavior in a naturalistic driving environment. To achieve this objective, three sub-goals were established. First, we developed a framework for extracting features pertinent to comprehending the behavior of natural-environment drivers. Using the extracted features, we then analyzed the car-following behaviors of various demographic groups. Thirdly, using a machine learning algorithm, we modeled the acceleration of both the ego-vehicle and the leading vehicle. Younger drivers are more likely to be aggressive, according to the findings of this study. In addition, the study revealed that drivers tend to accelerate when the distance between them and the vehicle in front of them is substantial. Lastly, compared to younger drivers, elderly motorists maintain a significantly larger following distance. This study's results have numerous safety implications. First, the analysis of the driving behavior of different demographic groups will enable safety engineers to develop the most effective crash countermeasures by enhancing their understanding of the driving styles of different demographic groups and the causes of collisions. Second, the models developed to predict the acceleration of both the ego-vehicle and the leading vehicle will provide enough information to explain the behavior of the ego-driver.Includes bibliographical references