Human Motion Trajectory Prediction: A Survey

With growing numbers of intelligent autonomous systems in human environments, the ability of such systems to perceive, understand and anticipate human behavior becomes increasingly important. Specifically, predicting future positions of dynamic agents and planning considering such predictions are key tasks for self-driving vehicles, service robots and advanced surveillance systems. This paper provides a survey of human motion trajectory prediction. We review, analyze and structure a large selection of work from different communities and propose a taxonomy that categorizes existing methods based on the motion modeling approach and level of contextual information used. We provide an overview of the existing datasets and performance metrics. We discuss limitations of the state of the art and outline directions for further research.Comment: Submitted to the International Journal of Robotics Research (IJRR), 37 page

자율 주행을 위한 학습 기반의 다중 교통 참여자 경로예측 방법

학위논문 (석사)-- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2019. 2. 서승우.자율주행차량이 안전하면서도, 교통 흐름을 방해하지 않는 인간 수준의 interactive한 주행을 실현하기 위해서는 주변 운전자와 보행자를 포함한 교통 참여자들의 의도를 파악하는 것이 필수적이다. 동일한 도로 환경에서도 주변의 교통 참여자들이 어떤 의도를 가지고 이동하고 있는가에 따라서 적합한 주행 전략은 매번 달라진다. 특히 좁은 공간을 많은 에이전트들이 공유하고 있는 골목길 상황에서는 각 에이전트들의 선택은 제한되지만, 각 에이전트들의 의사결정과정은 그들간의 상호작용이 고려되어 매우 복잡하다.이런 상황에 맞는 경로예측은 과거의 궤적, 현재 인식하고 있는 도로 환경, 주변 교통 참여자의 상태 등을 고려하여 수행되어야 한다. 또한 일반화된 대부분의 환경에서 수행되려면, 빠른 러닝 타임과 동시에 고려하는 교통 참여자의 숫자에 대한 제약이 적어야 한다. 이 논문에서는 각 교통 참여자들의 과거 궤적과 현재 위치 상황, 도로 상황을 동시에 고려하여 모든 참여자들의 상호작용이 고려된 경로예측 방법을 제안한다. 각 물체들의 과거 궤적과 동시에 자율주행 차량이 인식하고 있는 도로와 주변 물체를 입력으로하고, 이 두개의 임베딩 결과를 혼합하여 모든 물체들에 대한 경로 예측을 동시에 수행한다. 이 과정에서 네트워크 구조 내부에서 각각 물체들의 임베딩 결과를 위치 정보에 매칭시킴으로써 효과적으로 주변 상황과 과거의 궤적을 동시에 고려하는 경로 예측을 학습할 수 있는 구조를 제안한다.몇가지 네트워크 구조에 따른 성능 비교와 함께, 다양한 주행 환경에서 정량적인 평가와 정성적인 평가로 유효성을 입증한다.또한 보행자 경로예측 데이터에 테스트를 진행함으로써 타 알고리즘과의 성능을 비교한다.Recent autonomous driving research has shown remarkable and promising results. However, safe, sociable driving in an urban environment still has many challenges ahead. For realizing safe, interactive driving in complex alley scenario which shares a narrow area among traffic participants, It is essential to grasp each other's intention. Even in the same road environment, safe, and sociable driving policy may differ depending on the intention of the traffic participant agents around the ego vehicle. But understanding others intention and predicting their trajectories are complicated because each one basically considers multiple factorsroad environment, state of their surrounding traffic participants at the same time which realized as interaction. In this thesis dissertation, we propose a new trajectory prediction algorithm that considers all the information that each of the traffic participants would consider when they make a decision. By combining both each of history trajectories and grid map of surroundings as a latent vector representation, it predicts all the future trajectories of traffic participant agents around ego vehicle at once. This dissertation suggests two main module that fuses spatial and temporal information effectively. We verify the effectiveness of network structure by testing on the various driving scenario comparing with some network variants through quantitative and qualitative evaluation. Also, the proposed network is verified by applying it to public pedestrian trajectory prediction dataset to verify usability as a generalized methodology and to compare it with other SOTA algorithms.Abstract Contents List of Tables List of Figures 1 Introduction 1.1 Background and Motivation 2 Related Work 2.1 Contributions of the Dissertation 3 Conditional Neural Process 3.1 Conditional Neural Process(CNP) Overview 3.2 Trajectory Prediction with Scene Information as CNP 3.2.1 Formulation 3.2.2 Loss and Training Algorithm 4 Efficient Network Architecture for Intention Prediction 4.1 Network Overview 4.2 Trajectory Encoder 4.2.1 Spatio-Temporal Representation 4.3 Scene Feature Extraction 4.3.1 Side Spatial Extraction 4.4 Trajectory Decoder 5 Experiment 5.1 Driving environment dataset 5.1.1 Data acquisition method 5.1.2 Overview 5.1.3 Alley scenario 5.1.4 Urban Scenario 5.2 Public Pedestrian Dataset 6 ConclusionMaste

Effect of Pedestrian and Crowds on Vehicle Motion and Traffic Flow

69A3551747111Vehicle-pedestrian interactions in shared spaces represents a complex safety problem. Ideally, the vehicle must react safely to any pedestrian behavior, while the pedestrian behavior itself can be very complex and unpredictable. To emphasize this safety problem, in a 2019 Traffic Safety Facts report by the National Highway Traffic Safety Administration (NHTSA) [1], it was shown that the percentage of pedestrian fatalities was increasing from 2008 to 2017, even as advanced driving assist systems (ADASs) were being developed and deployed

Pedestrian Models for Autonomous Driving Part II: High-Level Models of Human Behavior

Abstract—Autonomous vehicles (AVs) must share space with pedestrians, both in carriageway cases such as cars at pedestrian crossings and off-carriageway cases such as delivery vehicles navigating through crowds on pedestrianized high-streets. Unlike static obstacles, pedestrians are active agents with complex, inter- active motions. Planning AV actions in the presence of pedestrians thus requires modelling of their probable future behaviour as well as detecting and tracking them. This narrative review article is Part II of a pair, together surveying the current technology stack involved in this process, organising recent research into a hierarchical taxonomy ranging from low-level image detection to high-level psychological models, from the perspective of an AV designer. This self-contained Part II covers the higher levels of this stack, consisting of models of pedestrian behaviour, from prediction of individual pedestrians’ likely destinations and paths, to game-theoretic models of interactions between pedestrians and autonomous vehicles. This survey clearly shows that, although there are good models for optimal walking behaviour, high-level psychological and social modelling of pedestrian behaviour still remains an open research question that requires many conceptual issues to be clarified. Early work has been done on descriptive and qualitative models of behaviour, but much work is still needed to translate them into quantitative algorithms for practical AV control

RS2G: Data-Driven Scene-Graph Extraction and Embedding for Robust Autonomous Perception and Scenario Understanding

Human drivers naturally reason about interactions between road users to understand and safely navigate through traffic. Thus, developing autonomous vehicles necessitates the ability to mimic such knowledge and model interactions between road users to understand and navigate unpredictable, dynamic environments. However, since real-world scenarios often differ from training datasets, effectively modeling the behavior of various road users in an environment remains a significant research challenge. This reality necessitates models that generalize to a broad range of domains and explicitly model interactions between road users and the environment to improve scenario understanding. Graph learning methods address this problem by modeling interactions using graph representations of scenarios. However, existing methods cannot effectively transfer knowledge gained from the training domain to real-world scenarios. This constraint is caused by the domain-specific rules used for graph extraction that can vary in effectiveness across domains, limiting generalization ability. To address these limitations, we propose RoadScene2Graph (RS2G): a data-driven graph extraction and modeling approach that learns to extract the best graph representation of a road scene for solving autonomous scene understanding tasks. We show that RS2G enables better performance at subjective risk assessment than rule-based graph extraction methods and deep-learning-based models. RS2G also improves generalization and Sim2Real transfer learning, which denotes the ability to transfer knowledge gained from simulation datasets to unseen real-world scenarios. We also present ablation studies showing how RS2G produces a more useful graph representation for downstream classifiers. Finally, we show how RS2G can identify the relative importance of rule-based graph edges and enables intelligent graph sparsity tuning

Review of graph-based hazardous event detection methods for autonomous driving systems

Automated and autonomous vehicles are often required to operate in complex road environments with potential hazards that may lead to hazardous events causing injury or even death. Therefore, a reliable autonomous hazardous event detection system is a key enabler for highly autonomous vehicles (e.g., Level 4 and 5 autonomous vehicles) to operate without human supervision for significant periods of time. One promising solution to the problem is the use of graph-based methods that are powerful tools for relational reasoning. Using graphs to organise heterogeneous knowledge about the operational environment, link scene entities (e.g., road users, static objects, traffic rules) and describe how they affect each other. Due to a growing interest and opportunity presented by graph-based methods for autonomous hazardous event detection, this paper provides a comprehensive review of the state-of-the-art graph-based methods that we categorise as rule-based, probabilistic, and machine learning-driven. Additionally, we present an in-depth overview of the available datasets to facilitate hazardous event training and evaluation metrics to assess model performance. In doing so, we aim to provide a thorough overview and insight into the key research opportunities and open challenges

Driver Attention based on Deep Learning for a Smart Vehicle to Driver (V2D) Interaction

La atención del conductor es un tópico interesante dentro del mundo de los vehículos inteligentes para la consecución de tareas que van desde la monitorización del conductor hasta la conducción autónoma. Esta tesis aborda este tópico basándose en algoritmos de aprendizaje profundo para conseguir una interacción inteligente entre el vehículo y el conductor. La monitorización del conductor requiere una estimación precisa de su mirada en un entorno 3D para conocer el estado de su atención. En esta tesis se aborda este problema usando una única cámara, para que pueda ser utilizada en aplicaciones reales, sin un alto coste y sin molestar al conductor. La herramienta desarrollada ha sido evaluada en una base de datos pública (DADA2000), obteniendo unos resultados similares a los obtenidos mediante un seguidor de ojos caro que no puede ser usado en un vehículo real. Además, ha sido usada en una aplicación que evalúa la atención del conductor en la transición de modo autónomo a manual de forma simulada, proponiendo el uso de una métrica novedosa para conocer el estado de la situación del conductor en base a su atención sobre los diferentes objetos de la escena. Por otro lado, se ha propuesto un algoritmo de estimación de atención del conductor, utilizando las últimas técnicas de aprendizaje profundo como son las conditional Generative Adversarial Networks (cGANs) y el Multi-Head Self-Attention. Esto permite enfatizar ciertas zonas de la escena al igual que lo haría un humano. El modelo ha sido entrenado y validado en dos bases de datos públicas (BDD-A y DADA2000) superando a otras propuestas del estado del arte y consiguiendo unos tiempos de inferencia que permiten su uso en aplicaciones reales. Por último, se ha desarrollado un modelo que aprovecha nuestro algoritmo de atención del conductor para comprender una escena de tráfico obteniendo la decisión tomada por el vehículo y su explicación, en base a las imágenes tomadas por una cámara situada en la parte frontal del vehículo. Ha sido entrenado en una base de datos pública (BDD-OIA) proponiendo un modelo que entiende la secuencia temporal de los eventos usando un Transformer Encoder, consiguiendo superar a otras propuestas del estado del arte. Además de su validación en la base de datos, ha sido implementado en una aplicación que interacciona con el conductor aconsejando sobre las decisiones a tomar y sus explicaciones ante diferentes casos de uso en un entorno simulado. Esta tesis explora y demuestra los beneficios de la atención del conductor para el mundo de los vehículos inteligentes, logrando una interacción vehículo conductor a través de las últimas técnicas de aprendizaje profundo