Human Motion Trajectory Prediction: A Survey

With growing numbers of intelligent autonomous systems in human environments, the ability of such systems to perceive, understand and anticipate human behavior becomes increasingly important. Specifically, predicting future positions of dynamic agents and planning considering such predictions are key tasks for self-driving vehicles, service robots and advanced surveillance systems. This paper provides a survey of human motion trajectory prediction. We review, analyze and structure a large selection of work from different communities and propose a taxonomy that categorizes existing methods based on the motion modeling approach and level of contextual information used. We provide an overview of the existing datasets and performance metrics. We discuss limitations of the state of the art and outline directions for further research.Comment: Submitted to the International Journal of Robotics Research (IJRR), 37 page

자율 주행을 위한 학습 기반의 다중 교통 참여자 경로예측 방법

학위논문 (석사)-- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2019. 2. 서승우.자율주행차량이 안전하면서도, 교통 흐름을 방해하지 않는 인간 수준의 interactive한 주행을 실현하기 위해서는 주변 운전자와 보행자를 포함한 교통 참여자들의 의도를 파악하는 것이 필수적이다. 동일한 도로 환경에서도 주변의 교통 참여자들이 어떤 의도를 가지고 이동하고 있는가에 따라서 적합한 주행 전략은 매번 달라진다. 특히 좁은 공간을 많은 에이전트들이 공유하고 있는 골목길 상황에서는 각 에이전트들의 선택은 제한되지만, 각 에이전트들의 의사결정과정은 그들간의 상호작용이 고려되어 매우 복잡하다.이런 상황에 맞는 경로예측은 과거의 궤적, 현재 인식하고 있는 도로 환경, 주변 교통 참여자의 상태 등을 고려하여 수행되어야 한다. 또한 일반화된 대부분의 환경에서 수행되려면, 빠른 러닝 타임과 동시에 고려하는 교통 참여자의 숫자에 대한 제약이 적어야 한다. 이 논문에서는 각 교통 참여자들의 과거 궤적과 현재 위치 상황, 도로 상황을 동시에 고려하여 모든 참여자들의 상호작용이 고려된 경로예측 방법을 제안한다. 각 물체들의 과거 궤적과 동시에 자율주행 차량이 인식하고 있는 도로와 주변 물체를 입력으로하고, 이 두개의 임베딩 결과를 혼합하여 모든 물체들에 대한 경로 예측을 동시에 수행한다. 이 과정에서 네트워크 구조 내부에서 각각 물체들의 임베딩 결과를 위치 정보에 매칭시킴으로써 효과적으로 주변 상황과 과거의 궤적을 동시에 고려하는 경로 예측을 학습할 수 있는 구조를 제안한다.몇가지 네트워크 구조에 따른 성능 비교와 함께, 다양한 주행 환경에서 정량적인 평가와 정성적인 평가로 유효성을 입증한다.또한 보행자 경로예측 데이터에 테스트를 진행함으로써 타 알고리즘과의 성능을 비교한다.Recent autonomous driving research has shown remarkable and promising results. However, safe, sociable driving in an urban environment still has many challenges ahead. For realizing safe, interactive driving in complex alley scenario which shares a narrow area among traffic participants, It is essential to grasp each other's intention. Even in the same road environment, safe, and sociable driving policy may differ depending on the intention of the traffic participant agents around the ego vehicle. But understanding others intention and predicting their trajectories are complicated because each one basically considers multiple factorsroad environment, state of their surrounding traffic participants at the same time which realized as interaction. In this thesis dissertation, we propose a new trajectory prediction algorithm that considers all the information that each of the traffic participants would consider when they make a decision. By combining both each of history trajectories and grid map of surroundings as a latent vector representation, it predicts all the future trajectories of traffic participant agents around ego vehicle at once. This dissertation suggests two main module that fuses spatial and temporal information effectively. We verify the effectiveness of network structure by testing on the various driving scenario comparing with some network variants through quantitative and qualitative evaluation. Also, the proposed network is verified by applying it to public pedestrian trajectory prediction dataset to verify usability as a generalized methodology and to compare it with other SOTA algorithms.Abstract Contents List of Tables List of Figures 1 Introduction 1.1 Background and Motivation 2 Related Work 2.1 Contributions of the Dissertation 3 Conditional Neural Process 3.1 Conditional Neural Process(CNP) Overview 3.2 Trajectory Prediction with Scene Information as CNP 3.2.1 Formulation 3.2.2 Loss and Training Algorithm 4 Efficient Network Architecture for Intention Prediction 4.1 Network Overview 4.2 Trajectory Encoder 4.2.1 Spatio-Temporal Representation 4.3 Scene Feature Extraction 4.3.1 Side Spatial Extraction 4.4 Trajectory Decoder 5 Experiment 5.1 Driving environment dataset 5.1.1 Data acquisition method 5.1.2 Overview 5.1.3 Alley scenario 5.1.4 Urban Scenario 5.2 Public Pedestrian Dataset 6 ConclusionMaste

FollowMe: Vehicle Behaviour Prediction in Autonomous Vehicle Settings

An ego vehicle following a virtual lead vehicle planned route is an essential component when autonomous and non-autonomous vehicles interact. Yet, there is a question about the driver's ability to follow the planned lead vehicle route. Thus, predicting the trajectory of the ego vehicle route given a lead vehicle route is of interest. We introduce a new dataset, the FollowMe dataset, which offers a motion and behavior prediction problem by answering the latter question of the driver's ability to follow a lead vehicle. We also introduce a deep spatio-temporal graph model FollowMe-STGCNN as a baseline for the dataset. In our experiments and analysis, we show the design benefits of FollowMe-STGCNN in capturing the interactions that lie within the dataset. We contrast the performance of FollowMe-STGCNN with prior motion prediction models showing the need to have a different design mechanism to address the lead vehicle following settings

Long-term future prediction under uncertainty and multi-modality

Humans have an innate ability to excel at activities that involve prediction of complex object dynamics such as predicting the possible trajectory of a billiard ball after it has been hit by the player or the prediction of motion of pedestrians while on the road. A key feature that enables humans to perform such tasks is anticipation. There has been continuous research in the area of Computer Vision and Artificial Intelligence to mimic this human ability for autonomous agents to succeed in the real world scenarios. Recent advances in the field of deep learning and the availability of large scale datasets has enabled the pursuit of fully autonomous agents with complex decision making abilities such as self-driving vehicles or robots. One of the main challenges encompassing the deployment of these agents in the real world is their ability to perform anticipation tasks with at least human level efficiency. To advance the field of autonomous systems, particularly, self-driving agents, in this thesis, we focus on the task of future prediction in diverse real world settings, ranging from deterministic scenarios such as prediction of paths of balls on a billiard table to the predicting the future of non-deterministic street scenes. Specifically, we identify certain core challenges for long-term future prediction: long-term prediction, uncertainty, multi-modality, and exact inference. To address these challenges, this thesis makes the following core contributions. Firstly, for accurate long-term predictions, we develop approaches that effectively utilize available observed information in the form of image boundaries in videos or interactions in street scenes. Secondly, as uncertainty increases into the future in case of non-deterministic scenarios, we leverage Bayesian inference frameworks to capture calibrated distributions of likely future events. Finally, to further improve performance in highly-multimodal non-deterministic scenarios such as street scenes, we develop deep generative models based on conditional variational autoencoders as well as normalizing flow based exact inference methods. Furthermore, we introduce a novel dataset with dense pedestrian-vehicle interactions to further aid the development of anticipation methods for autonomous driving applications in urban environments.Menschen haben die angeborene Fähigkeit, Vorgänge mit komplexer Objektdynamik vorauszusehen, wie z. B. die Vorhersage der möglichen Flugbahn einer Billardkugel, nachdem sie vom Spieler gestoßen wurde, oder die Vorhersage der Bewegung von Fußgängern auf der Straße. Eine Schlüsseleigenschaft, die es dem Menschen ermöglicht, solche Aufgaben zu erfüllen, ist die Antizipation. Im Bereich der Computer Vision und der Künstlichen Intelligenz wurde kontinuierlich daran geforscht, diese menschliche Fähigkeit nachzuahmen, damit autonome Agenten in der realen Welt erfolgreich sein können. Jüngste Fortschritte auf dem Gebiet des Deep Learning und die Verfügbarkeit großer Datensätze haben die Entwicklung vollständig autonomer Agenten mit komplexen Entscheidungsfähigkeiten wie selbstfahrende Fahrzeugen oder Roboter ermöglicht. Eine der größten Herausforderungen beim Einsatz dieser Agenten in der realen Welt ist ihre Fähigkeit, Antizipationsaufgaben mit einer Effizienz durchzuführen, die mindestens der menschlichen entspricht. Um das Feld der autonomen Systeme, insbesondere der selbstfahrenden Agenten, voranzubringen, konzentrieren wir uns in dieser Arbeit auf die Aufgabe der Zukunftsvorhersage in verschiedenen realen Umgebungen, die von deterministischen Szenarien wie der Vorhersage der Bahnen von Kugeln auf einem Billardtisch bis zur Vorhersage der Zukunft von nicht-deterministischen Straßenszenen reichen. Insbesondere identifizieren wir bestimmte grundlegende Herausforderungen für langfristige Zukunftsvorhersagen: Langzeitvorhersage, Unsicherheit, Multimodalität und exakte Inferenz. Um diese Herausforderungen anzugehen, leistet diese Arbeit die folgenden grundlegenden Beiträge. Erstens: Für genaue Langzeitvorhersagen entwickeln wir Ansätze, die verfügbare Beobachtungsinformationen in Form von Bildgrenzen in Videos oder Interaktionen in Straßenszenen effektiv nutzen. Zweitens: Da die Unsicherheit in der Zukunft bei nicht-deterministischen Szenarien zunimmt, nutzen wir Bayes’sche Inferenzverfahren, um kalibrierte Verteilungen wahrscheinlicher zukünftiger Ereignisse zu erfassen. Drittens: Um die Leistung in hochmultimodalen, nichtdeterministischen Szenarien wie Straßenszenen weiter zu verbessern, entwickeln wir tiefe generative Modelle, die sowohl auf konditionalen Variations-Autoencodern als auch auf normalisierenden fließenden exakten Inferenzmethoden basieren. Darüber hinaus stellen wir einen neuartigen Datensatz mit dichten Fußgänger-Fahrzeug- Interaktionen vor, um Antizipationsmethoden für autonome Fahranwendungen in urbanen Umgebungen weiter zu entwickeln

A system for learning statistical motion patterns

Analysis of motion patterns is an effective approach for anomaly detection and behavior prediction. Current approaches for the analysis of motion patterns depend on known scenes, where objects move in predefined ways. It is highly desirable to automatically construct object motion patterns which reflect the knowledge of the scene. In this paper, we present a system for automatically learning motion patterns for anomaly detection and behavior prediction based on a proposed algorithm for robustly tracking multiple objects. In the tracking algorithm, foreground pixels are clustered using a fast accurate fuzzy k-means algorithm. Growing and prediction of the cluster centroids of foreground pixels ensure that each cluster centroid is associated with a moving object in the scene. In the algorithm for learning motion patterns, trajectories are clustered hierarchically using spatial and temporal information and then each motion pattern is represented with a chain of Gaussian distributions. Based on the learned statistical motion patterns, statistical methods are used to detect anomalies and predict behaviors. Our system is tested using image sequences acquired, respectively, from a crowded real traffic scene and a model traffic scene. Experimental results show the robustness of the tracking algorithm, the efficiency of the algorithm for learning motion patterns, and the encouraging performance of algorithms for anomaly detection and behavior prediction