Human Motion Trajectory Prediction: A Survey

With growing numbers of intelligent autonomous systems in human environments, the ability of such systems to perceive, understand and anticipate human behavior becomes increasingly important. Specifically, predicting future positions of dynamic agents and planning considering such predictions are key tasks for self-driving vehicles, service robots and advanced surveillance systems. This paper provides a survey of human motion trajectory prediction. We review, analyze and structure a large selection of work from different communities and propose a taxonomy that categorizes existing methods based on the motion modeling approach and level of contextual information used. We provide an overview of the existing datasets and performance metrics. We discuss limitations of the state of the art and outline directions for further research.Comment: Submitted to the International Journal of Robotics Research (IJRR), 37 page

Design of a cognitive neural predictive controller for mobile robot

This thesis was submitted for the degree of Doctor of Philosophy and was awarded by Brunel UniversityIn this thesis, a cognitive neural predictive controller system has been designed to guide a nonholonomic wheeled mobile robot during continuous and non-continuous trajectory tracking and to navigate through static obstacles with collision-free and minimum tracking error. The structure of the controller consists of two layers; the first layer is a neural network system that controls the mobile robot actuators in order to track a desired path. The second layer of the controller is cognitive layer that collects information from the environment and plans the optimal path. In addition to this, it detects if there is any obstacle in the path so it can be avoided by re-planning the trajectory using particle swarm optimisation (PSO) technique. Two neural networks models are used: the first model is modified Elman recurrent neural network model that describes the kinematic and dynamic model of the mobile robot and it is trained off-line and on-line stages to guarantee that the outputs of the model will accurately represent the actual outputs of the mobile robot system. The trained neural model acts as the position and orientation identifier. The second model is feedforward multi-layer perceptron neural network that describes a feedforward neural controller and it is trained off-line and its weights are adapted on-line to find the reference torques, which controls the steady-state outputs of the mobile robot system. The feedback neural controller is based on the posture neural identifier and quadratic performance index predictive optimisation algorithm for N step-ahead prediction in order to find the optimal torque action in the transient to stabilise the tracking error of the mobile robot system when the trajectory of the robot is drifted from the desired path during transient state. Three controller methodologies were developed: the first is the feedback neural controller; the second is the nonlinear PID neural feedback controller and the third is nonlinear inverse dynamic neural feedback controller, based on the back-stepping method and Lyapunov criterion. The main advantages of the presented approaches are to plan an optimal path for itself avoiding obstructions by using intelligent (PSO) technique as well as the analytically derived control law, which has significantly high computational accuracy with predictive optimisation technique to obtain the optimal torques control action and lead to minimum tracking error of the mobile robot for different types of trajectories. The proposed control algorithm has been applied to monitor a nonholonomic wheeled mobile robot, has demonstrated the capability of tracking different trajectories with continuous gradients (lemniscates and circular) or non-continuous gradients (square) with bounded external disturbances and static obstacles. Simulations results and experimental work showed the effectiveness of the proposed cognitive neural predictive control algorithm; this is demonstrated by the minimised tracking error to less than (1 cm) and obtained smoothness of the torque control signal less than maximum torque (0.236 N.m), especially when external disturbances are applied and navigating through static obstacles. Results show that the five steps-ahead prediction algorithm has better performance compared to one step-ahead for all the control methodologies because of a more complex control structure and taking into account future values of the desired one, not only the current value, as with one step-ahead method. The mean-square error method is used for each component of the state error vector to compare between each of the performance control methodologies in order to give better control results

Multi-agent Collision Avoidance Using Interval Analysis and Symbolic Modelling with its Application to the Novel Polycopter

Coordination is fundamental component of autonomy when a system is defined by multiple mobile agents. For unmanned aerial systems (UAS), challenges originate from their low-level systems, such as their flight dynamics, which are often complex. The thesis begins by examining these low-level dynamics in an analysis of several well known UAS using a novel symbolic component-based framework. It is shown how this approach is used effectively to define key model and performance properties necessary of UAS trajectory control. This is demonstrated initially under the context of linear quadratic regulation (LQR) and model predictive control (MPC) of a quadcopter. The symbolic framework is later extended in the proposal of a novel UAS platform, referred to as the ``Polycopter" for its morphing nature. This dual-tilt axis system has unique authority over is thrust vector, in addition to an ability to actively augment its stability and aerodynamic characteristics. This presents several opportunities in exploitative control design. With an approach to low-level UAS modelling and control proposed, the focus of the thesis shifts to investigate the challenges associated with local trajectory generation for the purpose of multi-agent collision avoidance. This begins with a novel survey of the state-of-the-art geometric approaches with respect to performance, scalability and tolerance to uncertainty. From this survey, the interval avoidance (IA) method is proposed, to incorporate trajectory uncertainty in the geometric derivation of escape trajectories. The method is shown to be more effective in ensuring safe separation in several of the presented conditions, however performance is shown to deteriorate in denser conflicts. Finally, it is shown how by re-framing the IA problem, three dimensional (3D) collision avoidance is achieved. The novel 3D IA method is shown to out perform the original method in three conflict cases by maintaining separation under the effects of uncertainty and in scenarios with multiple obstacles. The performance, scalability and uncertainty tolerance of each presented method is then examined in a set of scenarios resembling typical coordinated UAS operations in an exhaustive Monte-Carlo analysis

Advances in Robot Navigation

Robot navigation includes different interrelated activities such as perception - obtaining and interpreting sensory information; exploration - the strategy that guides the robot to select the next direction to go; mapping - the construction of a spatial representation by using the sensory information perceived; localization - the strategy to estimate the robot position within the spatial map; path planning - the strategy to find a path towards a goal location being optimal or not; and path execution, where motor actions are determined and adapted to environmental changes. This book integrates results from the research work of authors all over the world, addressing the abovementioned activities and analyzing the critical implications of dealing with dynamic environments. Different solutions providing adaptive navigation are taken from nature inspiration, and diverse applications are described in the context of an important field of study: social robotics

Mobile Robots Navigation

Mobile robots navigation includes different interrelated activities: (i) perception, as obtaining and interpreting sensory information; (ii) exploration, as the strategy that guides the robot to select the next direction to go; (iii) mapping, involving the construction of a spatial representation by using the sensory information perceived; (iv) localization, as the strategy to estimate the robot position within the spatial map; (v) path planning, as the strategy to find a path towards a goal location being optimal or not; and (vi) path execution, where motor actions are determined and adapted to environmental changes. The book addresses those activities by integrating results from the research work of several authors all over the world. Research cases are documented in 32 chapters organized within 7 categories next described

Computational Methods for Cognitive and Cooperative Robotics

In the last decades design methods in control engineering made substantial progress in the areas of robotics and computer animation. Nowadays these methods incorporate the newest developments in machine learning and artificial intelligence. But the problems of flexible and online-adaptive combinations of motor behaviors remain challenging for human-like animations and for humanoid robotics. In this context, biologically-motivated methods for the analysis and re-synthesis of human motor programs provide new insights in and models for the anticipatory motion synthesis. This thesis presents the author’s achievements in the areas of cognitive and developmental robotics, cooperative and humanoid robotics and intelligent and machine learning methods in computer graphics. The first part of the thesis in the chapter “Goal-directed Imitation for Robots” considers imitation learning in cognitive and developmental robotics. The work presented here details the author’s progress in the development of hierarchical motion recognition and planning inspired by recent discoveries of the functions of mirror-neuron cortical circuits in primates. The overall architecture is capable of ‘learning for imitation’ and ‘learning by imitation’. The complete system includes a low-level real-time capable path planning subsystem for obstacle avoidance during arm reaching. The learning-based path planning subsystem is universal for all types of anthropomorphic robot arms, and is capable of knowledge transfer at the level of individual motor acts. Next, the problems of learning and synthesis of motor synergies, the spatial and spatio-temporal combinations of motor features in sequential multi-action behavior, and the problems of task-related action transitions are considered in the second part of the thesis “Kinematic Motion Synthesis for Computer Graphics and Robotics”. In this part, a new approach of modeling complex full-body human actions by mixtures of time-shift invariant motor primitives in presented. The online-capable full-body motion generation architecture based on dynamic movement primitives driving the time-shift invariant motor synergies was implemented as an online-reactive adaptive motion synthesis for computer graphics and robotics applications. The last chapter of the thesis entitled “Contraction Theory and Self-organized Scenarios in Computer Graphics and Robotics” is dedicated to optimal control strategies in multi-agent scenarios of large crowds of agents expressing highly nonlinear behaviors. This last part presents new mathematical tools for stability analysis and synthesis of multi-agent cooperative scenarios.In den letzten Jahrzehnten hat die Forschung in den Bereichen der Steuerung und Regelung komplexer Systeme erhebliche Fortschritte gemacht, insbesondere in den Bereichen Robotik und Computeranimation. Die Entwicklung solcher Systeme verwendet heutzutage neueste Methoden und Entwicklungen im Bereich des maschinellen Lernens und der künstlichen Intelligenz. Die flexible und echtzeitfähige Kombination von motorischen Verhaltensweisen ist eine wesentliche Herausforderung für die Generierung menschenähnlicher Animationen und in der humanoiden Robotik. In diesem Zusammenhang liefern biologisch motivierte Methoden zur Analyse und Resynthese menschlicher motorischer Programme neue Erkenntnisse und Modelle für die antizipatorische Bewegungssynthese. Diese Dissertation präsentiert die Ergebnisse der Arbeiten des Autors im Gebiet der kognitiven und Entwicklungsrobotik, kooperativer und humanoider Robotersysteme sowie intelligenter und maschineller Lernmethoden in der Computergrafik. Der erste Teil der Dissertation im Kapitel “Zielgerichtete Nachahmung für Roboter” behandelt das Imitationslernen in der kognitiven und Entwicklungsrobotik. Die vorgestellten Arbeiten beschreiben neue Methoden für die hierarchische Bewegungserkennung und -planung, die durch Erkenntnisse zur Funktion der kortikalen Spiegelneuronen-Schaltkreise bei Primaten inspiriert wurden. Die entwickelte Architektur ist in der Lage, ‘durch Imitation zu lernen’ und ‘zu lernen zu imitieren’. Das komplette entwickelte System enthält ein echtzeitfähiges Pfadplanungssubsystem zur Hindernisvermeidung während der Durchführung von Armbewegungen. Das lernbasierte Pfadplanungssubsystem ist universell und für alle Arten von anthropomorphen Roboterarmen in der Lage, Wissen auf der Ebene einzelner motorischer Handlungen zu übertragen. Im zweiten Teil der Arbeit “Kinematische Bewegungssynthese für Computergrafik und Robotik” werden die Probleme des Lernens und der Synthese motorischer Synergien, d.h. von räumlichen und räumlich-zeitlichen Kombinationen motorischer Bewegungselemente bei Bewegungssequenzen und bei aufgabenbezogenen Handlungs übergängen behandelt. Es wird ein neuer Ansatz zur Modellierung komplexer menschlicher Ganzkörperaktionen durch Mischungen von zeitverschiebungsinvarianten Motorprimitiven vorgestellt. Zudem wurde ein online-fähiger Synthesealgorithmus für Ganzköperbewegungen entwickelt, der auf dynamischen Bewegungsprimitiven basiert, die wiederum auf der Basis der gelernten verschiebungsinvarianten Primitive konstruiert werden. Dieser Algorithmus wurde für verschiedene Probleme der Bewegungssynthese für die Computergrafik- und Roboteranwendungen implementiert. Das letzte Kapitel der Dissertation mit dem Titel “Kontraktionstheorie und selbstorganisierte Szenarien in der Computergrafik und Robotik” widmet sich optimalen Kontrollstrategien in Multi-Agenten-Szenarien, wobei die Agenten durch eine hochgradig nichtlineare Kinematik gekennzeichnet sind. Dieser letzte Teil präsentiert neue mathematische Werkzeuge für die Stabilitätsanalyse und Synthese von kooperativen Multi-Agenten-Szenarien

Learning Algorithm Design for Human-Robot Skill Transfer

In this research, we develop an intelligent learning scheme for performing human-robot skills transfer. Techniques adopted in the scheme include the Dynamic Movement Prim- itive (DMP) method with Dynamic Time Warping (DTW), Gaussian Mixture Model (G- MM) with Gaussian Mixture Regression (GMR) and the Radical Basis Function Neural Networks (RBFNNs). A series of experiments are conducted on a Baxter robot, a NAO robot and a KUKA iiwa robot to verify the effectiveness of the proposed design.During the design of the intelligent learning scheme, an online tracking system is de- veloped to control the arm and head movement of the NAO robot using a Kinect sensor. The NAO robot is a humanoid robot with 5 degrees of freedom (DOF) for each arm. The joint motions of the operator’s head and arm are captured by a Kinect V2 sensor, and this information is then transferred into the workspace via the forward and inverse kinematics. In addition, to improve the tracking performance, a Kalman filter is further employed to fuse motion signals from the operator sensed by the Kinect V2 sensor and a pair of MYO armbands, so as to teleoperate the Baxter robot. In this regard, a new strategy is developed using the vector approach to accomplish a specific motion capture task. For instance, the arm motion of the operator is captured by a Kinect sensor and programmed through a processing software. Two MYO armbands with embedded inertial measurement units are worn by the operator to aid the robots in detecting and replicating the operator’s arm movements. For this purpose, the armbands help to recognize and calculate the precise velocity of motion of the operator’s arm. Additionally, a neural network based adaptive controller is designed and implemented on the Baxter robot to illustrate the validation forthe teleoperation of the Baxter robot.Subsequently, an enhanced teaching interface has been developed for the robot using DMP and GMR. Motion signals are collected from a human demonstrator via the Kinect v2 sensor, and the data is sent to a remote PC for teleoperating the Baxter robot. At this stage, the DMP is utilized to model and generalize the movements. In order to learn from multiple demonstrations, DTW is used for the preprocessing of the data recorded on the robot platform, and GMM is employed for the evaluation of DMP to generate multiple patterns after the completion of the teaching process. Next, we apply the GMR algorithm to generate a synthesized trajectory to minimize position errors in the three dimensional (3D) space. This approach has been tested by performing tasks on a KUKA iiwa and a Baxter robot, respectively.Finally, an optimized DMP is added to the teaching interface. A character recombination technology based on DMP segmentation that uses verbal command has also been developed and incorporated in a Baxter robot platform. To imitate the recorded motion signals produced by the demonstrator, the operator trains the Baxter robot by physically guiding it to complete the given task. This is repeated five times, and the generated training data set is utilized via the playback system. Subsequently, the DTW is employed to preprocess the experimental data. For modelling and overall movement control, DMP is chosen. The GMM is used to generate multiple patterns after implementing the teaching process. Next, we employ the GMR algorithm to reduce position errors in the 3D space after a synthesized trajectory has been generated. The Baxter robot, remotely controlled by the user datagram protocol (UDP) in a PC, records and reproduces every trajectory. Additionally, Dragon Natural Speaking software is adopted to transcribe the voice data. This proposed approach has been verified by enabling the Baxter robot to perform a writing task of drawing robot has been taught to write only one character