Extended Object Tracking: Introduction, Overview and Applications

This article provides an elaborate overview of current research in extended object tracking. We provide a clear definition of the extended object tracking problem and discuss its delimitation to other types of object tracking. Next, different aspects of extended object modelling are extensively discussed. Subsequently, we give a tutorial introduction to two basic and well used extended object tracking approaches - the random matrix approach and the Kalman filter-based approach for star-convex shapes. The next part treats the tracking of multiple extended objects and elaborates how the large number of feasible association hypotheses can be tackled using both Random Finite Set (RFS) and Non-RFS multi-object trackers. The article concludes with a summary of current applications, where four example applications involving camera, X-band radar, light detection and ranging (lidar), red-green-blue-depth (RGB-D) sensors are highlighted.Comment: 30 pages, 19 figure

A Gaussian inverse Wishart PHD Filter using Stochastic Partitioning for Multiple Extended Object Tracking

This thesis deals with the object tracking problem of multiple extended objects. For instance, this tracking problem occurs when a car with sensors drives on the road and detects multiple other cars in front of it. When the setup between the senor and the other cars is in a such way that multiple measurements are created by each single car, the cars are called extended objects. This can occur in real world scenarios, mainly with the use of high resolution sensors in near field applications. Such a near field scenario leads a single object to occupy several resolution cells of the sensor so that multiple measurements are generated per scan. The measurements are additionally superimposed by the sensor’s noise. Beside the object generated measurements, there occur false alarms, which are not caused by any object and sometimes in a sensor scan, single objects could be missed so that they not generate any measurements. To handle these scenarios, object tracking filters are needed to process the sensor measurements in order to obtain a stable and accurate estimate of the objects in each sensor scan. In this thesis, the scope is to implement such a tracking filter that handles the extended objects, i.e. the filter estimates their positions and extents. In context of this, the topic of measurement partitioning occurs, which is a pre-processing of the measurement data. With the use of partitioning, the measurements that are likely generated by one object are put into one cluster, also called cell. Then, the obtained cells are processed by the tracking filter for the estimation process. The partitioning of measurement data is a crucial part for the performance of tracking filter because insufficient partitioning leads to bad tracking performance, i.e. inaccurate object estimates. In this thesis, a Gaussian inverse Wishart Probability Hypothesis Density (GIW-PHD) filter was implemented to handle the multiple extended object tracking problem. Within this filter framework, the number of objects are modelled as Random Finite Sets (RFSs) and the objects’ extent as random matrices (RM). The partitioning methods that are used to cluster the measurement data are existing ones as well as a new approach that is based on likelihood sampling methods. The applied classical heuristic methods are Distance Partitioning (DP) and Sub-Partitioning (SP), whereas the proposed likelihood-based approach is called Stochastic Partitioning (StP). The latter was developed in this thesis based on the Stochastic Optimisation approach by Granström et al. An implementation, including the StP method and its integration into the filter framework, is provided within this thesis. The implementations, using the different partitioning methods, were tested on simulated random multi-object scenarios and in a fixed parallel tracking scenario using Monte Carlo methods. Further, a runtime analysis was done to provide an insight into the computational effort using the different partitioning methods. It emphasized, that the StP method outperforms the classical partitioning methods in scenarios, where the objects move spatially close. The filter using StP performs more stable and with more accurate estimates. However, this advantage is associated with a higher computational effort compared to the classical heuristic partitioning methods

Multi Sensor Multi Target Perception and Tracking for Informed Decisions in Public Road Scenarios

Multi-target tracking in public traffic calls for a tracking system with automated track initiation and termination facilities in a randomly evolving driving environment. Besides, the key problem of data association needs to be handled effectively considering the limitations in the computational resources on-board an autonomous car. The challenge of the tracking problem is further evident in the use of high-resolution automotive sensors which return multiple detections per object. Furthermore, it is customary to use multiple sensors that cover different and/or over-lapping Field of View and fuse sensor detections to provide robust and reliable tracking. As a consequence, in high-resolution multi-sensor settings, the data association uncertainty, and the corresponding tracking complexity increases pointing to a systematic approach to handle and process sensor detections. In this work, we present a multi-target tracking system that addresses target birth/initiation and death/termination processes with automatic track management features. These tracking functionalities can help facilitate perception during common events in public traffic as participants (suddenly) change lanes, navigate intersections, overtake and/or brake in emergencies, etc. Various tracking approaches including the ones based on joint integrated probability data association (JIPDA) filter, Linear Multi-target Integrated Probabilistic Data Association (LMIPDA) Filter, and their multi-detection variants are adapted to specifically include algorithms that handle track initiation and termination, clutter density estimation and track management. The utility of the filtering module is further elaborated by integrating it into a trajectory tracking problem based on model predictive control. To cope with tracking complexity in the case of multiple high-resolution sensors, we propose a hybrid scheme that combines the approaches of data clustering at the local sensor and multiple detections tracking schemes at the fusion layer. We implement a track-to-track fusion scheme that de-correlates local (sensor) tracks to avoid double counting and apply a measurement partitioning scheme to re-purpose the LMIPDA tracking algorithm to multi-detection cases. In addition to the measurement partitioning approach, a joint extent and kinematic state estimation scheme are integrated into the LMIPDA approach to facilitate perception and tracking of an individual as well as group targets as applied to multi-lane public traffic. We formulate the tracking problem as a two hierarchical layer. This arrangement enhances the multi-target tracking performance in situations including but not limited to target initialization(birth process), target occlusion, missed detections, unresolved measurement, target maneuver, etc. Also, target groups expose complex individual target interactions to help in situation assessment which is challenging to capture otherwise. The simulation studies are complemented by experimental studies performed on single and multiple (group) targets. Target detections are collected from a high-resolution radar at a frequency of 20Hz; whereas RTK-GPS data is made available as ground truth for one of the target vehicle\u27s trajectory

Estimation and control of multi-object systems with high-fidenlity sensor models: A labelled random finite set approach

Principled and novel multi-object tracking algorithms are proposed, that have the ability to optimally process realistic sensor data, by accommodating complex observational phenomena such as merged measurements and extended targets. Additionally, a sensor control scheme based on a tractable, information theoretic objective is proposed, the goal of which is to optimise tracking performance in multi-object scenarios. The concept of labelled random finite sets is adopted in the development of these new techniques

Approximate Gaussian conjugacy: parametric recursive filtering under nonlinearity, multimodality, uncertainty, and constraint, and beyond

Since the landmark work of R. E. Kalman in the 1960s, considerable efforts have been devoted to time series state space models for a large variety of dynamic estimation problems. In particular, parametric filters that seek analytical estimates based on a closed-form Markov–Bayes recursion, e.g., recursion from a Gaussian or Gaussian mixture (GM) prior to a Gaussian/GM posterior (termed ‘Gaussian conjugacy’ in this paper), form the backbone for a general time series filter design. Due to challenges arising from nonlinearity, multimodality (including target maneuver), intractable uncertainties (such as unknown inputs and/or non-Gaussian noises) and constraints (including circular quantities), etc., new theories, algorithms, and technologies have been developed continuously to maintain such a conjugacy, or to approximate it as close as possible. They had contributed in large part to the prospective developments of time series parametric filters in the last six decades. In this paper, we review the state of the art in distinctive categories and highlight some insights that may otherwise be easily overlooked. In particular, specific attention is paid to nonlinear systems with an informative observation, multimodal systems including Gaussian mixture posterior and maneuvers, and intractable unknown inputs and constraints, to fill some gaps in existing reviews and surveys. In addition, we provide some new thoughts on alternatives to the first-order Markov transition model and on filter evaluation with regard to computing complexity

Advanced signal processing techniques for multi-target tracking

The multi-target tracking problem essentially involves the recursive joint estimation of the state of unknown and time-varying number of targets present in a tracking scene, given a series of observations. This problem becomes more challenging because the sequence of observations is noisy and can become corrupted due to miss-detections and false alarms/clutter. Additionally, the detected observations are indistinguishable from clutter. Furthermore, whether the target(s) of interest are point or extended (in terms of spatial extent) poses even more technical challenges. An approach known as random finite sets provides an elegant and rigorous framework for the handling of the multi-target tracking problem. With a random finite sets formulation, both the multi-target states and multi-target observations are modelled as finite set valued random variables, that is, random variables which are random in both the number of elements and the values of the elements themselves. Furthermore, compared to other approaches, the random finite sets approach possesses a desirable characteristic of being free of explicit data association prior to tracking. In addition, a framework is available for dealing with random finite sets and is known as finite sets statistics. In this thesis, advanced signal processing techniques are employed to provide enhancements to and develop new random finite sets based multi-target tracking algorithms for the tracking of both point and extended targets with the aim to improve tracking performance in cluttered environments. To this end, firstly, a new and efficient Kalman-gain aided sequential Monte Carlo probability hypothesis density (KG-SMC-PHD) filter and a cardinalised particle probability hypothesis density (KG-SMC-CPHD) filter are proposed. These filters employ the Kalman- gain approach during weight update to correct predicted particle states by minimising the mean square error between the estimated measurement and the actual measurement received at a given time in order to arrive at a more accurate posterior. This technique identifies and selects those particles belonging to a particular target from a given PHD for state correction during weight computation. The proposed SMC-CPHD filter provides a better estimate of the number of targets. Besides the improved tracking accuracy, fewer particles are required in the proposed approach. Simulation results confirm the improved tracking performance when evaluated with different measures. Secondly, the KG-SMC-(C)PHD filters are particle filter (PF) based and as with PFs, they require a process known as resampling to avoid the problem of degeneracy. This thesis proposes a new resampling scheme to address a problem with the systematic resampling method which causes a high tendency of resampling very low weight particles especially when a large number of resampled particles are required; which in turn affect state estimation. Thirdly, the KG-SMC-(C)PHD filters proposed in this thesis perform filtering and not tracking , that is, they provide only point estimates of target states but do not provide connected estimates of target trajectories from one time step to the next. A new post processing step using game theory as a solution to this filtering - tracking problem is proposed. This approach was named the GTDA method. This method was employed in the KG-SMC-(C)PHD filter as a post processing technique and was evaluated using both simulated and real data obtained using the NI-USRP software defined radio platform in a passive bi-static radar system. Lastly, a new technique for the joint tracking and labelling of multiple extended targets is proposed. To achieve multiple extended target tracking using this technique, models for the target measurement rate, kinematic component and target extension are defined and jointly propagated in time under the generalised labelled multi-Bernoulli (GLMB) filter framework. The GLMB filter is a random finite sets-based filter. In particular, a Poisson mixture variational Bayesian (PMVB) model is developed to simultaneously estimate the measurement rate of multiple extended targets and extended target extension was modelled using B-splines. The proposed method was evaluated with various performance metrics in order to demonstrate its effectiveness in tracking multiple extended targets

Tracking of Spline Modeled Extended Objects Using Random Finite Sets

In the field of autonomously driving vehicles the environment perception containing dynamic objects like other road users is essential. Especially, detecting other vehicles in the road traffic using sensor data is of utmost importance. As the sensor data and the applied system model for the objects of interest are noise corrupted, a filter algorithm must be used to track moving objects. Using LIDAR sensors one object gives rise to more than one measurement per time step and is therefore called extended object. This allows to jointly estimate the objects, position, as well as its orientation, extension and shape. Estimating an arbitrary shaped object comes with a higher computational effort than estimating the shape of an object that can be approximated using a basic geometrical shape like an ellipse or a rectangle. In the case of a vehicle, assuming a rectangular shape is an accurate assumption. A recently developed approach models the contour of a vehicle as periodic B-spline function. This representation is an easy to use tool, as the contour can be specified by some basis points in Cartesian coordinates. Also rotating, scaling and moving the contour is easy to handle using a spline contour. This contour model can be used to develop a measurement model for extended objects, that can be integrated into a tracking filter. Another approach modeling the shape of a vehicle is the so-called bounding box that represents the shape as rectangle. In this thesis the basics of single, multi and extended object tracking, as well as the basics of B-spline functions are addressed. Afterwards, the spline measurement model is established in detail and integrated into an extended Kalman filter to track a single extended object. An implementation of the resulting algorithm is compared with the rectangular shape estimator. The implementation of the rectangular shape estimator is provided. The comparison is done using long-term considerations with Monte Carlo simulations and by analyzing the results of a single run. Therefore, both algorithms are applied to the same measurements. The measurements are generated using an artificial LIDAR sensor in a simulation environment. In a real-world tracking scenario detecting several extended objects and measurements that do not originate from a real object, named clutter measurements, is possible. Also, the sudden appearance and disappearance of an object is possible. A filter framework investigated in recent years that can handle tracking multiple objects in a cluttered environment is a random finite set based approach. The idea of random finite sets and its use in a tracking filter is recapped in this thesis. Afterwards, the spline measurement model is included in a multi extended object tracking framework. An implementation of the resulting filter is investigated in a long-term consideration using Monte Carlo simulations and by analyzing the results of a single run. The multi extended object filter is also applied to artificial LIDAR measurements generated in a simulation environment. The results of comparing the spline based and rectangular based extended object trackers show a more stable performance of the spline extended object tracker. Also, some problems that have to be addressed in future works are discussed. The investigation of the resulting multi extended object tracker shows a successful integration of the spline measurement model in a multi extended object tracker. Also, with these results some problems remain, that have to be solved in future works

Parametric face alignment : generative and discriminative approaches

Tese de doutoramento em Engenharia Electrotécnica e de Computadores, apresentada à Faculdade de Ciências e Tecnologia da Universidade de CoimbraThis thesis addresses the matching of deformable human face models into 2D images. Two di erent approaches are detailed: generative and discriminative methods. Generative or holistic methods model the appearance/texture of all image pixels describing the face by synthesizing the expected appearance (it builds synthetic versions of the target face). Discriminative or patch-based methods model the local correlations between pixel values. Such approach uses an ensemble of local feature detectors all connected by a shape regularization model. Typically, generative approaches can achieve higher tting accuracy, but discriminative methods perform a lot better in unseen images. The Active Appearance Models (AAMs) are probably the most widely used generative technique. AAMs match parametric models of shape and appearance into new images by solving a nonlinear optimization that minimizes the di erence between a synthetic template and the real appearance. The rst part of this thesis describes the 2.5D AAM, an extension of the original 2D AAM that deals with a full perspective projection model. The 2.5D AAM uses a 3D Point Distribution Model (PDM) and a 2D appearance model whose control points are de ned by a perspective projection of the PDM. Two model tting algorithms and their computational e cient approximations are proposed: the Simultaneous Forwards Additive (SFA) and the Normalization Forwards Additive (NFA). Robust solutions for the SFA and NFA are also proposed in order to take into account the self-occlusion and/or partial occlusion of the face. Extensive results, involving the tting convergence, tting performance in unseen data, robustness to occlusion, tracking performance and pose estimation are shown. The second main part of this thesis concerns to discriminative methods such as the Constrained Local Models (CLM) or the Active Shape Models (ASM), where an ensemble of local feature detectors are constrained to lie within the subspace spanned by a PDM. Fitting such a model to an image typically involves two steps: (1) a local search using a detector, obtaining response maps for each landmark and (2) a global optimization that nds the shape parameters that jointly maximize all the detection responses. This work proposes: Discriminative Bayesian Active Shape Models (DBASM) a new global optimization strategy, using a Bayesian approach, where the posterior distribution of the shape parameters are inferred in a maximum a posteriori (MAP) sense by means of a Linear Dynamical System (LDS). The DBASM approach models the covariance of the latent variables i.e. it uses 2nd order statistics of the shape (and pose) parameters. Later, Bayesian Active Shape Models (BASM) is presented. BASM is an extension of the previous DBASM formulation where the prior distribution is explicitly modeled by means of recursive Bayesian estimation. Extensive results are presented, evaluating DBASM and BASM global optimization strategies, local face parts detectors and tracking performance in several standard datasets. Qualitative results taken from the challenging Labeled Faces in the Wild (LFW) dataset are also shown. Finally, the last part of this thesis, addresses the identity and facial expression recognition. Face geometry is extracted from input images using the AAM and low dimensional manifolds were then derived using Laplacian EigenMaps (LE) resulting in two types of manifolds, one for representing identity and the other for person-speci c facial expression. The identity and facial expression recognition system uses a two stage approach: First, a Support Vector Machines (SVM) is used to establish identity across expression changes, then the second stage deals with person-speci c expression recognition with a network of Hidden Markov Models (HMMs). Results taken from people exhibiting the six basic expressions (happiness, sadness, anger, fear, surprise and disgust) plus the neutral emotion are shown.Esta tese aborda a correspond^encia de modelos humanos de faces deform aveis em imagens 2D. S~ao apresentadas duas abordagens diferentes: m etodos generativos e discriminativos. Os modelos generativos ou hol sticos modelam a apar^encia/textura de todos os pixeis que descrevem a face, sintetizando a apar^encia esperada (s~ao criadas vers~oes sint eticas da face alvo). Os modelos discriminativos ou baseados em partes modelam correla c~oes locais entre valores de pixeis. Esta abordagem utiliza um conjunto de detectores locais de caracter sticas, conectados por um modelo de regulariza c~ao geom etrico. Normalmente, as abordagens generativas permitem obter uma maior precis~ ao de ajuste do modelo, mas os m etodos discriminativos funcionam bastante melhor em imagens nunca antes vistas. Os Modelos Activos de Apar^encia (AAMs) s~ao provavelmente a t ecnica generativa mais utilizada. Os AAMs ajustam modelos param etricos de forma e apar^encia em imagens, resolvendo uma optimiza c~ao n~ao linear que minimiza a diferen ca entre o modelo sint etico e a apar^encia real. A primeira parte desta tese descreve os AAM 2.5D, uma extens~ao do AAM original 2D que permite a utiliza c~ao de um modelo de projec c~ao em perspectiva. Os AAM 2.5D utilizam um Modelo de Distribui c~ao de Pointos (PDM) e um modelo de apar^encia 2D cujos pontos de controlo s~ao de nidos por uma projec c~ao em perspectiva do PDM. Dois algoritmos de ajuste do modelo e as suas aproxima c~oes e cientes s~ao propostas: Simultaneous Forwards Additive (SFA) e o Normalization Forwards Additive (NFA). Solu c~oes robustas para o SFA e NFA, que contemplam a oclus~ao parcial da face, s~ao igualmente propostas. Resultados extensos, envolvendo a converg^encia de ajuste, o desempenho em imagens nunca vistas, robustez a oclus~ao, desempenho de seguimento e estimativa de pose s~ao apresentados. A segunda parte desta da tese diz respeito os m etodos discriminativos, tais como os Modelos Locais com Restri c~oes (CLM) ou os Modelos Activos de Forma (ASM), onde um conjunto de detectores de caracteristicas locais est~ao restritos a pertencer ao subespa co gerado por um PDM. O ajuste de um modelo deste tipo, envolve tipicamente duas et apas: (1) uma pesquisa local utilizando um detector, obtendo mapas de resposta para cada ponto de refer^encia e (2) uma estrat egia de optimiza c~ao global que encontra os par^ametros do PDM que permitem maximizar todas as respostas conjuntamente. Neste trabalho e proposto o Discriminative Bayesian Active Shape Models (DBASM), uma nova estrat egia de optimiza c~ao global que utiliza uma abordagem Bayesiana, onde a distribui c~ao a posteriori dos par^ametros de forma s~ao inferidos por meio de um sistema din^amico linear. A abordagem DBASM modela a covari^ancia das vari aveis latentes ou seja, e utilizado estat stica de segunda ordem na modela c~ao dos par^ametros. Posteriormente e apresentada a formula c~ao Bayesian Active Shape Models (BASM). O BASM e uma extens~ao do DBASM, onde a distribui c~ao a priori e explicitamente modelada por meio de estima c~ao Bayesiana recursiva. S~ao apresentados resultados extensos, avaliando as estrat egias de optimiza c~ao globais DBASM e BASM, detectores locais de componentes da face, e desempenho de seguimento em v arias bases de dados padr~ao. Resultados qualitativos extra dos da desa ante base de dados Labeled Faces in the Wild (LFW) s~ao tamb em apresentados. Finalmente, a ultima parte desta tese aborda o reconhecimento de id^entidade e express~oes faciais. A geometria da face e extra da de imagens utilizando o AAM e variedades de baixa dimensionalidade s~ao derivadas utilizando Laplacian EigenMaps (LE), obtendo-se dois tipos de variedades, uma para representar a id^entidade e a outra para express~oes faciais espec cas de cada pessoa. A id^entidade e o sistema de reconhecimento de express~oes faciais utiliza uma abordagem de duas fases: Num primeiro est agio e utilizado uma M aquina de Vectores de Suporte (SVM) para determinar a id^entidade, dedicando-se o segundo est agio ao reconhecimento de express~oes. Este est agio e especi co para cada pessoa e utiliza Modelos de Markov Escondidos (HMM). S~ao mostrados resultados obtidos em pessoas exibindo as seis express~oes b asicas (alegria, tristeza, raiva, medo, surpresa e nojo), e ainda a emo c~ao neutra