Extended Object Tracking: Introduction, Overview and Applications

This article provides an elaborate overview of current research in extended object tracking. We provide a clear definition of the extended object tracking problem and discuss its delimitation to other types of object tracking. Next, different aspects of extended object modelling are extensively discussed. Subsequently, we give a tutorial introduction to two basic and well used extended object tracking approaches - the random matrix approach and the Kalman filter-based approach for star-convex shapes. The next part treats the tracking of multiple extended objects and elaborates how the large number of feasible association hypotheses can be tackled using both Random Finite Set (RFS) and Non-RFS multi-object trackers. The article concludes with a summary of current applications, where four example applications involving camera, X-band radar, light detection and ranging (lidar), red-green-blue-depth (RGB-D) sensors are highlighted.Comment: 30 pages, 19 figure

Multi Sensor Multi Target Perception and Tracking for Informed Decisions in Public Road Scenarios

Multi-target tracking in public traffic calls for a tracking system with automated track initiation and termination facilities in a randomly evolving driving environment. Besides, the key problem of data association needs to be handled effectively considering the limitations in the computational resources on-board an autonomous car. The challenge of the tracking problem is further evident in the use of high-resolution automotive sensors which return multiple detections per object. Furthermore, it is customary to use multiple sensors that cover different and/or over-lapping Field of View and fuse sensor detections to provide robust and reliable tracking. As a consequence, in high-resolution multi-sensor settings, the data association uncertainty, and the corresponding tracking complexity increases pointing to a systematic approach to handle and process sensor detections. In this work, we present a multi-target tracking system that addresses target birth/initiation and death/termination processes with automatic track management features. These tracking functionalities can help facilitate perception during common events in public traffic as participants (suddenly) change lanes, navigate intersections, overtake and/or brake in emergencies, etc. Various tracking approaches including the ones based on joint integrated probability data association (JIPDA) filter, Linear Multi-target Integrated Probabilistic Data Association (LMIPDA) Filter, and their multi-detection variants are adapted to specifically include algorithms that handle track initiation and termination, clutter density estimation and track management. The utility of the filtering module is further elaborated by integrating it into a trajectory tracking problem based on model predictive control. To cope with tracking complexity in the case of multiple high-resolution sensors, we propose a hybrid scheme that combines the approaches of data clustering at the local sensor and multiple detections tracking schemes at the fusion layer. We implement a track-to-track fusion scheme that de-correlates local (sensor) tracks to avoid double counting and apply a measurement partitioning scheme to re-purpose the LMIPDA tracking algorithm to multi-detection cases. In addition to the measurement partitioning approach, a joint extent and kinematic state estimation scheme are integrated into the LMIPDA approach to facilitate perception and tracking of an individual as well as group targets as applied to multi-lane public traffic. We formulate the tracking problem as a two hierarchical layer. This arrangement enhances the multi-target tracking performance in situations including but not limited to target initialization(birth process), target occlusion, missed detections, unresolved measurement, target maneuver, etc. Also, target groups expose complex individual target interactions to help in situation assessment which is challenging to capture otherwise. The simulation studies are complemented by experimental studies performed on single and multiple (group) targets. Target detections are collected from a high-resolution radar at a frequency of 20Hz; whereas RTK-GPS data is made available as ground truth for one of the target vehicle\u27s trajectory

Radar-based Application of Pedestrian and Cyclist Micro-Doppler Signatures for Automotive Safety Systems

Die sensorbasierte Erfassung des Nahfeldes im Kontext des hochautomatisierten Fahrens erfährt einen spürbaren Trend bei der Integration von Radarsensorik. Fortschritte in der Mikroelektronik erlauben den Einsatz von hochauflösenden Radarsensoren, die durch effiziente Verfahren sowohl im Winkel als auch in der Entfernung und im Doppler die Messgenauigkeit kontinuierlich ansteigen lassen. Dadurch ergeben sich neuartige Möglichkeiten bei der Bestimmung der geometrischen und kinematischen Beschaffenheit ausgedehnter Ziele im Fahrzeugumfeld, die zur gezielten Entwicklung von automotiven Sicherheitssystemen herangezogen werden können. Im Rahmen dieser Arbeit werden ungeschützte Verkehrsteilnehmer wie Fußgänger und Radfahrer mittels eines hochauflösenden Automotive-Radars analysiert. Dabei steht die Erscheinung des Mikro-Doppler-Effekts, hervorgerufen durch das hohe Maß an kinematischen Freiheitsgraden der Objekte, im Vordergrund der Betrachtung. Die durch den Mikro-Doppler-Effekt entstehenden charakteristischen Radar-Signaturen erlauben eine detailliertere Perzeption der Objekte und können in direkten Zusammenhang zu ihren aktuellen Bewegungszuständen gesetzt werden. Es werden neuartige Methoden vorgestellt, die die geometrischen und kinematischen Ausdehnungen der Objekte berücksichtigen und echtzeitfähige Ansätze zur Klassifikation und Verhaltensindikation realisieren. Wird ein ausgedehntes Ziel (z.B. Radfahrer) von einem Radarsensor detektiert, können aus dessen Mikro-Doppler-Signatur wesentliche Eigenschaften bezüglich seines Bewegungszustandes innerhalb eines Messzyklus erfasst werden. Die Geschwindigkeitsverteilungen der sich drehenden Räder erlauben eine adaptive Eingrenzung der Tretbewegung, deren Verhalten essentielle Merkmale im Hinblick auf eine vorausschauende Unfallprädiktion aufweist. Ferner unterliegen ausgedehnte Radarziele einer Orientierungsabhängigkeit, die deren geometrischen und kinematischen Profile direkt beeinflusst. Dies kann sich sowohl negativ auf die Klassifikations-Performance als auch auf die Verwertbarkeit von Parametern auswirken, die eine Absichtsbekundung des Radarziels konstituieren. Am Beispiel des Radfahrers wird hierzu ein Verfahren vorgestellt, das die orientierungsabhängigen Parameter in Entfernung und Doppler normalisiert und die gemessenen Mehrdeutigkeiten kompensiert. Ferner wird in dieser Arbeit eine Methodik vorgestellt, die auf Grundlage des Mikro- Doppler-Profils eines Fußgängers dessen Beinbewegungen über die Zeit schätzt (Tracking) und wertvolle Objektinformationen hinsichtlich seines Bewegungsverhaltens offenbart. Dazu wird ein Bewegungsmodell entwickelt, das die nichtlineare Fortbewegung des Beins approximiert und dessen hohes Maß an biomechanischer Variabilität abbildet. Durch die Einbeziehung einer wahrscheinlichkeitsbasierten Datenassoziation werden die Radar-Detektionen ihren jeweils hervorrufenden Quellen (linkes und rechtes Bein) zugeordnet und eine Trennung der Gliedmaßen realisiert. Im Gegensatz zu bisherigen Tracking-Verfahren weist die vorgestellte Methodik eine Steigerung in der Genauigkeit der Objektinformationen auf und stellt damit einen entscheidenden Vorteil für zukünftige Fahrerassistenzsysteme dar, um deutlich schneller auf kritische Verkehrssituationen reagieren zu können.:1 Introduction 1 1.1 Automotive environmental perception 2 1.2 Contributions of this work 4 1.3 Thesis overview 6 2 Automotive radar 9 2.1 Physical fundamentals 9 2.1.1 Radar cross section 9 2.1.2 Radar equation 10 2.1.3 Micro-Doppler effect 11 2.2 Radar measurement model 15 2.2.1 FMCW radar 15 2.2.2 Chirp sequence modulation 17 2.2.3 Direction-of-arrival estimation 22 2.3 Signal processing 25 2.3.1 Target properties 26 2.3.2 Target extraction 28 Power detection 28 Clustering 30 2.3.3 Real radar data example 31 2.4 Conclusion 33 3 Micro-Doppler applications of a cyclist 35 3.1 Physical fundamentals 35 3.1.1 Micro-Doppler signatures of a cyclist 35 3.1.2 Orientation dependence 36 3.2 Cyclist feature extraction 38 3.2.1 Adaptive pedaling extraction 38 Ellipticity constraints 38 Ellipse fitting algorithm 39 3.2.2 Experimental results 42 3.3 Normalization of the orientation dependence 44 3.3.1 Geometric correction 44 3.3.2 Kinematic correction 45 3.3.3 Experimental results 45 3.4 Conclusion 47 3.5 Discussion and outlook 47 4 Micro-Doppler applications of a pedestrian 49 4.1 Pedestrian detection 49 4.1.1 Human kinematics 49 4.1.2 Micro-Doppler signatures of a pedestrian 51 4.1.3 Experimental results 52 Radially moving pedestrian 52 Crossing pedestrian 54 4.2 Pedestrian feature extraction 57 4.2.1 Frequency-based limb separation 58 4.2.2 Extraction of body parts 60 4.2.3 Experimental results 62 4.3 Pedestrian tracking 64 4.3.1 Probabilistic state estimation 65 4.3.2 Gaussian filters 67 4.3.3 The Kalman filter 67 4.3.4 The extended Kalman filter 69 4.3.5 Multiple-object tracking 71 4.3.6 Data association 74 4.3.7 Joint probabilistic data association 80 4.4 Kinematic-based pedestrian tracking 84 4.4.1 Kinematic modeling 84 4.4.2 Tracking motion model 87 4.4.3 4-D radar point cloud 91 4.4.4 Tracking implementation 92 4.4.5 Experimental results 96 Longitudinal trajectory 96 Crossing trajectory with sudden turn 98 4.5 Conclusion 102 4.6 Discussion and outlook 103 5 Summary and outlook 105 5.1 Developed algorithms 105 5.1.1 Adaptive pedaling extraction 105 5.1.2 Normalization of the orientation dependence 105 5.1.3 Model-based pedestrian tracking 106 5.2 Outlook 106 Bibliography 109 List of Acronyms 119 List of Figures 124 List of Tables 125 Appendix 127 A Derivation of the rotation matrix 2.26 127 B Derivation of the mixed radar signal 2.52 129 C Calculation of the marginal association probabilities 4.51 131 Curriculum Vitae 135Sensor-based detection of the near field in the context of highly automated driving is experiencing a noticeable trend in the integration of radar sensor technology. Advances in microelectronics allow the use of high-resolution radar sensors that continuously increase measurement accuracy through efficient processes in angle as well as distance and Doppler. This opens up novel possibilities in determining the geometric and kinematic nature of extended targets in the vehicle environment, which can be used for the specific development of automotive safety systems. In this work, vulnerable road users such as pedestrians and cyclists are analyzed using a high-resolution automotive radar. The focus is on the appearance of the micro-Doppler effect, caused by the objects’ high kinematic degree of freedom. The characteristic radar signatures produced by the micro-Doppler effect allow a clearer perception of the objects and can be directly related to their current state of motion. Novel methods are presented that consider the geometric and kinematic extents of the objects and realize real-time approaches to classification and behavioral indication. When a radar sensor detects an extended target (e.g., bicyclist), its motion state’s fundamental properties can be captured from its micro-Doppler signature within a measurement cycle. The spinning wheels’ velocity distributions allow an adaptive containment of the pedaling motion, whose behavior exhibits essential characteristics concerning predictive accident prediction. Furthermore, extended radar targets are subject to orientation dependence, directly affecting their geometric and kinematic profiles. This can negatively affect both the classification performance and the usability of parameters constituting the radar target’s intention statement. For this purpose, using the cyclist as an example, a method is presented that normalizes the orientation-dependent parameters in range and Doppler and compensates for the measured ambiguities. Furthermore, this paper presents a methodology that estimates a pedestrian’s leg motion over time (tracking) based on the pedestrian’s micro-Doppler profile and reveals valuable object information regarding his motion behavior. To this end, a motion model is developed that approximates the leg’s nonlinear locomotion and represents its high degree of biomechanical variability. By incorporating likelihood-based data association, radar detections are assigned to their respective evoking sources (left and right leg), and limb separation is realized. In contrast to previous tracking methods, the presented methodology shows an increase in the object information’s accuracy. It thus represents a decisive advantage for future driver assistance systems in order to be able to react significantly faster to critical traffic situations.:1 Introduction 1 1.1 Automotive environmental perception 2 1.2 Contributions of this work 4 1.3 Thesis overview 6 2 Automotive radar 9 2.1 Physical fundamentals 9 2.1.1 Radar cross section 9 2.1.2 Radar equation 10 2.1.3 Micro-Doppler effect 11 2.2 Radar measurement model 15 2.2.1 FMCW radar 15 2.2.2 Chirp sequence modulation 17 2.2.3 Direction-of-arrival estimation 22 2.3 Signal processing 25 2.3.1 Target properties 26 2.3.2 Target extraction 28 Power detection 28 Clustering 30 2.3.3 Real radar data example 31 2.4 Conclusion 33 3 Micro-Doppler applications of a cyclist 35 3.1 Physical fundamentals 35 3.1.1 Micro-Doppler signatures of a cyclist 35 3.1.2 Orientation dependence 36 3.2 Cyclist feature extraction 38 3.2.1 Adaptive pedaling extraction 38 Ellipticity constraints 38 Ellipse fitting algorithm 39 3.2.2 Experimental results 42 3.3 Normalization of the orientation dependence 44 3.3.1 Geometric correction 44 3.3.2 Kinematic correction 45 3.3.3 Experimental results 45 3.4 Conclusion 47 3.5 Discussion and outlook 47 4 Micro-Doppler applications of a pedestrian 49 4.1 Pedestrian detection 49 4.1.1 Human kinematics 49 4.1.2 Micro-Doppler signatures of a pedestrian 51 4.1.3 Experimental results 52 Radially moving pedestrian 52 Crossing pedestrian 54 4.2 Pedestrian feature extraction 57 4.2.1 Frequency-based limb separation 58 4.2.2 Extraction of body parts 60 4.2.3 Experimental results 62 4.3 Pedestrian tracking 64 4.3.1 Probabilistic state estimation 65 4.3.2 Gaussian filters 67 4.3.3 The Kalman filter 67 4.3.4 The extended Kalman filter 69 4.3.5 Multiple-object tracking 71 4.3.6 Data association 74 4.3.7 Joint probabilistic data association 80 4.4 Kinematic-based pedestrian tracking 84 4.4.1 Kinematic modeling 84 4.4.2 Tracking motion model 87 4.4.3 4-D radar point cloud 91 4.4.4 Tracking implementation 92 4.4.5 Experimental results 96 Longitudinal trajectory 96 Crossing trajectory with sudden turn 98 4.5 Conclusion 102 4.6 Discussion and outlook 103 5 Summary and outlook 105 5.1 Developed algorithms 105 5.1.1 Adaptive pedaling extraction 105 5.1.2 Normalization of the orientation dependence 105 5.1.3 Model-based pedestrian tracking 106 5.2 Outlook 106 Bibliography 109 List of Acronyms 119 List of Figures 124 List of Tables 125 Appendix 127 A Derivation of the rotation matrix 2.26 127 B Derivation of the mixed radar signal 2.52 129 C Calculation of the marginal association probabilities 4.51 131 Curriculum Vitae 13

Tracking and Fusion Methods for Extended Targets Parameterized by Center, Orientation, and Semi-axes

The improvements in sensor technology, e.g., the development of automotive Radio Detection and Ranging (RADAR) or Light Detection and Ranging (LIDAR), which are able to provide a higher detail of the sensor’s environment, have introduced new opportunities but also new challenges to target tracking. In classic target tracking, targets are assumed as points. However, this assumption is no longer valid if targets occupy more than one sensor resolution cell, creating the need for extended targets, modeling the shape in addition to the kinematic parameters. Different shape models are possible and this thesis focuses on an elliptical shape, parameterized with center, orientation, and semi-axes lengths. This parameterization can be used to model rectangles as well. Furthermore, this thesis is concerned with multi-sensor fusion for extended targets, which can be used to improve the target tracking by providing information gathered from different sensors or perspectives. We also consider estimation of extended targets, i.e., to account for uncertainties, the target is modeled by a probability density, so we need to find a so-called point estimate. Extended target tracking provides a variety of challenges due to the spatial extent, which need to be handled, even for basic shapes like ellipses and rectangles. Among these challenges are the choice of the target model, e.g., how the measurements are distributed across the shape. Additional challenges arise for sensor fusion, as it is unclear how to best consider the geometric properties when combining two extended targets. Finally, the extent needs to be involved in the estimation. Traditional methods often use simple uniform distributions across the shape, which do not properly portray reality, while more complex methods require the use of optimization techniques or large amounts of data. In addition, for traditional estimation, metrics such as the Euclidean distance between state vectors are used. However, they might no longer be valid because they do not consider the geometric properties of the targets’ shapes, e.g., rotating an ellipse by 180 degree results in the same ellipse, but the Euclidean distance between them is not 0. In multi-sensor fusion, the same holds, i.e., simply combining the corresponding elements of the state vectors can lead to counter-intuitive fusion results. In this work, we compare different elliptic trackers and discuss more complex measurement distributions across the shape’s surface or contour. Furthermore, we discuss the problems which can occur when fusing extended target estimates from different sensors and how to handle them by providing a transformation into a special density. We then proceed to discuss how a different metric, namely the Gaussian Wasserstein (GW) distance, can be used to improve target estimation. We define an estimator and propose an approximation based on an extension of the square root distance. It can be applied on the posterior densities of the aforementioned trackers to incorporate the unique properties of ellipses in the estimation process. We also discuss how this can be applied to rectangular targets as well. Finally, we evaluate and discuss our approaches. We show the benefits of more complex target models in simulations and on real data and we demonstrate our estimation and fusion approaches compared to classic methods on simulated data.2022-01-2

Conjugate priors for Bayesian object tracking

Object tracking refers to the problem of using noisy sensor measurements to determine the location and characteristics of objects of interest in clutter. Nowadays, object tracking has found applications in numerous research venues as well as application areas, including air traffic control, maritime navigation, remote sensing, intelligent video surveillance, and more recently environmental perception, which is a key enabling technology in autonomous vehicles. This thesis studies conjugate priors for Bayesian object tracking with focus on multi-object tracking (MOT) based on sets of trajectories. Finite Set Statistics provides an elegant Bayesian formulation of MOT in terms of the theory of random finite sets (RFSs). Conjugate priors are also of great interest as they provide families of distributions that are suitable to work with when seeking accurate approximations to the true posterior distributions. Many RFS-based MOT approaches are only concerned with multi-object filtering without attempting to estimate object trajectories. An appealing approach to building tracks is by computing the multi-object densities on sets of trajectories. This leads to the development of trajectory filters, e.g., filters based on Poisson multi-Bernoulli mixture (PMBM) conjugate priors.In this thesis, [Paper A] and [Paper B] consider the problem of point object tracking where an object generates at most one measurement per scan. In [Paper A], it is shown that the trajectory MBM filter is the solution to the MOT problem for standard point object models with multi-Bernoulli birth. In addition, the multi-scan implementations of trajectory PMBM and MBM filters are presented. In [Paper B], a solution for recovering full trajectory information, via the calculation of the posterior of the set of trajectories from a sequence of multi-object filtering densities and the multi-object dynamic model, is presented. [Paper C] and [Paper D] consider the problem of ex- tended object tracking where an object may generate multiple measurements per scan. In [Paper C], the extended object PMBM filter for sets of objects is generalized to sets of trajectories. In [Paper D], a learning-based extended ob- ject tracking algorithm using a hierarchical truncated Gaussian measurement model tailored for automotive radar measurements is presented

Which Framework is Suitable for Online 3D Multi-Object Tracking for Autonomous Driving with Automotive 4D Imaging Radar?

Online 3D multi-object tracking (MOT) has recently received significant research interests due to the expanding demand of 3D perception in advanced driver assistance systems (ADAS) and autonomous driving (AD). Among the existing 3D MOT frameworks for ADAS and AD, conventional point object tracking (POT) framework using the tracking-by-detection (TBD) strategy has been well studied and accepted for LiDAR and 4D imaging radar point clouds. In contrast, extended object tracking (EOT), another important framework which accepts the joint-detection-and-tracking (JDT) strategy, has rarely been explored for online 3D MOT applications. This paper provides the first systematical investigation of the EOT framework for online 3D MOT in real-world ADAS and AD scenarios. Specifically, the widely accepted TBD-POT framework, the recently investigated JDT-EOT framework, and our proposed TBD-EOT framework are compared via extensive evaluations on two open source 4D imaging radar datasets: View-of-Delft and TJ4DRadSet. Experiment results demonstrate that the conventional TBD-POT framework remains preferable for online 3D MOT with high tracking performance and low computational complexity, while the proposed TBD-EOT framework has the potential to outperform it in certain situations. However, the results also show that the JDT-EOT framework encounters multiple problems and performs inadequately in evaluation scenarios. After analyzing the causes of these phenomena based on various evaluation metrics and visualizations, we provide possible guidelines to improve the performance of these MOT frameworks on real-world data. These provide the first benchmark and important insights for the future development of 4D imaging radar-based online 3D MOT.Comment: 8 pages, 5 figures, submitted to the 2024 IEEE International Conference on Robotics and Automation (ICRA2024