Extended Object Tracking: Introduction, Overview and Applications

This article provides an elaborate overview of current research in extended object tracking. We provide a clear definition of the extended object tracking problem and discuss its delimitation to other types of object tracking. Next, different aspects of extended object modelling are extensively discussed. Subsequently, we give a tutorial introduction to two basic and well used extended object tracking approaches - the random matrix approach and the Kalman filter-based approach for star-convex shapes. The next part treats the tracking of multiple extended objects and elaborates how the large number of feasible association hypotheses can be tackled using both Random Finite Set (RFS) and Non-RFS multi-object trackers. The article concludes with a summary of current applications, where four example applications involving camera, X-band radar, light detection and ranging (lidar), red-green-blue-depth (RGB-D) sensors are highlighted.Comment: 30 pages, 19 figure

Radar-based Application of Pedestrian and Cyclist Micro-Doppler Signatures for Automotive Safety Systems

Die sensorbasierte Erfassung des Nahfeldes im Kontext des hochautomatisierten Fahrens erfährt einen spürbaren Trend bei der Integration von Radarsensorik. Fortschritte in der Mikroelektronik erlauben den Einsatz von hochauflösenden Radarsensoren, die durch effiziente Verfahren sowohl im Winkel als auch in der Entfernung und im Doppler die Messgenauigkeit kontinuierlich ansteigen lassen. Dadurch ergeben sich neuartige Möglichkeiten bei der Bestimmung der geometrischen und kinematischen Beschaffenheit ausgedehnter Ziele im Fahrzeugumfeld, die zur gezielten Entwicklung von automotiven Sicherheitssystemen herangezogen werden können. Im Rahmen dieser Arbeit werden ungeschützte Verkehrsteilnehmer wie Fußgänger und Radfahrer mittels eines hochauflösenden Automotive-Radars analysiert. Dabei steht die Erscheinung des Mikro-Doppler-Effekts, hervorgerufen durch das hohe Maß an kinematischen Freiheitsgraden der Objekte, im Vordergrund der Betrachtung. Die durch den Mikro-Doppler-Effekt entstehenden charakteristischen Radar-Signaturen erlauben eine detailliertere Perzeption der Objekte und können in direkten Zusammenhang zu ihren aktuellen Bewegungszuständen gesetzt werden. Es werden neuartige Methoden vorgestellt, die die geometrischen und kinematischen Ausdehnungen der Objekte berücksichtigen und echtzeitfähige Ansätze zur Klassifikation und Verhaltensindikation realisieren. Wird ein ausgedehntes Ziel (z.B. Radfahrer) von einem Radarsensor detektiert, können aus dessen Mikro-Doppler-Signatur wesentliche Eigenschaften bezüglich seines Bewegungszustandes innerhalb eines Messzyklus erfasst werden. Die Geschwindigkeitsverteilungen der sich drehenden Räder erlauben eine adaptive Eingrenzung der Tretbewegung, deren Verhalten essentielle Merkmale im Hinblick auf eine vorausschauende Unfallprädiktion aufweist. Ferner unterliegen ausgedehnte Radarziele einer Orientierungsabhängigkeit, die deren geometrischen und kinematischen Profile direkt beeinflusst. Dies kann sich sowohl negativ auf die Klassifikations-Performance als auch auf die Verwertbarkeit von Parametern auswirken, die eine Absichtsbekundung des Radarziels konstituieren. Am Beispiel des Radfahrers wird hierzu ein Verfahren vorgestellt, das die orientierungsabhängigen Parameter in Entfernung und Doppler normalisiert und die gemessenen Mehrdeutigkeiten kompensiert. Ferner wird in dieser Arbeit eine Methodik vorgestellt, die auf Grundlage des Mikro- Doppler-Profils eines Fußgängers dessen Beinbewegungen über die Zeit schätzt (Tracking) und wertvolle Objektinformationen hinsichtlich seines Bewegungsverhaltens offenbart. Dazu wird ein Bewegungsmodell entwickelt, das die nichtlineare Fortbewegung des Beins approximiert und dessen hohes Maß an biomechanischer Variabilität abbildet. Durch die Einbeziehung einer wahrscheinlichkeitsbasierten Datenassoziation werden die Radar-Detektionen ihren jeweils hervorrufenden Quellen (linkes und rechtes Bein) zugeordnet und eine Trennung der Gliedmaßen realisiert. Im Gegensatz zu bisherigen Tracking-Verfahren weist die vorgestellte Methodik eine Steigerung in der Genauigkeit der Objektinformationen auf und stellt damit einen entscheidenden Vorteil für zukünftige Fahrerassistenzsysteme dar, um deutlich schneller auf kritische Verkehrssituationen reagieren zu können.:1 Introduction 1 1.1 Automotive environmental perception 2 1.2 Contributions of this work 4 1.3 Thesis overview 6 2 Automotive radar 9 2.1 Physical fundamentals 9 2.1.1 Radar cross section 9 2.1.2 Radar equation 10 2.1.3 Micro-Doppler effect 11 2.2 Radar measurement model 15 2.2.1 FMCW radar 15 2.2.2 Chirp sequence modulation 17 2.2.3 Direction-of-arrival estimation 22 2.3 Signal processing 25 2.3.1 Target properties 26 2.3.2 Target extraction 28 Power detection 28 Clustering 30 2.3.3 Real radar data example 31 2.4 Conclusion 33 3 Micro-Doppler applications of a cyclist 35 3.1 Physical fundamentals 35 3.1.1 Micro-Doppler signatures of a cyclist 35 3.1.2 Orientation dependence 36 3.2 Cyclist feature extraction 38 3.2.1 Adaptive pedaling extraction 38 Ellipticity constraints 38 Ellipse fitting algorithm 39 3.2.2 Experimental results 42 3.3 Normalization of the orientation dependence 44 3.3.1 Geometric correction 44 3.3.2 Kinematic correction 45 3.3.3 Experimental results 45 3.4 Conclusion 47 3.5 Discussion and outlook 47 4 Micro-Doppler applications of a pedestrian 49 4.1 Pedestrian detection 49 4.1.1 Human kinematics 49 4.1.2 Micro-Doppler signatures of a pedestrian 51 4.1.3 Experimental results 52 Radially moving pedestrian 52 Crossing pedestrian 54 4.2 Pedestrian feature extraction 57 4.2.1 Frequency-based limb separation 58 4.2.2 Extraction of body parts 60 4.2.3 Experimental results 62 4.3 Pedestrian tracking 64 4.3.1 Probabilistic state estimation 65 4.3.2 Gaussian filters 67 4.3.3 The Kalman filter 67 4.3.4 The extended Kalman filter 69 4.3.5 Multiple-object tracking 71 4.3.6 Data association 74 4.3.7 Joint probabilistic data association 80 4.4 Kinematic-based pedestrian tracking 84 4.4.1 Kinematic modeling 84 4.4.2 Tracking motion model 87 4.4.3 4-D radar point cloud 91 4.4.4 Tracking implementation 92 4.4.5 Experimental results 96 Longitudinal trajectory 96 Crossing trajectory with sudden turn 98 4.5 Conclusion 102 4.6 Discussion and outlook 103 5 Summary and outlook 105 5.1 Developed algorithms 105 5.1.1 Adaptive pedaling extraction 105 5.1.2 Normalization of the orientation dependence 105 5.1.3 Model-based pedestrian tracking 106 5.2 Outlook 106 Bibliography 109 List of Acronyms 119 List of Figures 124 List of Tables 125 Appendix 127 A Derivation of the rotation matrix 2.26 127 B Derivation of the mixed radar signal 2.52 129 C Calculation of the marginal association probabilities 4.51 131 Curriculum Vitae 135Sensor-based detection of the near field in the context of highly automated driving is experiencing a noticeable trend in the integration of radar sensor technology. Advances in microelectronics allow the use of high-resolution radar sensors that continuously increase measurement accuracy through efficient processes in angle as well as distance and Doppler. This opens up novel possibilities in determining the geometric and kinematic nature of extended targets in the vehicle environment, which can be used for the specific development of automotive safety systems. In this work, vulnerable road users such as pedestrians and cyclists are analyzed using a high-resolution automotive radar. The focus is on the appearance of the micro-Doppler effect, caused by the objects’ high kinematic degree of freedom. The characteristic radar signatures produced by the micro-Doppler effect allow a clearer perception of the objects and can be directly related to their current state of motion. Novel methods are presented that consider the geometric and kinematic extents of the objects and realize real-time approaches to classification and behavioral indication. When a radar sensor detects an extended target (e.g., bicyclist), its motion state’s fundamental properties can be captured from its micro-Doppler signature within a measurement cycle. The spinning wheels’ velocity distributions allow an adaptive containment of the pedaling motion, whose behavior exhibits essential characteristics concerning predictive accident prediction. Furthermore, extended radar targets are subject to orientation dependence, directly affecting their geometric and kinematic profiles. This can negatively affect both the classification performance and the usability of parameters constituting the radar target’s intention statement. For this purpose, using the cyclist as an example, a method is presented that normalizes the orientation-dependent parameters in range and Doppler and compensates for the measured ambiguities. Furthermore, this paper presents a methodology that estimates a pedestrian’s leg motion over time (tracking) based on the pedestrian’s micro-Doppler profile and reveals valuable object information regarding his motion behavior. To this end, a motion model is developed that approximates the leg’s nonlinear locomotion and represents its high degree of biomechanical variability. By incorporating likelihood-based data association, radar detections are assigned to their respective evoking sources (left and right leg), and limb separation is realized. In contrast to previous tracking methods, the presented methodology shows an increase in the object information’s accuracy. It thus represents a decisive advantage for future driver assistance systems in order to be able to react significantly faster to critical traffic situations.:1 Introduction 1 1.1 Automotive environmental perception 2 1.2 Contributions of this work 4 1.3 Thesis overview 6 2 Automotive radar 9 2.1 Physical fundamentals 9 2.1.1 Radar cross section 9 2.1.2 Radar equation 10 2.1.3 Micro-Doppler effect 11 2.2 Radar measurement model 15 2.2.1 FMCW radar 15 2.2.2 Chirp sequence modulation 17 2.2.3 Direction-of-arrival estimation 22 2.3 Signal processing 25 2.3.1 Target properties 26 2.3.2 Target extraction 28 Power detection 28 Clustering 30 2.3.3 Real radar data example 31 2.4 Conclusion 33 3 Micro-Doppler applications of a cyclist 35 3.1 Physical fundamentals 35 3.1.1 Micro-Doppler signatures of a cyclist 35 3.1.2 Orientation dependence 36 3.2 Cyclist feature extraction 38 3.2.1 Adaptive pedaling extraction 38 Ellipticity constraints 38 Ellipse fitting algorithm 39 3.2.2 Experimental results 42 3.3 Normalization of the orientation dependence 44 3.3.1 Geometric correction 44 3.3.2 Kinematic correction 45 3.3.3 Experimental results 45 3.4 Conclusion 47 3.5 Discussion and outlook 47 4 Micro-Doppler applications of a pedestrian 49 4.1 Pedestrian detection 49 4.1.1 Human kinematics 49 4.1.2 Micro-Doppler signatures of a pedestrian 51 4.1.3 Experimental results 52 Radially moving pedestrian 52 Crossing pedestrian 54 4.2 Pedestrian feature extraction 57 4.2.1 Frequency-based limb separation 58 4.2.2 Extraction of body parts 60 4.2.3 Experimental results 62 4.3 Pedestrian tracking 64 4.3.1 Probabilistic state estimation 65 4.3.2 Gaussian filters 67 4.3.3 The Kalman filter 67 4.3.4 The extended Kalman filter 69 4.3.5 Multiple-object tracking 71 4.3.6 Data association 74 4.3.7 Joint probabilistic data association 80 4.4 Kinematic-based pedestrian tracking 84 4.4.1 Kinematic modeling 84 4.4.2 Tracking motion model 87 4.4.3 4-D radar point cloud 91 4.4.4 Tracking implementation 92 4.4.5 Experimental results 96 Longitudinal trajectory 96 Crossing trajectory with sudden turn 98 4.5 Conclusion 102 4.6 Discussion and outlook 103 5 Summary and outlook 105 5.1 Developed algorithms 105 5.1.1 Adaptive pedaling extraction 105 5.1.2 Normalization of the orientation dependence 105 5.1.3 Model-based pedestrian tracking 106 5.2 Outlook 106 Bibliography 109 List of Acronyms 119 List of Figures 124 List of Tables 125 Appendix 127 A Derivation of the rotation matrix 2.26 127 B Derivation of the mixed radar signal 2.52 129 C Calculation of the marginal association probabilities 4.51 131 Curriculum Vitae 13

Uncertainty Quantification in Composite Materials

The random nature of the micro-structural attributes in materials in general and composite material systems in particular requires expansion of material modeling in a way that will incorporate their inherent uncertainty and predict its impact on material properties and mechanical response in multiple scales. Despite the importance of capturing and modeling material randomness, there are numerous challenges in structural characterization that are yet to be addressed. The work presented in this essay takes a few steps towards an improved material modeling approach which encompasses structural randomness in order to produce a more realistic representation of material systems. For this end a computational framework was developed to generate a realistic representative volume element which reflects the inherent structural randomness. First stochastic structural elements were identified and registered from imaging data and parameters were assigned to represent those elements. Statistical characterization of the random attributes was followed by the construction of a representative volume element which shared the same structural statistical characteristics with the original material system. The resultant statistical equivalent representative volume element (SERVE) was then used in finite element simulations which provided homogenized properties and mechanical response predictions. The suggested framework was developed and then implemented on 3 different material systems. Image processing and analysis in one of the material systems extended the original scope of this work to solving a machine vision and learning problem. Object segmentation for the purpose object and pattern recognition has been a long standing subject of interest in the field of machine vision. Despite the significant attention given to the development of segmentation and recognition methods, the critical challenge of separating merged objects did not share the spotlight. A simple yet original approach to overcome this hurdle was developed using unsupervised classification and separation of objects in 3D. Lower dimensionality classifiers were joined to provide a powerful higher dimensionality classification tool. The robustness of this approach is illustrated through its implementation on two case studies of merged objects. Applications of this methodology can further extend from structural classification to general problems of clustering and classification in various fields

UAV Optimal Cooperative Obstacle Avoidance and Target Tracking in Dynamic Stochastic Environments

Cette thèse propose une stratégie de contrôle avancée pour guider une flotte d'aéronefs sans pilote (UAV) dans un environnement à la fois stochastique et dynamique. Pour ce faire, un simulateur de vol 3D a été développé avec MATLAB® pour tester les algorithmes de la stratégie de guidage en fonctions de différents scénarios. L'objectif des missions simulées est de s'assurer que chaque UAV intercepte une cible ellipsoïdale mobile tout en évitant une panoplie d'obstacles ellipsoïdaux mobiles détectés en route. Les UAVs situés à l'intérieur des limites de communication peuvent coopérer afin d'améliorer leurs performances au cours de la mission. Le simulateur a été conçu de façon à ce que les UAV soient dotés de capteurs et d'appareils de communication de portée limitée. De plus, chaque UAV possède un pilote automatique qui stabilise l'aéronef en vol et un planificateur de trajectoires qui génère les commandes à envoyer au pilote automatique. Au coeur du planificateur de trajectoires se trouve un contrôleur prédictif à horizon fuyant qui détermine les commandes à envoyer à l'UAV. Ces commandes optimisent un critère de performance assujetti à des contraintes. Le critère de performance est conçu de sorte que les UAV atteignent les objectifs de la mission, alors que les contraintes assurent que les commandes générées adhèrent aux limites de manoeuvrabilité de l'aéronef. La planification de trajectoires pour UAV opérant dans un environnement dynamique et stochastique dépend fortement des déplacements anticipés des objets (obstacle, cible). Un filtre de Kalman étendu est donc utilisé pour prédire les trajectoires les plus probables des objets à partir de leurs états estimés. Des stratégies de poursuite et d'évitement ont aussi été développées en fonction des trajectoires prédites des objets détectés. Pour des raisons de sécurité, la conception de stratégies d'évitement de collision à la fois efficaces et robustes est primordiale au guidage d'UAV. Une nouvelle stratégie d'évitement d'obstacles par approche probabiliste a donc été développée. La méthode cherche à minimiser la probabilité de collision entre l'UAV et tous ses obstacles détectés sur l'horizon de prédiction, tout en s'assurant que, à chaque pas de prédiction, la probabilité de collision entre l'UAV et chacun de ses obstacles détectés ne surpasse pas un seuil prescrit. Des simulations sont présentées au cours de cette thèse pour démontrer l'efficacité des algorithmes proposés

Efficient Constellation-Based Map-Merging for Semantic SLAM

Data association in SLAM is fundamentally challenging, and handling ambiguity well is crucial to achieve robust operation in real-world environments. When ambiguous measurements arise, conservatism often mandates that the measurement is discarded or a new landmark is initialized rather than risking an incorrect association. To address the inevitable `duplicate' landmarks that arise, we present an efficient map-merging framework to detect duplicate constellations of landmarks, providing a high-confidence loop-closure mechanism well-suited for object-level SLAM. This approach uses an incrementally-computable approximation of landmark uncertainty that only depends on local information in the SLAM graph, avoiding expensive recovery of the full system covariance matrix. This enables a search based on geometric consistency (GC) (rather than full joint compatibility (JC)) that inexpensively reduces the search space to a handful of `best' hypotheses. Furthermore, we reformulate the commonly-used interpretation tree to allow for more efficient integration of clique-based pairwise compatibility, accelerating the branch-and-bound max-cardinality search. Our method is demonstrated to match the performance of full JC methods at significantly-reduced computational cost, facilitating robust object-based loop-closure over large SLAM problems.Comment: Accepted to IEEE International Conference on Robotics and Automation (ICRA) 201

Screening Data Points in Empirical Risk Minimization via Ellipsoidal Regions and Safe Loss Functions

We design simple screening tests to automatically discard data samples in empirical risk minimization without losing optimization guarantees. We derive loss functions that produce dual objectives with a sparse solution. We also show how to regularize convex losses to ensure such a dual sparsity-inducing property, and propose a general method to design screening tests for classification or regression based on ellipsoidal approximations of the optimal set. In addition to producing computational gains, our approach also allows us to compress a dataset into a subset of representative points

Constructing a reference standard for sports science and clinical movement sets using IMU-based motion capture technology

Motion analysis has improved greatly over the years through the development of low-cost inertia sensors. Such sensors have shown promising accuracy for both sport and medical applications, facilitating the possibility of a new reference standard to be constructed. Current gold standards within motion capture, such as high-speed camera-based systems and image processing, are not suitable for many movement-sets within both sports science and clinical movement analysis due to restrictions introduced by the movement sets. These restrictions include cost, portability, local environment constraints (such as light level) and poor line of sight accessibility. This thesis focusses on developing a magnetometer-less IMU-based motion capturing system to detect and classify two challenging movement sets: Basic stances during a Shaolin Kung Fu dynamic form, and severity levels from the modified UPDRS (Unified Parkinson’s Disease Rating Scale) analysis tapping exercise. This project has contributed three datasets. The Shaolin Kung Fu dataset is comprised of 5 dynamic movements repeated over 350 times by 8 experienced practitioners. The dataset was labelled by a professional Shaolin Kung Fu master. Two modified UPDRS datasets were constructed, one for each of the two locations measured. The modified UPDRS datasets comprised of 5 severity levels each with 100 self-emulated movement samples. The modified UPDRS dataset was labelled by a researcher in neuropsychological assessment. The errors associated with IMU systems has been reduced significantly through a combination of a Complementary filter and applying the constraints imposed by the range of movements available in human joints. Novel features have been extracted from each dataset. A piecewise feature set based on a moving window approach has been applied to the Shaolin Kung Fu dataset. While a combination of standard statistical features and a Durbin Watson analysis has been extracted from the modified UPDRS measurements. The project has also contributed a comparison of 24 models has been done on all 3 datasets and the optimal model for each dataset has been determined. The resulting models were commensurate with current gold standards. The Shaolin Kung Fu dataset was classified with the computational costly fine decision tree algorithm using 400 splits, resulting in: an accuracy of 98.9%, a precision of 96.9%, a recall value of 99.1%, and a F1-score of 98.0%. A novel approach of using sequential forward feature analysis was used to determine the minimum number of IMU devices required as well as the optimal number of IMU devices. The modified UPDRS datasets were then classified using a support vector machine algorithm requiring various kernels to achieve their highest accuracies. The measurements were repeated with a sensor located on the wrist and finger, with the wrist requiring a linear kernel and the finger a quadratic kernel. Both locations achieved an accuracy, precision, recall, and F1-score of 99.2%. Additionally, the project contributed an evaluation to the effect sensor location has on the proposed models. It was concluded that the IMU-based system has the potential to construct a reference standard both in sports science and clinical movement analysis. Data protection security and communication speeds were limitations in the system constructed due to the measured data being transferred from the devices via Bluetooth Low Energy communication. These limitations were considered and evaluated in the future works of this project

Genetically Engineered Adaptive Resonance Theory (art) Neural Network Architectures

Fuzzy ARTMAP (FAM) is currently considered to be one of the premier neural network architectures in solving classification problems. One of the limitations of Fuzzy ARTMAP that has been extensively reported in the literature is the category proliferation problem. That is Fuzzy ARTMAP has the tendency of increasing its network size, as it is confronted with more and more data, especially if the data is of noisy and/or overlapping nature. To remedy this problem a number of researchers have designed modifications to the training phase of Fuzzy ARTMAP that had the beneficial effect of reducing this phenomenon. In this thesis we propose a new approach to handle the category proliferation problem in Fuzzy ARTMAP by evolving trained FAM architectures. We refer to the resulting FAM architectures as GFAM. We demonstrate through extensive experimentation that an evolved FAM (GFAM) exhibits good (sometimes optimal) generalization, small size (sometimes optimal size), and requires reasonable computational effort to produce an optimal or sub-optimal network. Furthermore, comparisons of the GFAM with other approaches, proposed in the literature, which address the FAM category proliferation problem, illustrate that the GFAM has a number of advantages (i.e. produces smaller or equal size architectures, of better or as good generalization, with reduced computational complexity). Furthermore, in this dissertation we have extended the approach used with Fuzzy ARTMAP to other ART architectures, such as Ellipsoidal ARTMAP (EAM) and Gaussian ARTMAP (GAM) that also suffer from the ART category proliferation problem. Thus, we have designed and experimented with genetically engineered EAM and GAM architectures, named GEAM and GGAM. Comparisons of GEAM and GGAM with other ART architectures that were introduced in the ART literature, addressing the category proliferation problem, illustrate similar advantages observed by GFAM (i.e, GEAM and GGAM produce smaller size ART architectures, of better or improved generalization, with reduced computational complexity). Moverover, to optimally cover the input space of a problem, we proposed a genetically engineered ART architecture that combines the category structures of two different ART networks, FAM and EAM. We named this architecture UART (Universal ART). We analyzed the order of search in UART, that is the order according to which a FAM category or an EAM category is accessed in UART. This analysis allowed us to better understand UART\u27s functionality. Experiments were also conducted to compare UART with other ART architectures, in a similar fashion as GFAM and GEAM were compared. Similar conclusions were drawn from this comparison, as in the comparison of GFAM and GEAM with other ART architectures. Finally, we analyzed the computational complexity of the genetically engineered ART architectures and we compared it with the computational complexity of other ART architectures, introduced into the literature. This analytical comparison verified our claim that the genetically engineered ART architectures produce better generalization and smaller sizes ART structures, at reduced computational complexity, compared to other ART approaches. In review, a methodology was introduced of how to combine the answers (categories) of ART architectures, using genetic algorithms. This methodology was successfully applied to FAM, EAM and FAM and EAM ART architectures, with success, resulting in ART neural networks which outperformed other ART architectures, previously introduced into the literature, and quite often produced ART architectures that attained optimal classification results, at reduced computational complexity

Bayesian multiple extended target tracking using labelled random finite sets and splines

In this paper, we propose a technique for the joint tracking and labelling of multiple extended targets. To achieve multiple extended target tracking using this technique, models for the target measurement rate, kinematic component and target extension are defined and jointly propagated in time under the generalised labelled multi-Bernoulli (GLMB) filter framework. In particular, we developed a Poisson mixture variational Bayesian (PMVB) model to simultaneously estimate the measurement rate of multiple extended targets and extended target extension was modelled using B-splines. We evaluated our proposed method with various performance metrics. Results demonstrate the effectiveness of our approach