Tracking and Fusion Methods for Extended Targets Parameterized by Center, Orientation, and Semi-axes

The improvements in sensor technology, e.g., the development of automotive Radio Detection and Ranging (RADAR) or Light Detection and Ranging (LIDAR), which are able to provide a higher detail of the sensor’s environment, have introduced new opportunities but also new challenges to target tracking. In classic target tracking, targets are assumed as points. However, this assumption is no longer valid if targets occupy more than one sensor resolution cell, creating the need for extended targets, modeling the shape in addition to the kinematic parameters. Different shape models are possible and this thesis focuses on an elliptical shape, parameterized with center, orientation, and semi-axes lengths. This parameterization can be used to model rectangles as well. Furthermore, this thesis is concerned with multi-sensor fusion for extended targets, which can be used to improve the target tracking by providing information gathered from different sensors or perspectives. We also consider estimation of extended targets, i.e., to account for uncertainties, the target is modeled by a probability density, so we need to find a so-called point estimate. Extended target tracking provides a variety of challenges due to the spatial extent, which need to be handled, even for basic shapes like ellipses and rectangles. Among these challenges are the choice of the target model, e.g., how the measurements are distributed across the shape. Additional challenges arise for sensor fusion, as it is unclear how to best consider the geometric properties when combining two extended targets. Finally, the extent needs to be involved in the estimation. Traditional methods often use simple uniform distributions across the shape, which do not properly portray reality, while more complex methods require the use of optimization techniques or large amounts of data. In addition, for traditional estimation, metrics such as the Euclidean distance between state vectors are used. However, they might no longer be valid because they do not consider the geometric properties of the targets’ shapes, e.g., rotating an ellipse by 180 degree results in the same ellipse, but the Euclidean distance between them is not 0. In multi-sensor fusion, the same holds, i.e., simply combining the corresponding elements of the state vectors can lead to counter-intuitive fusion results. In this work, we compare different elliptic trackers and discuss more complex measurement distributions across the shape’s surface or contour. Furthermore, we discuss the problems which can occur when fusing extended target estimates from different sensors and how to handle them by providing a transformation into a special density. We then proceed to discuss how a different metric, namely the Gaussian Wasserstein (GW) distance, can be used to improve target estimation. We define an estimator and propose an approximation based on an extension of the square root distance. It can be applied on the posterior densities of the aforementioned trackers to incorporate the unique properties of ellipses in the estimation process. We also discuss how this can be applied to rectangular targets as well. Finally, we evaluate and discuss our approaches. We show the benefits of more complex target models in simulations and on real data and we demonstrate our estimation and fusion approaches compared to classic methods on simulated data.2022-01-2

Automotive Target Models for Point Cloud Sensors

One of the major challenges to enable automated driving is the perception of other road users in the host vehicle’s vicinity. Various automotive sensors that provide detailed information about other traffic participants have been developed to handle this challenge. Of particular interest for this work are Light Detection and Ranging (LIDAR) and Radio Detection and Ranging (RADAR) sensors, which generate multiple, spatially distributed, noise corrupted point measurements on other traffic participants. Based on these point measurements, the traffic participant’s kinematic and shape parameters have to be estimated. The choice of a suitable extent model is paramount to accurately track a target’s position, orientation and other parameters. How well a model performs typically depends on the type of target that has to be tracked, e.g. pedestrians, bikes or cars, as well as the sensor’s setup and measurement principle itself. This work considers the creation of extended object models and corresponding inference strategies for tracking automotive vehicles based on accumulated point cloud data. We gain insights into the extended object model’s requirements by analysing automotive LIDAR and RADAR sensor data. This analysis aids in the identification of relevant features from the measurement’s spatial distribution and their incorporation into an accurate target model. The analysis lays the foundation for our main contributions. We developed a constrained Spline-based geometric representation and a corresponding inference strategy for the contour of cars in LIDAR data. We further developed a heuristic to account for the integration of the measurement distribution on cars, generated by LIDAR sensors mounted on the roof of the recording vessel. Last, we developed an extended target model for cars based on automotive RADAR sensors. The model provides an interpretation of a learned Gaussian Mixture Model (GMM) as scatter sources and uses the Probabilistic Multi-Hypothesis Tracker (PMHT) to formulate a closed form Maximum a Posteriori (MAP) update. All developed approaches are evaluated on real world data sets.2022-02-0

A New Wave in Robotics: Survey on Recent mmWave Radar Applications in Robotics

We survey the current state of millimeterwave (mmWave) radar applications in robotics with a focus on unique capabilities, and discuss future opportunities based on the state of the art. Frequency Modulated Continuous Wave (FMCW) mmWave radars operating in the 76--81GHz range are an appealing alternative to lidars, cameras and other sensors operating in the near visual spectrum. Radar has been made more widely available in new packaging classes, more convenient for robotics and its longer wavelengths have the ability to bypass visual clutter such as fog, dust, and smoke. We begin by covering radar principles as they relate to robotics. We then review the relevant new research across a broad spectrum of robotics applications beginning with motion estimation, localization, and mapping. We then cover object detection and classification, and then close with an analysis of current datasets and calibration techniques that provide entry points into radar research.Comment: 19 Pages, 11 Figures, 2 Tables, TRO Submission pendin

Anomaly Detection in Autonomous Driving: A Survey

Nowadays, there are outstanding strides towards a future with autonomous vehicles on our roads. While the perception of autonomous vehicles performs well under closed-set conditions, they still struggle to handle the unexpected. This survey provides an extensive overview of anomaly detection techniques based on camera, lidar, radar, multimodal and abstract object level data. We provide a systematization including detection approach, corner case level, ability for an online application, and further attributes. We outline the state-of-the-art and point out current research gaps.Comment: Daniel Bogdoll and Maximilian Nitsche contributed equally. Accepted for publication at CVPR 2022 WAD worksho

Novel Hybrid-Learning Algorithms for Improved Millimeter-Wave Imaging Systems

Increasing attention is being paid to millimeter-wave (mmWave), 30 GHz to 300 GHz, and terahertz (THz), 300 GHz to 10 THz, sensing applications including security sensing, industrial packaging, medical imaging, and non-destructive testing. Traditional methods for perception and imaging are challenged by novel data-driven algorithms that offer improved resolution, localization, and detection rates. Over the past decade, deep learning technology has garnered substantial popularity, particularly in perception and computer vision applications. Whereas conventional signal processing techniques are more easily generalized to various applications, hybrid approaches where signal processing and learning-based algorithms are interleaved pose a promising compromise between performance and generalizability. Furthermore, such hybrid algorithms improve model training by leveraging the known characteristics of radio frequency (RF) waveforms, thus yielding more efficiently trained deep learning algorithms and offering higher performance than conventional methods. This dissertation introduces novel hybrid-learning algorithms for improved mmWave imaging systems applicable to a host of problems in perception and sensing. Various problem spaces are explored, including static and dynamic gesture classification; precise hand localization for human computer interaction; high-resolution near-field mmWave imaging using forward synthetic aperture radar (SAR); SAR under irregular scanning geometries; mmWave image super-resolution using deep neural network (DNN) and Vision Transformer (ViT) architectures; and data-level multiband radar fusion using a novel hybrid-learning architecture. Furthermore, we introduce several novel approaches for deep learning model training and dataset synthesis.Comment: PhD Dissertation Submitted to UTD ECE Departmen

Aprendizagem automática aplicada à deteção de pessoas baseada em radar

The present dissertation describes the development and implementation of a radar-based system with the purpose of being able to detect people amidst other objects that are moving in an indoor scenario. The detection methods implemented exploit radar data that is processed by a system that includes the data acquisition, the pre-processing of the data, the feature extraction, and the application of these data to machine learning models specifically designed to attain the objective of target classification. Beyond the basic theoretical research necessary for its sucessful development, the work contamplates an important component of software development and experimental tests. Among others, the following topics were covered in this dissertation: the study of radar working principles and hardware; radar signal processing; techniques of clutter removal, feature exctraction, and data clustering applied to radar signals; implementation and hyperparameter tuning of machine learning classification systems; study of multi-target detection and tracking methods. The people detection application was tested in different indoor scenarios that include a static radar and a radar dynamically deployed by a mobile robot. This application can be executed in real time and perform multiple target detection and classification using basic clustering and tracking algorithms. A study of the effects of the detection of multiple targets in the performance of the application, as well as an assessment of the efficiency of the different classification methods is presented. The envisaged applications of the proposed detection system include intrusion detection in indoor environments and acquisition of anonymized data for people tracking and counting in public spaces such as hospitals and schools.A presente dissertação descreve o desenvolvimento e implementação de um sistema baseado em radar que tem como objetivo detetar e distinguir pessoas de outros objetos que se movem num ambiente interior. Os métodos de deteção e distinção exploram os dados de radar que são processados por um sistema que abrange a aquisição e pré-processamento dos dados, a extração de características, e a aplicação desses dados a modelos de aprendizagem automática especificamente desenhados para atingir o objetivo de classificação de alvos. Além do estudo da teoria básica de radar para o desenvolvimento bem sucedido desta dissertação, este trabalho contempla uma componente importante de desenvolvimento de software e testes experimentais. Entre outros, os seguintes tópicos foram abordados nesta dissertação: o estudo dos princípios básicos do funcionamento do radar e do seu equipamento; processamento de sinal do radar; técnicas de remoção de ruído, extração de características, e segmentação de dados aplicada ao sinal de radar; implementação e calibração de hiper-parâmetros dos modelos de aprendizagem automática para sistemas de classificação; estudo de métodos de deteção e seguimento de múltiplos alvos. A aplicação para deteção de pessoas foi testada em diferentes cenários interiores que incluem o radar estático ou transportado por um robot móvel. Esta aplicação pode ser executada em tempo real e realizar deteção e classificação de múltiplos alvos usando algoritmos básicos de segmentação e seguimento. O estudo do impacto da deteção de múltiplos alvos no funcionamento da aplicação é apresentado, bem como a avaliação da eficiência dos diferentes métodos de classificação usados. As possíveis aplicações do sistema de deteção proposto incluem a deteção de intrusão em ambientes interiores e aquisição de dados anónimos para seguimento e contagem de pessoas em espaços públicos tais como hospitais ou escolas.Mestrado em Engenharia de Computadores e Telemátic

Three-Dimensional Extended Object Tracking and Shape Learning Using Gaussian Processes

In this study, we investigate the problem of tracking objects with unknown shapes using three-dimensional (3D) point cloud data. We propose a Gaussian process-based model to jointly estimate object kinematics, including position, orientation and velocities, together with the shape of the object for online and offline applications. We describe the unknown shape by a radial function in 3D, and induce a correlation structure via a Gaussian process. Furthermore, we propose an efficient algorithm to reduce the computational complexity of working with 3D data. This is accomplished by casting the tracking problem into projection planes which are attached to the object's local frame. The resulting algorithms can process 3D point cloud data and accomplish tracking of a dynamic object. Furthermore, they provide analytical expressions for the representation of the object shape in 3D, together with confidence intervals. The confidence intervals, which quantify the uncertainty in the shape estimate, can later be used for solving the gating and association problems inherent in object tracking. The performance of the methods is demonstrated both on simulated and real data. The results are compared with an existing random matrix model, which is commonly used for extended object tracking in the literature

Long-Term Localization for Self-Driving Cars

Long-term localization is hard due to changing conditions, while relative localization within time sequences is much easier. To achieve long-term localization in a sequential setting, such as, for self-driving cars, relative localization should be used to the fullest extent, whenever possible.This thesis presents solutions and insights both for long-term sequential visual localization, and localization using global navigational satellite systems (GNSS), that push us closer to the goal of accurate and reliable localization for self-driving cars. It addresses the question: How to achieve accurate and robust, yet cost-effective long-term localization for self-driving cars?Starting in this question, the thesis explores how existing sensor suites for advanced driver-assistance systems (ADAS) can be used most efficiently, and how landmarks in maps can be recognized and used for localization even after severe changes in appearance. The findings show that:* State-of-the-art ADAS sensors are insufficient to meet the requirements for localization of a self-driving car in less than ideal conditions.GNSS and visual localization are identified as areas to improve.\ua0* Highly accurate relative localization with no convergence delay is possible by using time relative GNSS observations with a single band receiver, and no base stations.\ua0* Sequential semantic localization is identified as a promising focus point for further research based on a benchmark study comparing state-of-the-art visual localization methods in challenging autonomous driving scenarios including day-to-night and seasonal changes.\ua0* A novel sequential semantic localization algorithm improves accuracy while significantly reducing map size compared to traditional methods based on matching of local image features.\ua0* Improvements for semantic segmentation in challenging conditions can be made efficiently by automatically generating pixel correspondences between images from a multitude of conditions and enforcing a consistency constraint during training.\ua0* A segmentation algorithm with automatically defined and more fine-grained classes improves localization performance.\ua0* The performance advantage seen in single image localization for modern local image features, when compared to traditional ones, is all but erased when considering sequential data with odometry, thus, encouraging to focus future research more on sequential localization, rather than pure single image localization