Trajectory Poisson multi-Bernoulli mixture filter for traffic monitoring using a drone

This paper proposes a multi-object tracking (MOT) algorithm for traffic monitoring using a drone equipped with optical and thermal cameras. Object detections on the images are obtained using a neural network for each type of camera. The cameras are modelled as direction-of-arrival (DOA) sensors. Each DOA detection follows a von-Mises Fisher distribution, whose mean direction is obtain by projecting a vehicle position on the ground to the camera. We then use the trajectory Poisson multi-Bernoulli mixture filter (TPMBM), which is a Bayesian MOT algorithm, to optimally estimate the set of vehicle trajectories. We have also developed a parameter estimation algorithm for the measurement model. We have tested the accuracy of the resulting TPMBM filter in synthetic and experimental data sets

Acoustic Speaker Localization with Strong Reverberation and Adaptive Feature Filtering with a Bayes RFS Framework

The thesis investigates the challenges of speaker localization in presence of strong reverberation, multi-speaker tracking, and multi-feature multi-speaker state filtering, using sound recordings from microphones. Novel reverberation-robust speaker localization algorithms are derived from the signal and room acoustics models. A multi-speaker tracking filter and a multi-feature multi-speaker state filter are developed based upon the generalized labeled multi-Bernoulli random finite set framework. Experiments and comparative studies have verified and demonstrated the benefits of the proposed methods

Ground moving target tracking with space-time adaptive radar

Ground moving target tracking by airborne radar provides situational awareness of vehicle movements in the supervised region. Vehicles are detected by applying space time adaptive processing to the received multi channel radar data. The detections are then fed to a tracking algorithm that processes them to tracks. In literature, radar signal processing and ground target tracking are treated as two separate topics and results are not validated by experimental data. The first objective of this thesis is to provide a closer link between these fields. The second objective is to show that tracking performance can be improved by providing additional data from the radar signal processing to the tracking step. The third objective is to validate the algorithm and the performance improvement using experimental data. As a result this thesis presents a unified treatment of ground moving target tracking from radar raw data to established tracks. A complete reference algorithm for ground moving target tracking based on the Gaussian mixture probability hypothesis density filter is presented. In particular, Jacobians of the observation process are derived. They are presented in such a form that immediate implementation in a programming language is possible. In the course of this thesis a measurement campaign with the experimental radar PAMIR of Fraunhofer FHR was conducted. The experiment included two GPS equipped reference vehicles and a multitude of targets of opportunity. Tracking results obtained with this experimental data and the reference tracking algorithm of this thesis are shown. The thesis also enhances the reference target tracking algorithm by a parameter that characterizes the variance of the direction of arrival measurement of the target signal. This parameter is determined adaptively depending on the estimated signal strength and the clutter background. The major contribution with regard to this enhancement is a thorough experimental validation: Firstly, a comparison between GPS based measurements and radar based measurements of the direction of arrival shows that this variance captures the distribution of measurement errors excellently. Secondly, tracking results are compared to the GPS tracks of the ground truth vehicles. It is found that the enhanced algorithm yields superior track quality with respect to both track accuracy and track continuity.Bodenzielverfolgung mit luftgestütztem Radar liefert das Lagebild von Fahrzeug­bewegungen innerhalb des beobachteten Gebiets. Fahrzeuge werden durch die Anwendung von Raum-Zeit adaptiver Signalverarbeitung (STAP) entdeckt. Die Entdeckungen werden dann von einem Zielverfolgungsalgorithmus zu Zielspuren verarbeitet. In der Literatur werden Radarsignalverarbeitung und Zielverfolgung als zwei getrennte Forschungsfelder behandelt und die Bodenzielverfolgung wird nicht anhand von Realdaten validiert. Das erste Ziel dieser Arbeit ist, eine engere Verbindung zwischen beiden Feldern herzustellen. Das zweite Ziel ist zu zeigen, dass die Qualität der Zielverfolgung durch das Verwenden zusätzlicher, durch die Radarsignalverarbeitung gewonnene Information verbessert werden kann. Das dritte Ziel ist, die Funktionalität der Zielverfolgung und die Verbesserung der Leistung durch experimentelle Realdaten zu belegen. Somit stellt diese Arbeit eine Gesamtbehandlung der Bodenzielverfolgung von den Radar-Rohdaten bis zu Zielspuren dar. Es wird ein vollständiger, auf dem Gaussian Mixture Probability Hypothesis Density Filter basierender Referenzalgorithmus für die Bodenzielverfolgung entwickelt. Insbesondere werden Jacobimatrizen der Beobachtungsfunktion hergeleitet. Sie werden in der Arbeit so dargestellt, dass sie direkt in einer Programmiersprache implementiert werden können. Im Zuge dieser Arbeit wurde ein Zielverfolgungs-Experiment mit dem Experimentalsystem PAMIR des Fraunhofer FHR durchgeführt. In dem Experiment wurden neben einer Vielzahl von Gelegenheitszielen zwei mit GPS-Geräten ausgerüstete Fahrzeuge von dem Radar beobachtet. Auf Basis dieses Experiments und des Referenzalgorithmus werden Zielverfolgungsergebnisse vorgestellt. Darüber hinaus erweitert diese Arbeit den Referenzalgorithmus um einen Parameter, der die Varianz der Richtungsschätzung des Zielsignals charakterisiert. Dieser Parameter wird adaptiv anhand der geschätzten Signalstärke und der Stärke störender Bodenrückstreuungen festgelegt. Der wesentliche Beitrag dieser Arbeit in Bezug auf diese Erweiterung ist eine gründliche experimentelle Validierung. Erstens zeigt der Vergleich von GPS- und Radar-basierten Richtungsschätzungen, dass dieser Parameter die Verteilung des Messfehlers exzellent beschreibt. Zweitens werden Zielverfolgungsergebnisse mit den GPS-Spuren verglichen. Es zeigt sich, dass der erweiterte Algorithmus sowohl in Bezug auf die Spurgenauigkeit als auch in Bezug auf die Spurkontinuität die Zielverfolgung verbessert

Suivi Multi-Locuteurs avec des Informations Audio-Visuelles pour la Perception des Robots

Robot perception plays a crucial role in human-robot interaction (HRI). Perception system provides the robot information of the surroundings and enables the robot to give feedbacks. In a conversational scenario, a group of people may chat in front of the robot and move freely. In such situations, robots are expected to understand where are the people, who are speaking, or what are they talking about. This thesis concentrates on answering the first two questions, namely speaker tracking and diarization. We use different modalities of the robot’s perception system to achieve the goal. Like seeing and hearing for a human-being, audio and visual information are the critical cues for a robot in a conversational scenario. The advancement of computer vision and audio processing of the last decade has revolutionized the robot perception abilities. In this thesis, we have the following contributions: we first develop a variational Bayesian framework for tracking multiple objects. The variational Bayesian framework gives closed-form tractable problem solutions, which makes the tracking process efficient. The framework is first applied to visual multiple-person tracking. Birth and death process are built jointly with the framework to deal with the varying number of the people in the scene. Furthermore, we exploit the complementarity of vision and robot motorinformation. On the one hand, the robot’s active motion can be integrated into the visual tracking system to stabilize the tracking. On the other hand, visual information can be used to perform motor servoing. Moreover, audio and visual information are then combined in the variational framework, to estimate the smooth trajectories of speaking people, and to infer the acoustic status of a person- speaking or silent. In addition, we employ the model to acoustic-only speaker localization and tracking. Online dereverberation techniques are first applied then followed by the tracking system. Finally, a variant of the acoustic speaker tracking model based on von-Mises distribution is proposed, which is specifically adapted to directional data. All the proposed methods are validated on datasets according to applications.La perception des robots joue un rôle crucial dans l’interaction homme-robot (HRI). Le système de perception fournit les informations au robot sur l’environnement, ce qui permet au robot de réagir en consequence. Dans un scénario de conversation, un groupe de personnes peut discuter devant le robot et se déplacer librement. Dans de telles situations, les robots sont censés comprendre où sont les gens, ceux qui parlent et de quoi ils parlent. Cette thèse se concentre sur les deux premières questions, à savoir le suivi et la diarisation des locuteurs. Nous utilisons différentes modalités du système de perception du robot pour remplir cet objectif. Comme pour l’humain, l’ouie et la vue sont essentielles pour un robot dans un scénario de conversation. Les progrès de la vision par ordinateur et du traitement audio de la dernière décennie ont révolutionné les capacités de perception des robots. Dans cette thèse, nous développons les contributions suivantes : nous développons d’abord un cadre variationnel bayésien pour suivre plusieurs objets. Le cadre bayésien variationnel fournit des solutions explicites, rendant le processus de suivi très efficace. Cette approche est d’abord appliqué au suivi visuel de plusieurs personnes. Les processus de créations et de destructions sont en adéquation avecle modèle probabiliste proposé pour traiter un nombre variable de personnes. De plus, nous exploitons la complémentarité de la vision et des informations du moteur du robot : d’une part, le mouvement actif du robot peut être intégré au système de suivi visuel pour le stabiliser ; d’autre part, les informations visuelles peuvent être utilisées pour effectuer l’asservissement du moteur. Par la suite, les informations audio et visuelles sont combinées dans le modèle variationnel, pour lisser les trajectoires et déduire le statut acoustique d’une personne : parlant ou silencieux. Pour experimenter un scenario où l’informationvisuelle est absente, nous essayons le modèle pour la localisation et le suivi des locuteurs basé sur l’information acoustique uniquement. Les techniques de déréverbération sont d’abord appliquées, dont le résultat est fourni au système de suivi. Enfin, une variante du modèle de suivi des locuteurs basée sur la distribution de von-Mises est proposée, celle-ci étant plus adaptée aux données directionnelles. Toutes les méthodes proposées sont validées sur des bases de données specifiques à chaque application

Online Audio-Visual Multi-Source Tracking and Separation: A Labeled Random Finite Set Approach

The dissertation proposes an online solution for separating an unknown and time-varying number of moving sources using audio and visual data. The random finite set framework is used for the modeling and fusion of audio and visual data. This enables an online tracking algorithm to estimate the source positions and identities for each time point. With this information, a set of beamformers can be designed to separate each desired source and suppress the interfering sources

Adaptive detection probability for mmWave 5G SLAM

In 5G simultaneous localization and mapping (SLAM), estimates of angles and delays of mm Wave channels are used to localize the user equipment and map the environment. The interface from the channel estimator to the SLAM method, which was previously limited to the channel parameters estimates and their uncertainties, is here augmented to include the detection probabilities of hypothesized landmarks, given certain a user location. These detection probabilities are used during data association and measurement update, which are important steps in any SLAM method. Due to the nature of mm Wave communication, these detection probabilities depend on the physical layer signal parameters, including beamforming, precoding, bandwidth, observation time, etc. In this paper, we derive these detection probabilities for different deterministic and stochastic channel models and highlight the importance of beamforming

Modeling the Behavior of Multipath Components Pertinent to Indoor Geolocation

Recently, a number of empirical models have been introduced in the literature for the behavior of direct path used in the design of algorithms for RF based indoor geolocation. Frequent absence of direct path has been a major burden on the performance of these algorithms directing researchers to discover algorithms using multipath diversity. However, there is no reliable model for the behavior of multipath components pertinent to precise indoor geolocation. In this dissertation, we first examine the absence of direct path by statistical analysis of empirical data. Then we show how the concept of path persistency can be exploited to obtain accurate ranging using multipath diversity. We analyze the effects of building architecture on the multipath structure by demonstrating the effects of wall length and wall density on the path persistency. Finally, we introduce a comprehensive model for the spatial behavior of multipath components. We use statistical analysis of empirical data obtained by a measurement calibrated ray-tracing tool to model the time-of- arrival, angle-of-arrival and path gains. The relationship between the transmitter-receiver separation and the number of paths are also incorporated in our model. In addition, principles of ray optics are applied to explain the spatial evolution of path gains, time-of-arrival and angle-of-arrival of individual multipath components as a mobile terminal moves inside a typical indoor environment. We also use statistical modeling for the persistency and birth/death rate of the paths