Suivi Multi-Locuteurs avec des Informations Audio-Visuelles pour la Perception des Robots

Robot perception plays a crucial role in human-robot interaction (HRI). Perception system provides the robot information of the surroundings and enables the robot to give feedbacks. In a conversational scenario, a group of people may chat in front of the robot and move freely. In such situations, robots are expected to understand where are the people, who are speaking, or what are they talking about. This thesis concentrates on answering the first two questions, namely speaker tracking and diarization. We use different modalities of the robot’s perception system to achieve the goal. Like seeing and hearing for a human-being, audio and visual information are the critical cues for a robot in a conversational scenario. The advancement of computer vision and audio processing of the last decade has revolutionized the robot perception abilities. In this thesis, we have the following contributions: we first develop a variational Bayesian framework for tracking multiple objects. The variational Bayesian framework gives closed-form tractable problem solutions, which makes the tracking process efficient. The framework is first applied to visual multiple-person tracking. Birth and death process are built jointly with the framework to deal with the varying number of the people in the scene. Furthermore, we exploit the complementarity of vision and robot motorinformation. On the one hand, the robot’s active motion can be integrated into the visual tracking system to stabilize the tracking. On the other hand, visual information can be used to perform motor servoing. Moreover, audio and visual information are then combined in the variational framework, to estimate the smooth trajectories of speaking people, and to infer the acoustic status of a person- speaking or silent. In addition, we employ the model to acoustic-only speaker localization and tracking. Online dereverberation techniques are first applied then followed by the tracking system. Finally, a variant of the acoustic speaker tracking model based on von-Mises distribution is proposed, which is specifically adapted to directional data. All the proposed methods are validated on datasets according to applications.La perception des robots joue un rôle crucial dans l’interaction homme-robot (HRI). Le système de perception fournit les informations au robot sur l’environnement, ce qui permet au robot de réagir en consequence. Dans un scénario de conversation, un groupe de personnes peut discuter devant le robot et se déplacer librement. Dans de telles situations, les robots sont censés comprendre où sont les gens, ceux qui parlent et de quoi ils parlent. Cette thèse se concentre sur les deux premières questions, à savoir le suivi et la diarisation des locuteurs. Nous utilisons différentes modalités du système de perception du robot pour remplir cet objectif. Comme pour l’humain, l’ouie et la vue sont essentielles pour un robot dans un scénario de conversation. Les progrès de la vision par ordinateur et du traitement audio de la dernière décennie ont révolutionné les capacités de perception des robots. Dans cette thèse, nous développons les contributions suivantes : nous développons d’abord un cadre variationnel bayésien pour suivre plusieurs objets. Le cadre bayésien variationnel fournit des solutions explicites, rendant le processus de suivi très efficace. Cette approche est d’abord appliqué au suivi visuel de plusieurs personnes. Les processus de créations et de destructions sont en adéquation avecle modèle probabiliste proposé pour traiter un nombre variable de personnes. De plus, nous exploitons la complémentarité de la vision et des informations du moteur du robot : d’une part, le mouvement actif du robot peut être intégré au système de suivi visuel pour le stabiliser ; d’autre part, les informations visuelles peuvent être utilisées pour effectuer l’asservissement du moteur. Par la suite, les informations audio et visuelles sont combinées dans le modèle variationnel, pour lisser les trajectoires et déduire le statut acoustique d’une personne : parlant ou silencieux. Pour experimenter un scenario où l’informationvisuelle est absente, nous essayons le modèle pour la localisation et le suivi des locuteurs basé sur l’information acoustique uniquement. Les techniques de déréverbération sont d’abord appliquées, dont le résultat est fourni au système de suivi. Enfin, une variante du modèle de suivi des locuteurs basée sur la distribution de von-Mises est proposée, celle-ci étant plus adaptée aux données directionnelles. Toutes les méthodes proposées sont validées sur des bases de données specifiques à chaque application

Localization of sound sources : a systematic review

Sound localization is a vast field of research and advancement which is used in many useful applications to facilitate communication, radars, medical aid, and speech enhancement to but name a few. Many different methods are presented in recent times in this field to gain benefits. Various types of microphone arrays serve the purpose of sensing the incoming sound. This paper presents an overview of the importance of using sound localization in different applications along with the use and limitations of ad-hoc microphones over other microphones. In order to overcome these limitations certain approaches are also presented. Detailed explanation of some of the existing methods that are used for sound localization using microphone arrays in the recent literature is given. Existing methods are studied in a comparative fashion along with the factors that influence the choice of one method over the others. This review is done in order to form a basis for choosing the best fit method for our use

Performance analysis of a circular statistics based filter for pedestrian indoor tracking with bearings only measurements provided by low cost sensors

Trabalho de Conclusão de Curso (graduação)—Universidade de Brasília, Faculdade de Tecnologia, Departamento de Engenharia Elétrica, 2018.A literatura recente sugere que a estatística circular se apresenta uma ferramenta valiosa em algoritmos de filtragem que envolvem quantidades angulares, permitindo um tratamento correto das distribuições dessas. Nesse contexto, esse trabalho almeja proporcionar uma análise da performance de um de um algoritmo de filtragem, baseado em estatística circular, para rastreamento de pedestres em ambientes fechados. As medidas para o esquema de localização são oriundas de um sensor de baixo custo baseado em técnicas de direção de chegada (DOA). Dessa forma, um tratamento correto da incerteza nas medidas será abordado. Simulações realizadas no MATLAB indicam que o filtro apresentado possui, quando a incerteza é alta, uma performance superior à abordagens que utilizam EKF e UKF. Portanto, os resultados inspiram o uso do filtro apresentado para implementações físicas.Recent literature suggests that circular statistics presents itself as a powerful tool in filtering algorithms that involve angular quantities, allowing a correct treatment of their distributions. In this context, this work aims to provide a performance analysis of a circular statistics based filter algorithm for indoor pedestrian tracking. The measurements for the localization scheme are provided by low cost sensors based on direction of arrival (DOA) techniques. Hence, a correct treatment to the uncertainty of the measurements will be addressed. Simulation experiments carried out in MATLAB indicate that the presented filter outperforms EKF and UKF approaches when the uncertainty is high. Therefore, results inspire the use of the presented filter for physical implementations

First applications of sound-based control on a mobile robot equipped with two microphones

International audience— This paper validates experimentally a novel approach to robot audition, sound-based control, which consists in introducing auditory features directly as inputs of a closed-loop control scheme, that is, without any explicit localization process. The applications we present rely on the implicit bearings of the sound sources computed from the time difference of arrival (TDOA) between two microphones. By linking the motion of the robot to the aural perception of the environment, this approach has the benefit of being more robust to reverberation and noise. Therefore neither complex tracking method such as Kalman filtering nor TDOA enhancement with denoising or dereverberation methods are needed to track the correct TDOA measurements. The experiments conducted on a mobile robot instrumented with a pair of microphones show the validity of our approach. In a reverberating and noisy room, this approach is able to orient the robot to a mobile sound source in real time. A positioning task with respect to two sound sources is also performed while the robot perception is disturbed by altered and spurious TDOA measurements

Acoustic SLAM

An algorithm is presented that enables devices equipped with microphones, such as robots, to move within their environment in order to explore, adapt to and interact with sound sources of interest. Acoustic scene mapping creates a 3D representation of the positional information of sound sources across time and space. In practice, positional source information is only provided by Direction-of-Arrival (DoA) estimates of the source directions; the source-sensor range is typically difficult to obtain. DoA estimates are also adversely affected by reverberation, noise, and interference, leading to errors in source location estimation and consequent false DoA estimates. Moroever, many acoustic sources, such as human talkers, are not continuously active, such that periods of inactivity lead to missing DoA estimates. Withal, the DoA estimates are specified relative to the observer's sensor location and orientation. Accurate positional information about the observer therefore is crucial. This paper proposes Acoustic Simultaneous Localization and Mapping (aSLAM), which uses acoustic signals to simultaneously map the 3D positions of multiple sound sources whilst passively localizing the observer within the scene map. The performance of aSLAM is analyzed and evaluated using a series of realistic simulations. Results are presented to show the impact of the observer motion and sound source localization accuracy

Directional statistics and filtering using libDirectional

In this paper, we present libDirectional, a MATLAB library for directional statistics and directional estimation. It supports a variety of commonly used distributions on the unit circle, such as the von Mises, wrapped normal, and wrapped Cauchy distributions. Furthermore, various distributions on higher-dimensional manifolds such as the unit hypersphere and the hypertorus are available. Based on these distributions, several recursive filtering algorithms in libDirectional allow estimation on these manifolds. The functionality is implemented in a clear, well-documented, and object-oriented structure that is both easy to use and easy to extend

ROBOTIC SOUND SOURCE LOCALIZATION AND TRACKING USING BIO-INSPIRED MINIATURE ACOUSTIC SENSORS

Sound source localization and tracking using auditory systems has been widely investigated for robotics applications due to their inherent advantages over other systems, such as vision based systems. Most existing robotic sound localization and tracking systems utilize conventional microphone arrays with different arrangements, which are inherently limited by a size constraint and are thus difficult to implement on miniature robots. To overcome the size constraint, sensors that mimic the mechanically coupled ear of fly Ormia have been previously developed. However, there has not been any attempt to study robotic sound source localization and tracking with these sensors. In this dissertation, robotic sound source localization and tracking using the miniature fly-ear-inspired sensors are studied for the first time. First, through investigation into the Cramer Rao lower bound (CRLB) and variance of the sound incident angle estimation, an enhanced understanding of the influence of the mechanical coupling on the performance of the fly-ear inspired sensor for sound localization is achieved. It is found that due to the mechanical coupling between the membranes, at its working frequency, the fly-ear inspired sensor can achieve an estimation of incident angle that is 100 time better than that of the conventional microphone pair with same signal-to-noise ratio in detection of the membrane deflection. Second, development of sound localization algorithms that can be used for robotic sound source localization and tracking using the fly-ear inspired sensors is carried out. Two methods are developed to estimate the sound incident angle based on the sensor output. One is based on model-free gradient descent method and the other is based on fuzzy logic. In the first approach, different localization schemes and different objective functions are investigated through numerical simulations, in which two-dimensional sound source localization is achieved without ambiguity. To address the slow convergence due to the iterative nature of the first approach, a novel fuzzy logic model of the fly-ear sensor is developed in the second approach for sound incident angle estimation. This model is studied in both simulations and experiments for localization of a stationary source and tracking a moving source in one dimension with a good performance. Third, nonlinear and quadratic-linear controllers are developed for control of the kinematics of a robot for sound source localization and tracking, which is implemented later in a mobile platform equipped with a microphone pair. Both homing onto a stationary source and tracking of a moving source with pre-defined paths are successfully demonstrated. Through this dissertation work, new knowledge on robotic sound source localization and tracking using fly-ear inspired sensors is created, which can serve as a basis for future study of sound source localization and tracking with miniature robots

Laboratorij za autonomne sustave i mobilnu robotiku

Laboratorij za autonomne sustave i mobilnu robotiku (LAMOR) istraživački je laboratorij koji djeluje u okviru Zavoda za automatiku i računalno inženjerstvo Fakulteta elektrotehnike i računarstva Sveučilišta u Zagrebu. LAMOR je jedan od vodećih laboratorija u Republici Hrvatskoj u području robotike, a svoje je istraživačko djelovanje usmjerio na temeljna istraživanja algoritama upravljanja, estimacije i umjetne inteligencije s primjenom u razvoju sustava autonomije mobilnih robota i vozila u nepoznatim i dinamičkim okruženjima te sustava djelotvorne i sigurne interakcije autonomnih mobilnih robota i ljudi. U radu je opisana uspostava laboratorija i prikazan je njegov 20-godišnji razvoj, a potom su predstavljene osnovne informacije o njegovoj istraživačkoj djelatnosti, najznačajnijim znanstvenim postignućima, najvažnijim istraživačkim projektima, međunarodnoj suradnji te doprinosu razvoju znanosti i gospodarstva u Republici Hrvatskoj

Laboratorij za autonomne sustave i mobilnu robotiku

Laboratorij za autonomne sustave i mobilnu robotiku (LAMOR) istraživački je laboratorij koji djeluje u okviru Zavoda za automatiku i računalno inženjerstvo Fakulteta elektrotehnike i računarstva Sveučilišta u Zagrebu. LAMOR je jedan od vodećih laboratorija u Republici Hrvatskoj u području robotike, a svoje je istraživačko djelovanje usmjerio na temeljna istraživanja algoritama upravljanja, estimacije i umjetne inteligencije s primjenom u razvoju sustava autonomije mobilnih robota i vozila u nepoznatim i dinamičkim okruženjima te sustava djelotvorne i sigurne interakcije autonomnih mobilnih robota i ljudi. U radu je opisana uspostava laboratorija i prikazan je njegov 20-godišnji razvoj, a potom su predstavljene osnovne informacije o njegovoj istraživačkoj djelatnosti, najznačajnijim znanstvenim postignućima, najvažnijim istraživačkim projektima, međunarodnoj suradnji te doprinosu razvoju znanosti i gospodarstva u Republici Hrvatskoj