Vision-Guided Robot Hearing

International audienceNatural human-robot interaction (HRI) in complex and unpredictable environments is important with many potential applicatons. While vision-based HRI has been thoroughly investigated, robot hearing and audio-based HRI are emerging research topics in robotics. In typical real-world scenarios, humans are at some distance from the robot and hence the sensory (microphone) data are strongly impaired by background noise, reverberations and competing auditory sources. In this context, the detection and localization of speakers plays a key role that enables several tasks, such as improving the signal-to-noise ratio for speech recognition, speaker recognition, speaker tracking, etc. In this paper we address the problem of how to detect and localize people that are both seen and heard. We introduce a hybrid deterministic/probabilistic model. The deterministic component allows us to map 3D visual data onto an 1D auditory space. The probabilistic component of the model enables the visual features to guide the grouping of the auditory features in order to form audiovisual (AV) objects. The proposed model and the associated algorithms are implemented in real-time (17 FPS) using a stereoscopic camera pair and two microphones embedded into the head of the humanoid robot NAO. We perform experiments with (i)~synthetic data, (ii)~publicly available data gathered with an audiovisual robotic head, and (iii)~data acquired using the NAO robot. The results validate the approach and are an encouragement to investigate how vision and hearing could be further combined for robust HRI

Developing a Home Service Robot Platform for Smart Homes

The purpose of this work is to develop a testbed for a smart home environment integrated with a home service robot (ASH Testbed) as well as to build home service robot platforms. The architecture of ASH Testbed was proposed and implemented based on ROS (Robot Operating System). In addition, two robot platforms, ASCCHomeBots, were developed using an iRobot Create base and a Pioneer base. They are equipped with capabilities such as mapping, autonomous navigation. They are also equipped with the natural human interfaces including hand-gesture recognition using a RGB-D camera, online speech recognition through cloud computing services provided by Google, and local speech recognition based on PocketSphinx. Furthermore, the Pioneer based ASCCHomeBot was developed along with an open audition system. This allows the robot to serve the elderly living alone at home. We successfully implemented the software for this system that realizes robot services and audition services for high level applications such as telepresence video conference, sound source position estimation, multiple source speech recognition, and human assisted sound classification. Our experimental results validated the proposed framework and the effectiveness of the developed robots as well as the proposed testbed.Electrical Engineerin

Scene analysis in the natural environment

The problem of scene analysis has been studied in a number of different fields over the past decades. These studies have led to a number of important insights into problems of scene analysis, but not all of these insights are widely appreciated. Despite this progress, there are also critical shortcomings in current approaches that hinder further progress. Here we take the view that scene analysis is a universal problem solved by all animals, and that we can gain new insight by studying the problems that animals face in complex natural environments. In particular, the jumping spider, songbird, echolocating bat, and electric fish, all exhibit behaviors that require robust solutions to scene analysis problems encountered in the natural environment. By examining the behaviors of these seemingly disparate animals, we emerge with a framework for studying analysis comprising four essential properties: 1) the ability to solve ill-posed problems, 2) the ability to integrate and store information across time and modality, 3) efficient recovery and representation of 3D scene structure, and 4) the use of optimal motor actions for acquiring information to progress towards behavioral goals

Rehabilitation of gait after stroke: a review towards a top-down approach

This document provides a review of the techniques and therapies used in gait rehabilitation after stroke. It also examines the possible benefits of including assistive robotic devices and brain-computer interfaces in this field, according to a top-down approach, in which rehabilitation is driven by neural plasticity

Advances in Human-Robot Interaction

Rapid advances in the field of robotics have made it possible to use robots not just in industrial automation but also in entertainment, rehabilitation, and home service. Since robots will likely affect many aspects of human existence, fundamental questions of human-robot interaction must be formulated and, if at all possible, resolved. Some of these questions are addressed in this collection of papers by leading HRI researchers

Spatial-Temporal Characteristics of Multisensory Integration

abstract: We experience spatial separation and temporal asynchrony between visual and haptic information in many virtual-reality, augmented-reality, or teleoperation systems. Three studies were conducted to examine the spatial and temporal characteristic of multisensory integration. Participants interacted with virtual springs using both visual and haptic senses, and their perception of stiffness and ability to differentiate stiffness were measured. The results revealed that a constant visual delay increased the perceived stiffness, while a variable visual delay made participants depend more on the haptic sensations in stiffness perception. We also found that participants judged stiffness stiffer when they interact with virtual springs at faster speeds, and interaction speed was positively correlated with stiffness overestimation. In addition, it has been found that participants could learn an association between visual and haptic inputs despite the fact that they were spatially separated, resulting in the improvement of typing performance. These results show the limitations of Maximum-Likelihood Estimation model, suggesting that a Bayesian inference model should be used.Dissertation/ThesisDoctoral Dissertation Human Systems Engineering 201