Co-Localization of Audio Sources in Images Using Binaural Features and Locally-Linear Regression

This paper addresses the problem of localizing audio sources using binaural measurements. We propose a supervised formulation that simultaneously localizes multiple sources at different locations. The approach is intrinsically efficient because, contrary to prior work, it relies neither on source separation, nor on monaural segregation. The method starts with a training stage that establishes a locally-linear Gaussian regression model between the directional coordinates of all the sources and the auditory features extracted from binaural measurements. While fixed-length wide-spectrum sounds (white noise) are used for training to reliably estimate the model parameters, we show that the testing (localization) can be extended to variable-length sparse-spectrum sounds (such as speech), thus enabling a wide range of realistic applications. Indeed, we demonstrate that the method can be used for audio-visual fusion, namely to map speech signals onto images and hence to spatially align the audio and visual modalities, thus enabling to discriminate between speaking and non-speaking faces. We release a novel corpus of real-room recordings that allow quantitative evaluation of the co-localization method in the presence of one or two sound sources. Experiments demonstrate increased accuracy and speed relative to several state-of-the-art methods.Comment: 15 pages, 8 figure

Decoding neural responses to temporal cues for sound localization

The activity of sensory neural populations carries information about the environment. This may be extracted from neural activity using different strategies. In the auditory brainstem, a recent theory proposes that sound location in the horizontal plane is decoded from the relative summed activity of two populations in each hemisphere, whereas earlier theories hypothesized that the location was decoded from the identity of the most active cells. We tested the performance of various decoders of neural responses in increasingly complex acoustical situations, including spectrum variations, noise, and sound diffraction. We demonstrate that there is insufficient information in the pooled activity of each hemisphere to estimate sound direction in a reliable way consistent with behavior, whereas robust estimates can be obtained from neural activity by taking into account the heterogeneous tuning of cells. These estimates can still be obtained when only contralateral neural responses are used, consistently with unilateral lesion studies. DOI: http://dx.doi.org/10.7554/eLife.01312.001

Effects of cochlear implantation on binaural hearing in adults with unilateral hearing loss

A FDA clinical trial was carried out to evaluate the potential benefit of cochlear implant (CI) use for adults with unilateral moderate-to-profound sensorineural hearing loss. Subjects were 20 adults with moderate-to-profound unilateral sensorineural hearing loss and normal or near-normal hearing on the other side. A MED-EL standard electrode was implanted in the impaired ear. Outcome measures included: (a) sound localization on the horizontal plane (11 positions, −90° to 90°), (b) word recognition in quiet with the CI alone, and (c) masked sentence recognition with the target at 0° and the masker at −90°, 0°, or 90°. This battery was completed preoperatively and at 1, 3, 6, 9, and 12 months after CI activation. Normative data were also collected for 20 age-matched control subjects with normal or near-normal hearing bilaterally. The CI improved localization accuracy and reduced side bias. Word recognition with the CI alone was similar to performance of traditional CI recipients. The CI improved masked sentence recognition when the masker was presented from the front or from the side of normal or near-normal hearing. The binaural benefits observed with the CI increased between the 1- and 3-month intervals but appeared stable thereafter. In contrast to previous reports on localization and speech perception in patients with unilateral sensorineural hearing loss, CI benefits were consistently observed across individual subjects, and performance was at asymptote by the 3-month test interval. Cochlear implant settings, consistent CI use, and short duration of deafness could play a role in this result