Comparison of Binaural RTF-Vector-Based Direction of Arrival Estimation Methods Exploiting an External Microphone

In this paper we consider a binaural hearing aid setup, where in addition to the head-mounted microphones an external microphone is available. For this setup, we investigate the performance of several relative transfer function (RTF) vector estimation methods to estimate the direction of arrival(DOA) of the target speaker in a noisy and reverberant acoustic environment. More in particular, we consider the state-of-the-art covariance whitening (CW) and covariance subtraction (CS) methods, either incorporating the external microphone or not, and the recently proposed spatial coherence (SC) method, requiring the external microphone. To estimate the DOA from the estimated RTF vector, we propose to minimize the frequency-averaged Hermitian angle between the estimated head-mounted RTF vector and a database of prototype head-mounted RTF vectors. Experimental results with stationary and moving speech sources in a reverberant environment with diffuse-like noise show that the SC method outperforms the CS method and yields a similar DOA estimation accuracy as the CW method at a lower computational complexity.Comment: Submitted to EUSIPCO 202

Shaping the auditory peripersonal space with motor planning in immersive virtual reality

Immersive audio technologies require personalized binaural synthesis through headphones to provide perceptually plausible virtual and augmented reality (VR/AR) simulations. We introduce and apply for the first time in VR contexts the quantitative measure called premotor reaction time (pmRT) for characterizing sonic interactions between humans and the technology through motor planning. In the proposed basic virtual acoustic scenario, listeners are asked to react to a virtual sound approaching from different directions and stopping at different distances within their peripersonal space (PPS). PPS is highly sensitive to embodied and environmentally situated interactions, anticipating the motor system activation for a prompt preparation for action. Since immersive VR applications benefit from spatial interactions, modeling the PPS around the listeners is crucial to reveal individual behaviors and performances. Our methodology centered around the pmRT is able to provide a compact description and approximation of the spatiotemporal PPS processing and boundaries around the head by replicating several well-known neurophysiological phenomena related to PPS, such as auditory asymmetry, front/back calibration and confusion, and ellipsoidal action fields

Quadri-stability of a spatially ambiguous auditory illusion

In addition to vision, audition plays an important role in sound localization in our world. One way we estimate the motion of an auditory object moving towards or away from us is from changes in volume intensity. However, the human auditory system has unequally distributed spatial resolution, including difficulty distinguishing sounds in front vs. behind the listener. Here, we introduce a novel quadri-stable illusion, the Transverse-and-Bounce Auditory Illusion, which combines front-back confusion with changes in volume levels of a nonspatial sound to create ambiguous percepts of an object approaching and withdrawing from the listener. The sound can be perceived as traveling transversely from front to back or back to front, or “bouncing” to remain exclusively in front of or behind the observer. Here we demonstrate how human listeners experience this illusory phenomenon by comparing ambiguous and unambiguous stimuli for each of the four possible motion percepts. When asked to rate their confidence in perceiving each sound’s motion, participants reported equal confidence for the illusory and unambiguous stimuli. Participants perceived all four illusory motion percepts, and could not distinguish the illusion from the unambiguous stimuli. These results show that this illusion is effectively quadri-stable. In a second experiment, the illusory stimulus was looped continuously in headphones while participants identified its perceived path of motion to test properties of perceptual switching, locking, and biases. Participants were biased towards perceiving transverse compared to bouncing paths, and they became perceptually locked into alternating between front-to-back and back-to-front percepts, perhaps reflecting how auditory objects commonly move in the real world. This multi-stable auditory illusion opens opportunities for studying the perceptual, cognitive, and neural representation of objects in motion, as well as exploring multimodal perceptual awareness.United States. Dept. of Defense (National Defense Science and Engineering Graduate (NDSEG) Fellowships

Effects of amplitude modulation on sound localization in reverberant environments.

Auditory localization involves different cues depending on the spatial domain. Azimuth localization cues include interaural time differences (ITDs), interaural level differences (ILDs) and pinnae cues. Auditory distance perception (ADP) cues include intensity, spectral cues, binaural cues, and the direct-to-reverberant energy ratio (D/R). While D/R has been established as a primary ADP cue, it is unlikely that it is directly encoded in the auditory system because it can be difficult to extract from ongoing signals. It is also noteworthy that no neuronal population has been identified that specifically codes D/R. It has therefore been proposed that D/R is indirectly encoded in the auditory system, through sensitivity to other acoustic parameters that are correlated with D/R, such as temporal cues (Zahorik, 2002b), spectral properties (Jetzt, 1979; Larsen, 2008), and interaural correlation (Bronkhorst and Houtgast, 1999). An additional D/R correlate relies on attenuation of amplitude modulation (AM) as a function of distance. Room modulation transfer functions act as low-pass filters on AM signals, and therefore the direct portion of a signal will have less modulation depth attenuation than the reverberant portion. Although recent neural and behavioral work has demonstrated that this cue can provide distance information monaurally, the extent to which the modulation attenuation cue contributes to ADP relative to other ADP cues is not fully understood. It is also possible modulation attenuation by the room can provide additional directional localization information, perhaps through the resulting dynamic fluctuation of the ILD cue. The role of AM in directional sound localization has not been extensively studied, particularly in reverberant soundfields which can affect the modulation reaching the two ears in a directionally-dependent fashion. Three human psychophysical experiments assessed the role of AM signals in distance and directional auditory localization in reverberant soundfields. Experiment I focused on validating a graphical response method to be used in subsequent experiments. In Experiment II, an auditory distance estimation task was performed which yielded measures of the relative perceptual contributions of the modulation depth cue during ADP in a reverberant room. Experiment III investigated the effect of AM on binaural localization in the horizontal plane in a reverberant room

An Electroencephalographic Investigation of the Encoding of Sound Source Elevation in the Human Cortex

Sound localization is of great ecological importance because it provides spa- tial perception outside the visual field. However, unlike other sensory systems, the auditory system does not represent the location of a stimulus on the level of the sensory epithelium in the cochlea. Instead, the position of a sound source has to be computed based on different localization cues. Different cues are informative of a sound sources azimuth and elevation, which, when taken together, describe the sources location in a polar coordinate system. There is a body of knowledge regarding the acoustical cues and the neural circuits in the brainstem required to perceive sound source azimuth and elevation. However, our understanding of the encoding of sound source location on the level of the cortex is lacking especially what concerns elevation. Within the scope of this thesis, we established an experimental setup to study auditory spatial perception while recording the listeners brain activity using electroencephalography. We conducted two experiments on the encoding of sound source elevation in the human cortex. Both experiments results are compatible with the hypothesis that the cortex represents sound source elevation in a population rate code where the response amplitude decreases linearly with increasing elevation. Decoding of the recorded brain activity revealed that a distinct neural representation of differently elevated sound sources was predictive of behavioral performance. An exploratory analysis indicated an increase in the amplitude of oscillations in visual areas when the subject localized sounds during eccentric eye positions. More research in this direction could help shed light on the interactions between the visual and auditory systems regarding spatial perception. The experiments presented in this dissertation are, to our knowledge, the first studies that demonstrate the encoding of sound source elevation in the human cortex by using a direct measure of neural activity (i.e., electroencephalography).:Abstract . . . . . . . . . . . . . . . . . . . . . . 1 Zusammenfassung . . . . . . . . . . . . . . . . . . . . . . 7 1 Electroencephalography 13 1.1 Event Related Potentials and Oscillations . . . . . . . . . . . . 13 1.2 Comparison to other Methods . . . . . . . . . . . . . . . . . . . 14 1.3 EEG Apparatus . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.4 Preprocessing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.4.1 Filtering . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.4.2 Referencing . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.4.3 Eye Blinks . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.4.4 Epoch Rejection . . . . . . . . . . . . . . . . . . . . . . . 22 1.4.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.5.1 Decoding . . . . . . . . . . . . . . . . . . . . . . . . . . 25 1.5.2 Nonparametric Permutation Testing . . . . . . . . . . . 26 1.5.3 Source Separation . . . . . . . . . . . . . . . . . . . . . . 28 2 Sound Localization in the Brain . . . . . . . . . . . . . . . . . . . . 31 2.1 The Spatial Perception of Sound . . . . . . . . . . . . . . . . . . 32 2.1.1 Interaural Cues . . . . . . . . . . . . . . . . . . . . . . . 32 2.1.2 Spectral Cues . . . . . . . . . . . . . . . . . . . . . . . . 33 2.2 Brain Mechanisms for Sound Localization . . . . . . . . . . . . 37 2.2.1 Auditory Pathway . . . . . . . . . . . . . . . . . . . . . 38 2.2.2 Extracting Localization Cues . . . . . . . . . . . . . . . 40 2.2.3 Neural Representation of Auditory Space . . . . . . . . 42 2.2.4 The Dual Pathway Model . . . . . . . . . . . . . . . . . 45 2.2.5 A Dominant Hemisphere for Sound Localization? . . . 47 2.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3 A Free Field Setup for Psychoacoustics 51 3.1 Design of the Experimental Setup . . . . . . . . . . . . . . . . . 51 3.1.1 Loudspeakers . . . . . . . . . . . . . . . . . . . . . . . . 54 3.1.2 Processors . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.1.3 Cameras . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.1.4 Coordinate Systems . . . . . . . . . . . . . . . . . . . . 56 3.2 Operating the Setup . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.2.1 Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.2.2 Loudspeaker Equalization . . . . . . . . . . . . . . . . . 59 3.3 Head Pose Estimation . . . . . . . . . . . . . . . . . . . . . . . 61 3.3.1 Landmark Detection . . . . . . . . . . . . . . . . . . . . 62 3.3.2 Perspective-n-Point Problem . . . . . . . . . . . . . . . 62 3.3.3 Camera-to-World Conversion . . . . . . . . . . . . . . . 63 3.4 A Toolbox for Psychoacoustics . . . . . . . . . . . . . . . . . . 64 4 A Linear Population Rate Code for Elevation 67 4.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 68 4.1.2 Experimental Protocol . . . . . . . . . . . . . . . . . . . 69 4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.2.1 Behavioral Performance . . . . . . . . . . . . . . . . . . 70 4.2.2 ERP Components . . . . . . . . . . . . . . . . . . . . . . 70 4.2.3 Elevation Encoding . . . . . . . . . . . . . . . . . . . . . 72 4.2.4 Effect of Eye-Position . . . . . . . . . . . . . . . . . . . . 74 4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 5 Decoding of Brain Responses Predicts Localization Accuracy . . . 81 5.1 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.1.1 Participants . . . . . . . . . . . . . . . . . . . . . . . . . 82 5.1.2 Experimental Protocol . . . . . . . . . . . . . . . . . . . 82 5.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.2.1 Behavioral Performance . . . . . . . . . . . . . . . . . . 83 5.2.2 ERP Components . . . . . . . . . . . . . . . . . . . . . . 84 5.2.3 Decoding Brain Activity . . . . . . . . . . . . . . . . . . 86 5.2.4 Topography of Elevation Encoding . . . . . . . . . . . . 88 5.2.5 Elevation Tuning . . . . . . . . . . . . . . . . . . . . . . 89 5.2.6 Hemispheric Lateralization . . . . . . . . . . . . . . . . 91 5.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 A Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 B Publication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Seeing with ears: how we create an auditory representation of space with echoes and its relation with other senses

Spatial perception is the capability that allows us to learn about the environment. All our senses are involved in creating a representation of the external world. When we create the representation of space we rely primarily on visual information, but it is the integration with the other senses that allows us a more global and truthful representation of it. While the influence of vision and the integration of different senses among each other in spatial perception has been widely investigated, many questions remain about the role of the acoustic system in space perception and how it can be influenced by the other senses. Give an answer to these questions on healthy people can help to better understand whether the same \u201crules\u201d can be applied to, for example, people that have lost vision in the early stages of development. Understanding how spatial perception works in blind people from birth is essential to then develop rehabilitative methodologies or technologies to help these people to provide for lack of vision, since vision is the main source of spatial information. For this reason, one of the main scientific objective of this thesis is to increase knowledge about auditory spatial perception in sighted and visually impaired people, thanks to the development of new tasks to assess spatial abilities. Moreover, I focus my attention on a recent investigative topic in humans, i.e. echolocation. Echolocation has a great potential in terms of improvement regarding space and navigation skills for people with visual disabilities. Several studies demonstrate how the use of this technique can be favorable in the absence of vision, both on the level perceptual level and also at the social level. Based in the importance of echolocation, we developed some tasks to test the ability of novice people and we undergo the participants to an echolocation training to see how long does it take to manage this technique (in simple task). Instead of using blind individuals, we decide to test the ability of novice sighted people to see whether technique is blind related or not and whether it is possible to create a representation of space using echolocatio

Optimization and improvements in spatial sound reproduction systems through perceptual considerations

[ES] La reproducción de las propiedades espaciales del sonido es una cuestión cada vez más importante en muchas aplicaciones inmersivas emergentes. Ya sea en la reproducción de contenido audiovisual en entornos domésticos o en cines, en sistemas de videoconferencia inmersiva o en sistemas de realidad virtual o aumentada, el sonido espacial es crucial para una sensación de inmersión realista. La audición, más allá de la física del sonido, es un fenómeno perceptual influenciado por procesos cognitivos. El objetivo de esta tesis es contribuir con nuevos métodos y conocimiento a la optimización y simplificación de los sistemas de sonido espacial, desde un enfoque perceptual de la experiencia auditiva. Este trabajo trata en una primera parte algunos aspectos particulares relacionados con la reproducción espacial binaural del sonido, como son la escucha con auriculares y la personalización de la Función de Transferencia Relacionada con la Cabeza (Head Related Transfer Function - HRTF). Se ha realizado un estudio sobre la influencia de los auriculares en la percepción de la impresión espacial y la calidad, con especial atención a los efectos de la ecualización y la consiguiente distorsión no lineal. Con respecto a la individualización de la HRTF se presenta una implementación completa de un sistema de medida de HRTF y se introduce un nuevo método para la medida de HRTF en salas no anecoicas. Además, se han realizado dos experimentos diferentes y complementarios que han dado como resultado dos herramientas que pueden ser utilizadas en procesos de individualización de la HRTF, un modelo paramétrico del módulo de la HRTF y un ajuste por escalado de la Diferencia de Tiempo Interaural (Interaural Time Difference - ITD). En una segunda parte sobre reproducción con altavoces, se han evaluado distintas técnicas como la Síntesis de Campo de Ondas (Wave-Field Synthesis - WFS) o la panoramización por amplitud. Con experimentos perceptuales se han estudiado la capacidad de estos sistemas para producir sensación de distancia y la agudeza espacial con la que podemos percibir las fuentes sonoras si se dividen espectralmente y se reproducen en diferentes posiciones. Las aportaciones de esta investigación pretenden hacer más accesibles estas tecnologías al público en general, dada la demanda de experiencias y dispositivos audiovisuales que proporcionen mayor inmersión.[CA] La reproducció de les propietats espacials del so és una qüestió cada vegada més important en moltes aplicacions immersives emergents. Ja siga en la reproducció de contingut audiovisual en entorns domèstics o en cines, en sistemes de videoconferència immersius o en sistemes de realitat virtual o augmentada, el so espacial és crucial per a una sensació d'immersió realista. L'audició, més enllà de la física del so, és un fenomen perceptual influenciat per processos cognitius. L'objectiu d'aquesta tesi és contribuir a l'optimització i simplificació dels sistemes de so espacial amb nous mètodes i coneixement, des d'un criteri perceptual de l'experiència auditiva. Aquest treball tracta, en una primera part, alguns aspectes particulars relacionats amb la reproducció espacial binaural del so, com són l'audició amb auriculars i la personalització de la Funció de Transferència Relacionada amb el Cap (Head Related Transfer Function - HRTF). S'ha realitzat un estudi relacionat amb la influència dels auriculars en la percepció de la impressió espacial i la qualitat, dedicant especial atenció als efectes de l'equalització i la consegüent distorsió no lineal. Respecte a la individualització de la HRTF, es presenta una implementació completa d'un sistema de mesura de HRTF i s'inclou un nou mètode per a la mesura de HRTF en sales no anecoiques. A mès, s'han realitzat dos experiments diferents i complementaris que han donat com a resultat dues eines que poden ser utilitzades en processos d'individualització de la HRTF, un model paramètric del mòdul de la HRTF i un ajustament per escala de la Diferencià del Temps Interaural (Interaural Time Difference - ITD). En una segona part relacionada amb la reproducció amb altaveus, s'han avaluat distintes tècniques com la Síntesi de Camp d'Ones (Wave-Field Synthesis - WFS) o la panoramització per amplitud. Amb experiments perceptuals, s'ha estudiat la capacitat d'aquests sistemes per a produir una sensació de distància i l'agudesa espacial amb que podem percebre les fonts sonores, si es divideixen espectralment i es reprodueixen en diferents posicions. Les aportacions d'aquesta investigació volen fer més accessibles aquestes tecnologies al públic en general, degut a la demanda d'experiències i dispositius audiovisuals que proporcionen major immersió.[EN] The reproduction of the spatial properties of sound is an increasingly important concern in many emerging immersive applications. Whether it is the reproduction of audiovisual content in home environments or in cinemas, immersive video conferencing systems or virtual or augmented reality systems, spatial sound is crucial for a realistic sense of immersion. Hearing, beyond the physics of sound, is a perceptual phenomenon influenced by cognitive processes. The objective of this thesis is to contribute with new methods and knowledge to the optimization and simplification of spatial sound systems, from a perceptual approach to the hearing experience. This dissertation deals in a first part with some particular aspects related to the binaural spatial reproduction of sound, such as listening with headphones and the customization of the Head Related Transfer Function (HRTF). A study has been carried out on the influence of headphones on the perception of spatial impression and quality, with particular attention to the effects of equalization and subsequent non-linear distortion. With regard to the individualization of the HRTF a complete implementation of a HRTF measurement system is presented, and a new method for the measurement of HRTF in non-anechoic conditions is introduced. In addition, two different and complementary experiments have been carried out resulting in two tools that can be used in HRTF individualization processes, a parametric model of the HRTF magnitude and an Interaural Time Difference (ITD) scaling adjustment. In a second part concerning loudspeaker reproduction, different techniques such as Wave-Field Synthesis (WFS) or amplitude panning have been evaluated. With perceptual experiments it has been studied the capacity of these systems to produce a sensation of distance, and the spatial acuity with which we can perceive the sound sources if they are spectrally split and reproduced in different positions. The contributions of this research are intended to make these technologies more accessible to the general public, given the demand for audiovisual experiences and devices with increasing immersion.Gutiérrez Parera, P. (2020). Optimization and improvements in spatial sound reproduction systems through perceptual considerations [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/142696TESI

Sensory Communication

Contains table of contents for Section 2, an introduction and reports on fourteen research projects.National Institutes of Health Grant RO1 DC00117National Institutes of Health Grant RO1 DC02032National Institutes of Health/National Institute on Deafness and Other Communication Disorders Grant R01 DC00126National Institutes of Health Grant R01 DC00270National Institutes of Health Contract N01 DC52107U.S. Navy - Office of Naval Research/Naval Air Warfare Center Contract N61339-95-K-0014U.S. Navy - Office of Naval Research/Naval Air Warfare Center Contract N61339-96-K-0003U.S. Navy - Office of Naval Research Grant N00014-96-1-0379U.S. Air Force - Office of Scientific Research Grant F49620-95-1-0176U.S. Air Force - Office of Scientific Research Grant F49620-96-1-0202U.S. Navy - Office of Naval Research Subcontract 40167U.S. Navy - Office of Naval Research/Naval Air Warfare Center Contract N61339-96-K-0002National Institutes of Health Grant R01-NS33778U.S. Navy - Office of Naval Research Grant N00014-92-J-184