Towards a Synesthesia Laboratory: Real-time Localization and Visualization of a Sound Source for Virtual Reality Applications

In this paper, we present our findings related to the problem of localization and visualization of a sound source placed in the same room as the listener. The particular effect that we aim to investigate is called synesthesia—the act of experiencing one sense modality as another, e.g., a person may vividly experience flashes of colors when listening to a series of sounds. Towards that end, we apply a series of recently developed methods for detecting sound source in a three-dimensional space around the listener.We also apply a Kalman filter to smooth out the perceived motion. Further, we transform the audio signal into a series of visual shapes, such that the size of each shape is determined by theloudness of the sound source, and its color is determined by the dominant spectral component of the sound. The developed prototype is verified in real time. The prototype configuration is described and some initial experimental results are reported and discussed. Some ideas for further development are also presented

Towards a Synesthesia Laboratory: Real-time Localization and Visualization of a Sound Source for Virtual Reality Applications

In this paper, we present our findings related to the problem of localization and visualization of a sound source placed in the same room as the listener. The particular effect that we aim to investigate is called synesthesia—the act of experiencing one sense modality as another, e.g., a person may vividly experience flashes of colors when listening to a series of sounds. Towards that end, we apply a series of recently developed methods for detecting sound source in a three-dimensional space around the listener.We also apply a Kalman filter to smooth out the perceived motion. Further, we transform the audio signal into a series of visual shapes, such that the size of each shape is determined by theloudness of the sound source, and its color is determined by the dominant spectral component of the sound. The developed prototype is verified in real time. The prototype configuration is described and some initial experimental results are reported and discussed. Some ideas for further development are also presented

Integrating Olfaction in a Robotic Telepresence Loop

In this work we propose enhancing a typical robotic telepresence architecture by considering olfactory and wind flow information in addition to the common audio and video channels. The objective is to expand the range of applications where robotics telepresence can be applied, including those related to the detection of volatile chemical substances (e.g. land-mine detection, explosive deactivation, operations in noxious environments, etc.). Concretely, we analyze how the sense of smell can be integrated in the telepresence loop, covering the digitization of the gases and wind flow present in the remote environment, the transmission through the communication network, and their display at the user location. Experiments under different environmental conditions are presented to validate the proposed telepresence system when localizing a gas emission leak at the remote environment.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tech

Virtual acoustics displays

The real time acoustic display capabilities are described which were developed for the Virtual Environment Workstation (VIEW) Project at NASA-Ames. The acoustic display is capable of generating localized acoustic cues in real time over headphones. An auditory symbology, a related collection of representational auditory 'objects' or 'icons', can be designed using ACE (Auditory Cue Editor), which links both discrete and continuously varying acoustic parameters with information or events in the display. During a given display scenario, the symbology can be dynamically coordinated in real time with 3-D visual objects, speech, and gestural displays. The types of displays feasible with the system range from simple warnings and alarms to the acoustic representation of multidimensional data or events

3D sound for simulation of arthroscopic surgery

Arthroscopic surgery offers many advantages compared to traditional surgery. Nevertheless, the required skills to practice this kind of surgery need specific training. Surgery simulators are used to train surgeon apprentices to practice specific gestures. In this paper, we present a study showing the contribution of 3D sound in assisting the triangulation gesture in arthroscopic surgery simulation. This ability refers to the capacity of the subject to manipulate the instruments while having a modified and limited view provided by the video camera of the simulator. Our approach, based on the use of 3D sound metaphors, provides interaction cues to the subjects about the real position of the instrument. The paper reports a performance evaluation study based on the perception of 3D sound integrated in the process of training of surgical task. Despite the fact that 3D sound cueing was not shown useful to all subjects in terms of execution time, the results of the study revealed that the majority of subjects who participated to the experiment confirmed the added value of 3D sound in terms of ease of use

Effects of virtual acoustics on dynamic auditory distance perception

Sound propagation encompasses various acoustic phenomena including reverberation. Current virtual acoustic methods, ranging from parametric filters to physically-accurate solvers, can simulate reverberation with varying degrees of fidelity. We investigate the effects of reverberant sounds generated using different propagation algorithms on acoustic distance perception, i.e., how faraway humans perceive a sound source. In particular, we evaluate two classes of methods for real-time sound propagation in dynamic scenes based on parametric filters and ray tracing. Our study shows that the more accurate method shows less distance compression as compared to the approximate, filter-based method. This suggests that accurate reverberation in VR results in a better reproduction of acoustic distances. We also quantify the levels of distance compression introduced by different propagation methods in a virtual environment.Comment: 8 Pages, 7 figure

A framework for realistic 3D tele-immersion

Meeting, socializing and conversing online with a group of people using teleconferencing systems is still quite differ- ent from the experience of meeting face to face. We are abruptly aware that we are online and that the people we are engaging with are not in close proximity. Analogous to how talking on the telephone does not replicate the experi- ence of talking in person. Several causes for these differences have been identified and we propose inspiring and innova- tive solutions to these hurdles in attempt to provide a more realistic, believable and engaging online conversational expe- rience. We present the distributed and scalable framework REVERIE that provides a balanced mix of these solutions. Applications build on top of the REVERIE framework will be able to provide interactive, immersive, photo-realistic ex- periences to a multitude of users that for them will feel much more similar to having face to face meetings than the expe- rience offered by conventional teleconferencing systems