Measuring the Behavioral Response to Spatial Audio within a Multi-Modal Virtual Reality Environment in Children with Autism Spectrum Disorder

Virtual Reality (VR) has been an active area of research in the development of interactive interventions for individuals with autism spectrum disorder (ASD) for over two decades. These immersive environments create a safe platform in which therapy can address the core symptoms associated with this condition. Recent advancements in spatial audio rendering techniques for VR now allow for the creation of realistic audio environments that accurately match their visual counterparts. However, reported auditory processing impairments associated with autism may affect how an individual interacts with their virtual therapy application. This study aims to investigate if these difficulties in processing audio information would directly impact how individuals with autism interact with a presented virtual spatial audio environment. Two experiments were conducted with participants diagnosed with ASD (n = 29) that compared: (1) behavioral reaction between spatialized and non-spatialized audio; and (2) the effect of background noise on participant interaction. Participants listening to binaural-based spatial audio showed higher spatial attention towards target auditory events. In addition, the amount of competing background audio was reported to influence spatial attention and interaction. These findings suggest that despite associated sensory processing difficulties, those with ASD can correctly decode the auditory cues simulated in current spatial audio rendering techniques

Life-Sized Audiovisual Spatial Social Scenes with Multiple Characters: MARC & SMART-I²

International audienceWith the increasing use of virtual characters in virtual and mixed reality settings, the coordination of realism in audiovisual rendering and expressive virtual characters becomes a key issue. In this paper we introduce a new system combining two systems for tackling the issue of realism and high quality in audiovisual rendering and life-sized expressive characters. The goal of the resulting SMART-MARC platform is to investigate the impact of realism on multiple levels: spatial audiovisual rendering of a scene, appearance and expressive behaviors of virtual characters. Potential interactive applications include mediated communication in virtual worlds, therapy, game, arts and elearning. Future experimental studies will focus on 3D audio/visual coherence, social perception and ecologically valid interaction scenes

The Plausibility of a String Quartet Performance in Virtual Reality

We describe an experiment that explores the contribution of auditory and other features to the illusion of plausibility in a virtual environment that depicts the performance of a string quartet. ‘Plausibility’ refers to the component of presence that is the illusion that the perceived events in the virtual environment are really happening. The features studied were: Gaze (the musicians ignored the participant, the musicians sometimes looked towards and followed the participant’s movements), Sound Spatialization (Mono, Stereo, Spatial), Auralization (no sound reflections, reflections corresponding to a room larger than the one perceived, reflections that exactly matched the virtual room), and Environment (no sound from outside of the room, birdsong and wind corresponding to the outside scene). We adopted the methodology based on color matching theory, where 20 participants were first able to assess their feeling of plausibility in the environment with each of the four features at their highest setting. Then five times participants started from a low setting on all features and were able to make transitions from one system configuration to another until they matched their original feeling of plausibility. From these transitions a Markov transition matrix was constructed, and also probabilities of a match conditional on feature configuration. The results show that Environment and Gaze were individually the most important factors influencing the level of plausibility. The highest probability transitions were to improve Environment and Gaze, and then Auralization and Spatialization. We present this work as both a contribution to the methodology of assessing presence without questionnaires, and showing how various aspects of a musical performance can influence plausibility

Audio, visual, and audio-visual egocentric distance perception by moving participants in virtual environments

International audienceA study on audio, visual, and audio-visual egocentric distance perception by moving participants in virtual environments is presented. Audio-visual rendering is provided using tracked passive visual stereoscopy and acoustic wave fi eld synthesis (WFS). Distances are estimated using indirect blind-walking (triangulation) under each rendering condition. Experimental results show that distances perceived in the virtual environment are accurately estimated or overestimated for rendered distances closer than the position of the audio-visual rendering system and underestimated for distances farther. Interestingly, participants perceived each virtual object at a modality-independent distance when using the audio modality, the visual modality, or the combination of both. Results show WFS capable of synthesizing perceptually meaningful sound fields in terms of distance. Dynamic audio-visual cues were used by participants when estimating the distances in the virtual world. Moving may have provided participants with a better visual distance perception of close distances than if they were static. No correlation between the feeling of presence and the visual distance underestimation has been found. To explain the observed perceptual distance compression, it is proposed that, due to con flicting distance cues, the audio-visual rendering system physically anchors the virtual world to the real world. Virtual objects are thus attracted by the physical audio-visual rendering system

Virtual Heritage: Audio design for immersive virtual environments using researched spatializers.

This thesis work is based on a Virtual Heritage project being developed by the Systems of Representation research group. The objective of the project is to create a showcase demonstration on how the virtual reality (VR) could be used as an application for tourism in the heritage sector. In this context, my task was to develop a concept and prototype of how 'spatialized' sound could be used in a VR application. The initial location chosen for the concept was the ancient heritage burial site of Sammallahdenmäki, one of the Finnish heritage sites listed in the UNESCO register of World Heritage Sites. The thesis, that is written from an audio designer's perspective, focuses on three aspects of this project. First is the sound design for the Virtual Heritage project and the second is the quality of currently available 'spatializer' plug-ins used for headphone listening. In order to evaluate the process of designing 3D audio for virtual environments, the methods and principles within binaural rendering, sound design and immersion must be understood. Therefore, functions and theories within audio spatialization and 3D audio design are reviewed. Audio designers working on virtual reality content need the best possible solutions for creating believable 3D audio experiences. However, while working on the Virtual Heritage project, we did not find any comparative studies made about commercially available spatializer plug-ins for Unity. Thus, it was unknown what plug-in would have been the best possible solution for 3D audio spatialization. Consequently, two tests were conducted during this thesis work. First was an online test measuring which spatializer would be the most highly rated, in terms of perceived directional precision when utilizing head-related transfer functions without reverb or room simulations. The second was a comparative test studying if a spatialized audio rendering would increase immersion compared to non-spatialized audio rendering, when tested with the Virtual Heritage demonstration. The central aim in the showcase demonstration was to create an immersive virtual environment where users would feel as if they were travelling from the present, back to the Bronze Age, in order to understand and learn about the location’s unique history via auditory storytelling. The project was implemented utilising the Unity game engine. The research on music and other sound content used in the project’s sonic environment is explained. Finally, results of the project work are discussed.Tämä opinnäytetyö perustuu Virtual Heritage projektityöhön, joka on tehty Systems of Representation tutkimusryhmälle. Projektin tavoite on luoda malliesimerkki siitä, miten virtuaalitodellisuutta voitaisiin käyttää hyväksi turismisovelluksissa. Esimerkkikohteeksi projektille oli valittu Sammallahdenmäen hautaröykkiöt, joka on hyväksytty mukaan UNESCON maailmanperintöluetteloon. Tehtäväni oli toteuttaa Unity pelimoottorilla prototyyppi, jossa kartoitetaan virtuaalisen tilaäänen käyttömahdollisuuksia kyseisen teeman ympärillä. Opinnäyte on kirjoitettu äänisuunnittelijan näkökulmasta keskittyen kolmeen projektityöhön liittyvään keskeiseen osaan: prototyypin äänisuunnitteluun, immersion käsitteeseen sekä spatialisointi liitännäisten (plug-in) toimintaan ja laatuun. Virtuaalitodellisuuksiin sisältöä tuottavana äänisuunnittelijana tarvitsin parhaat mahdolliset työkalut uskottavan 3D äänimaailman luomiseen. Virtual Heritage projektia työstettäessä kävi kuitenkin ilmi, että ajankohtaista vertailevaa tutkimusta spatialisointi liitännäisten laatueroista ei ollut löydettävissä. Oli mahdotonta määritellä yksilöllisesti, mikä liitännäisistä on toimivin ratkaisu suurimmalle käyttäjäkunnalle. Täten oli tarpeellista toteuttaa kaksi tutkimusta. Ensimmäinen oli empiirinen verkkotutkimus, jolla arvioitiin spatialisointi liitännäisten suorituskykyä, kun mitataan subjektiivisesti äänen tilallisen sijoittumisen tarkkuutta kuulokekuuntelussa ilman kaikuprosessointeja. Tutkimuksessa parhaiten suoriutunut liitännäinen implementoitiin prototyyppiin. Toisessa kokeessa tutkittiin kuinka paljon implementoidun spatialisointi liitännäisen käyttäminen lisää virtuaalitodellisuuden immersiota verrattaessa spatialisoimattomaan ääneen, kun testialustana toimii Virtual Heritage prototyyppi. Projektin keskeisin tavoite oli luoda immersiivinen virtuaalitodellisuus, jossa käyttäjä voi kokea matkaavansa nykyajasta pronssikautiselle Sammallahdenmäelle ja oppia tällä tavoin kohteen ainutlaatuisesta historiasta äänikerronnan keinoin. Opinnäytetyössä esitellään äänikerronnan sisältöön ja toteutustapaan johtavat tutkimukset, tuotanto sekä ajatukset lopputuloksesta

The SMART-I²: A new approach for the design of immersive audio-visual environments

International audienceThe SMART-I² aims at creating a precise and coherent virtual environment by providing users with both audio and visual accurate localization cues. Wave field synthesis, for audio rendering, and tracked passive stereoscopy, for visual rendering, individually permit high quality spatial immersion within an extended space. The proposed system combines these two rendering approaches through the use of large multi-actuator panels used as both loudspeaker arrays and as projection screens, providing a more perceptually consistent rendering

Sonic interaction with a virtual orchestra of factory machinery

This paper presents an immersive application where users receive sound and visual feedbacks on their interactions with a virtual environment. In this application, the users play the part of conductors of an orchestra of factory machines since each of their actions on interaction devices triggers a pair of visual and audio responses. Audio stimuli were spatialized around the listener. The application was exhibited during the 2013 Science and Music day and designed to be used in a large immersive system with head tracking, shutter glasses and a 10.2 loudspeaker configuration.Comment: Sonic Interaction for Virtual Environments, Minneapolis : United States (2014

Experimental Analysis of Spatial Sound for Storytelling in Virtual Reality

Spatial sound is useful in enhancing immersion and presence of the user in a virtual world. This audio design allows the game designer to place audio cues that appropriately match with the visual cues in a virtual game environment. These localized audio cues placed in a story based game environment also help to evoke an emotional response from the user and construct the narrative of the game by capturing the user’s attention towards the guiding action events in the game. Our thesis explores the usefulness of spatial sound for improving the performance and experience of a user in a virtual game environment. Additionally, with the help of the relevant subjective and objective inferences collected from a user study conducted on three different evaluation models, the thesis also analyzes and establishes the potential of spatial sound as a powerful storytelling tool in a virtual game environment designed for Virtual Reality

3D sound for simulation of arthroscopic surgery

Arthroscopic surgery offers many advantages compared to traditional surgery. Nevertheless, the required skills to practice this kind of surgery need specific training. Surgery simulators are used to train surgeon apprentices to practice specific gestures. In this paper, we present a study showing the contribution of 3D sound in assisting the triangulation gesture in arthroscopic surgery simulation. This ability refers to the capacity of the subject to manipulate the instruments while having a modified and limited view provided by the video camera of the simulator. Our approach, based on the use of 3D sound metaphors, provides interaction cues to the subjects about the real position of the instrument. The paper reports a performance evaluation study based on the perception of 3D sound integrated in the process of training of surgical task. Despite the fact that 3D sound cueing was not shown useful to all subjects in terms of execution time, the results of the study revealed that the majority of subjects who participated to the experiment confirmed the added value of 3D sound in terms of ease of use