Progressive Perceptual Audio Rendering of Complex Scenes

International audienceDespite recent advances, including sound source clustering and perceptual auditory masking, high quality rendering of complex virtual scenes with thousands of sound sources remains a challenge. Two major bottlenecks appear as the scene complexity increases: the cost of clustering itself, and the cost of pre-mixing source signals within each cluster. In this paper, we first propose an improved hierarchical clustering algorithm that remains efficient for large numbers of sources and clusters while providing progressive refinement capabilities. We then present a lossy pre-mixing method based on a progressive representation of the input audio signals and the perceptual importance of each sound source. Our quality evaluation user tests indicate that the recently introduced audio saliency map is inappropriate for this task. Consequently we propose a "pinnacle", loudness-based metric, which gives the best results for a variety of target computing budgets. We also performed a perceptual pilot study which indicates that in audio-visual environments, it is better to allocate more clusters to visible sound sources. We propose a new clustering metric using this result. As a result of these three solutions, our system can provide high quality rendering of thousands of 3D-sound sources on a "gamer-style" PC

Cross-benefits between virtual reality and games

electronic version (8 pp.)International audienceIn one hand, video games are dedicated to entertainment. In recent years, the emerging of consumers hardware dedicated to games induced great progress for realism and gameplay. Graphics rendering and physical engines, digital surround sound and new interaction interfaces are examples of areas which have benefited of these last improvements and widely contribute to the gaming experience. In another hand, virtual reality focus on user's presence which is its indubitable feeling of belonging to the virtual environment. As this goal is very hard to reach, studies have to focus on human through several research directions like immersion (3D vision, sound spatialization, haptic devices) and interaction which has to be as natural and non intrusive as possible. Recent researches on intersensoriality possibilities, metaphorical interactions or brain computer interfaces are examples of what would be achieved in immersion and interaction. At this point, we can argue that virtual reality can be a provider of new methods and resources for games. Unfortunately virtual reality room are expensive and difficult to deploy, what is probably the main reasons why virtual reality is still a laboratory experiment or confined to industrial simulator. Here is our double contribution : to combine video games and virtual reality through two different virtual reality game solutions and to design them with consumer grade components. This paper first presents a survey of both current video game evolutions and virtual reality researches. We will also give some examples of cross-benefits between video games and virtual reality. To illustrate this last point we will describe two virtual reality applications created by our research team and dedicated to gaming. Finally, as a prospective talk we will deal with three points : some recent virtual reality systems supposedly applicable to home gaming, some good points from DG that VR developers should incorporate in VR systems and last point, some lines of enquiry so that the union between VR and DG be at last consummate

Auditory-visual virtual environment for the treatment of fear of crowds

ABSTRACT Fear of crowds is a symptom found in agoraphobia (among others) and which has both visual and auditory relevant components. In order to examine the potential interest of auditory-visual virtual reality to treat fear of crowds, we investigated the efficiency of an auditory-visual virtual environment in evoking affective reaction. We conducted an evaluation test with healthy participants sensitive to the fear of crowds to explore the influence of auditory-visual virtual environment on affective reactions. Our application involves both high fidelity visual stimulation and interactive 3D sound. Our specific presentation of animated crowds of humans creates an environment that is highly arousing, suggesting that manipulating the sensory presentation together with the spatial location of stimulation might provide a way to modulate affective reactions

Réalité virtuelle : Simulation sonore en trois dimensions par Nicolas Tsingos. Phobies à l'épreuve du virtuel, entretien avec Isabelle Viaud-Delmon ; propos recueillis par Dominique Chouchan. Le jeu vidéo est un produit de très haute technologie par David Alloza

National audienceEn synthèse d'images, on s'intéresse à la simulation des formes, des couleurs, des ombres, de l'éclairage... La sonorisation d'un monde synthétique est tout aussi exigeante si l'on veut qu'elle reflète notre perception spatiale des sons

Audio-visual-olfactory resource allocation for tri-modal virtual environments

© 2019 IEEE. Virtual Environments (VEs) provide the opportunity to simulate a wide range of applications, from training to entertainment, in a safe and controlled manner. For applications which require realistic representations of real world environments, the VEs need to provide multiple, physically accurate sensory stimuli. However, simulating all the senses that comprise the human sensory system (HSS) is a task that requires significant computational resources. Since it is intractable to deliver all senses at the highest quality, we propose a resource distribution scheme in order to achieve an optimal perceptual experience within the given computational budgets. This paper investigates resource balancing for multi-modal scenarios composed of aural, visual and olfactory stimuli. Three experimental studies were conducted. The first experiment identified perceptual boundaries for olfactory computation. In the second experiment, participants (N=25) were asked, across a fixed number of budgets (M=5), to identify what they perceived to be the best visual, acoustic and olfactory stimulus quality for a given computational budget. Results demonstrate that participants tend to prioritize visual quality compared to other sensory stimuli. However, as the budget size is increased, users prefer a balanced distribution of resources with an increased preference for having smell impulses in the VE. Based on the collected data, a quality prediction model is proposed and its accuracy is validated against previously unused budgets and an untested scenario in a third and final experiment

Auditory-Visual Aversive Stimuli Modulate the Conscious Experience of Fear

International audiencevision. However, the impact of multisensory information on affect remains relatively undiscovered. In this study, we investigated whether the auditory-visual presentation of aversive stimuli influences the experience of fear. We used the advantages of virtual reality to manipulate multisensory presentation and to display potentially fearful dog stimuli embedded in a natural context. We manipulated the affective reactions evoked by the dog stimuli by recruiting two groups of participants: dog-fearful and non-fearful participants. The sensitivity to dog fear was assessed psychometrically by a questionnaire and also at behavioral and subjective levels using a Behavioral Avoidance Test (BAT). Participants navigated in virtual environments, in which they encountered virtual dog stimuli presented through the auditory channel, the visual channel or both. They were asked to report their fear using Subjective Units of Distress. We compared the fear for unimodal (visual or auditory) and bimodal (auditory- visual) dog stimuli. Dog-fearful participants as well as non-fearful participants reported more fear in response to bimodal audiovisual compared to unimodal presentation of dog stimuli. These results suggest that fear is more intense when the affective information is processed via multiple sensory pathways, which might be due to a cross-modal potentiation. Our findings have implications for the field of virtual reality-based therapy of phobias. Therapies could be refined and improved by implicating and manipulating the multisensory presentation of the feared situations

Interactive Sound Propagation for Massive Multi-user and Dynamic Virtual Environments

Hearing is an important sense and it is known that rendering sound effects can enhance the level of immersion in virtual environments. Modeling sound waves is a complex problem, requiring vast computing resources to solve accurately. Prior methods are restricted to static scenes or limited acoustic effects. In this thesis, we present methods to improve the quality and performance of interactive geometric sound propagation in dynamic scenes and precomputation algorithms for acoustic propagation in enormous multi-user virtual environments. We present a method for finding edge diffraction propagation paths on arbitrary 3D scenes for dynamic sources and receivers. Using this algorithm, we present a unified framework for interactive simulation of specular reflections, diffuse reflections, diffraction scattering, and reverberation effects. We also define a guidance algorithm for ray tracing that responds to dynamic environments and reorders queries to minimize simulation time. Our approach works well on modern GPUs and can achieve more than an order of magnitude performance improvement over prior methods. Modern multi-user virtual environments support many types of client devices, and current phones and mobile devices may lack the resources to run acoustic simulations. To provide such devices the benefits of sound simulation, we have developed a precomputation algorithm that efficiently computes and stores acoustic data on a server in the cloud. Using novel algorithms, the server can render enhanced spatial audio in scenes spanning several square kilometers for hundreds of clients in realtime. Our method provides the benefits of immersive audio to collaborative telephony, video games, and multi-user virtual environments.Doctor of Philosoph

Sistema de simulación acústica virtual en tiempo real

Tesis Doctoral (DCI)--FCEFN-UNC, 2012Esta tesis se pudo realizar gracias al programa de becas doctorales internas del Consejo Nacional de Investigaciones Científicas y Tecnológicas (CONICET).Desarrolla un sistema experimental dinámico e interactivo de realidad acústica virtual, basado en arquitectura de computadoras de propósito general y que fuera flexible, escalable, fácil de mantener y modular. Para alcanzar este objetivo se modelaron tres subsistemas: la fuente sonora, el recinto y el medio de propagación, y el oyent