Efficient Techniques for Wave-based Sound Propagation in Interactive Applications

Sound propagation techniques model the effect of the environment on sound waves and predict their behavior from point of emission at the source to the final point of arrival at the listener. Sound is a pressure wave produced by mechanical vibration of a surface that propagates through a medium such as air or water, and the problem of sound propagation can be formulated mathematically as a second-order partial differential equation called the wave equation. Accurate techniques based on solving the wave equation, also called the wave-based techniques, are too expensive computationally and memory-wise. Therefore, these techniques face many challenges in terms of their applicability in interactive applications including sound propagation in large environments, time-varying source and listener directivity, and high simulation cost for mid-frequencies. In this dissertation, we propose a set of efficient wave-based sound propagation techniques that solve these three challenges and enable the use of wave-based sound propagation in interactive applications. Firstly, we propose a novel equivalent source technique for interactive wave-based sound propagation in large scenes spanning hundreds of meters. It is based on the equivalent source theory used for solving radiation and scattering problems in acoustics and electromagnetics. Instead of using a volumetric or surface-based approach, this technique takes an object-centric approach to sound propagation. The proposed equivalent source technique generates realistic acoustic effects and takes orders of magnitude less runtime memory compared to prior wave-based techniques. Secondly, we present an efficient framework for handling time-varying source and listener directivity for interactive wave-based sound propagation. The source directivity is represented as a linear combination of elementary spherical harmonic sources. This spherical harmonic-based representation of source directivity can support analytical, data-driven, rotating or time-varying directivity function at runtime. Unlike previous approaches, the listener directivity approach can be used to compute spatial audio (3D audio) for a moving, rotating listener at interactive rates. Lastly, we propose an efficient GPU-based time-domain solver for the wave equation that enables wave simulation up to the mid-frequency range in tens of minutes on a desktop computer. It is demonstrated that by carefully mapping all the components of the wave simulator to match the parallel processing capabilities of the graphics processors, significant improvement in performance can be achieved compared to the CPU-based simulators, while maintaining numerical accuracy. We validate these techniques with offline numerical simulations and measured data recorded in an outdoor scene. We present results of preliminary user evaluations conducted to study the impact of these techniques on user's immersion in virtual environment. We have integrated these techniques with the Half-Life 2 game engine, Oculus Rift head-mounted display, and Xbox game controller to enable users to experience high-quality acoustics effects and spatial audio in the virtual environment.Doctor of Philosoph

Acoustic pulse propagation in an urban environment using a three-dimensional numerical simulation

Acoustic pulse propagation in outdoor urban environments is a physically complex phenomenon due to the predominance of reflection, diffraction, and scattering. This is especially true in non-line-of-sight cases, where edge diffraction and high-order scattering are major components of acoustic energy transport. Past work by Albert and Liu [J. Acoust. Soc. Am. 127, 1335-1346 (2010)] has shown that many of these effects can be captured using a two-dimensional finite-difference time-domain method, which was compared to the measured data recorded in an army training village. In this paper, a full three-dimensional analysis of acoustic pulse propagation is presented. This analysis is enabled by the adaptive rectangular decomposition method by Raghuvanshi, Narain and Lin [IEEE Trans. Visual. Comput. Graphics 15, 789-801 (2009)], which models sound propagation in the same scene in three dimensions. The simulation is run at a much higher usable bandwidth (nearly 450 Hz) and took only a few minutes on a desktop computer. It is shown that a three-dimensional solution provides better agreement with measured data than two-dimensional modeling, especially in cases where propagation over rooftops is important. In general, the predicted acoustic responses match well with measured results for the source/sensor locations

Binaural Reproduction Based on Bilateral Ambisonics

Binaural reproduction of high-quality spatial sound has gained considerable interest with the recent technology developments in virtual and augmented reality. The reproduction of binaural signals in the Spherical-Harmonics (SH) domain using Ambisonics is now a well-established methodology, with flexible binaural processing realized using SH representations of the sound-field and the Head-Related Transfer Function (HRTF). However, in most practical cases, the binaural reproduction is order-limited, which introduces truncation errors that have a detrimental effect on the perception of the reproduced signals, mainly due to the truncation of the HRTF. Recently, it has been shown that manipulating the HRTF phase component, by ear-alignment, significantly reduces its effective SH order while preserving its phase information, which may be beneficial for alleviating the above detrimental effect. Incorporating the ear-aligned HRTF into the binaural reproduction process has been suggested by using Bilateral Ambisonics, which is an Ambisonics representation of the sound-field formulated at the two ears. While this method imposes challenges on acquiring the sound-field, and specifically, on applying head-rotations, it leads to a significant reduction in errors caused by the limited-order reproduction, which yields a substantial improvement in the perceived binaural reproduction quality even with first order SH

Roomalive: Magical experiences enabled by scalable, adaptive projector-camera units

ABSTRACT RoomAlive is a proof-of-concept prototype that transforms any room into an immersive, augmented entertainment experience. Our system enables new interactive projection mapping experiences that dynamically adapts content to any room. Users can touch, shoot, stomp, dodge and steer projected content that seamlessly co-exists with their existing physical environment. The basic building blocks of RoomAlive are projector-depth camera units, which can be combined through a scalable, distributed framework. The projector-depth camera units are individually autocalibrating, self-localizing, and create a unified model of the room with no user intervention. We investigate the design space of gaming experiences that are possible with RoomAlive and explore methods for dynamically mapping content based on room layout and user position. Finally we showcase four experience prototypes that demonstrate the novel interactive experiences that are possible with RoomAlive and discuss the design challenges of adapting any game to any room

‘Rafoogari’ of Najibabad

This paper will discuss the still continuous and centuries old skill of “Rafoogari” or the Darning and Maintenance of Pashmina Shawls by the Rafoogars or Darners of Najibabad, an historical town in western Uttar Pradesh. It is the home of several ‘Rafoogar’ families and the hub of the kani shawl trade. While Kashmiri pashmina shawls have been elaborately researched, the important role of darners in the maintenance of these priceless shawls has not yet been recognized. Although darning is a highly intricate and laborious task necessary to the maintenance, restoration, and renewal of the shawls, the role of the darners has remained unnoticed, possibly because the hallmark of good darning is to “share invisibility.” The tradition of production of these intricately designed and expensive shawls came from Central Asia to India along with Islam and got further refined by local cultural influence, pushing the technique to its creative limit in a process of appropriation and acculturation lasting more than five centuries. The production of these shawls has become almost extinct. The socio–cultural conditions that made such a practice possible have changed. Normal production of such exquisite pieces is not possible anymore. The continuing tradition of darning becomes extremely significant in this context. The special skill of the darners has been helping to rescue a substantial number of priceless shawls from destruction. Darning has kept them in circulation and continuous use until today in changing circumstances and in an interesting simultaneous transformation of the product and the market instead of being preserved only in museum collections. The paper will highlight this particular cultural approach to objects where historicity is maintained without sacrificing the use value. It will further discuss a complex range of issues in conservation, restoration and renewal of cultural products raised in the context of darning as an independent practice

Source Directivity and Spatial Audio for Interactive Wave-based Sound Propagation

Presented at the 20th International Conference on Auditory Display (ICAD2014), June 22-25, 2014, New York, NY.This paper presents an approach to model time-varying source directivity and HRTF-based spatial audio for wave-based sound propagation at interactive rates. The source directivity is expressed as a linear combination of elementary spherical harmonic sources. The propagated sound field due to each spherical harmonic source is precomputed and stored in an offline step. At runtime, the timevarying source directivity is decomposed into spherical harmonic coefficients. These coefficients are combined with precomputed spherical harmonic sound fields to generate propagated sound field at the listener position corresponding to the directional source. In order to compute spatial audio for a moving and rotating listener, an efficient plane-wave decomposition approach based on the derivatives of the sound field is presented. The source directivity and spatial audio approach have been integrated with the Half-Life 2 game engine and the Oculus Rift head-mounted display to enable realistic acoustic effects for virtual environments and games