Virtual Heritage: Audio design for immersive virtual environments using researched spatializers.

This thesis work is based on a Virtual Heritage project being developed by the Systems of Representation research group. The objective of the project is to create a showcase demonstration on how the virtual reality (VR) could be used as an application for tourism in the heritage sector. In this context, my task was to develop a concept and prototype of how 'spatialized' sound could be used in a VR application. The initial location chosen for the concept was the ancient heritage burial site of Sammallahdenmäki, one of the Finnish heritage sites listed in the UNESCO register of World Heritage Sites. The thesis, that is written from an audio designer's perspective, focuses on three aspects of this project. First is the sound design for the Virtual Heritage project and the second is the quality of currently available 'spatializer' plug-ins used for headphone listening. In order to evaluate the process of designing 3D audio for virtual environments, the methods and principles within binaural rendering, sound design and immersion must be understood. Therefore, functions and theories within audio spatialization and 3D audio design are reviewed. Audio designers working on virtual reality content need the best possible solutions for creating believable 3D audio experiences. However, while working on the Virtual Heritage project, we did not find any comparative studies made about commercially available spatializer plug-ins for Unity. Thus, it was unknown what plug-in would have been the best possible solution for 3D audio spatialization. Consequently, two tests were conducted during this thesis work. First was an online test measuring which spatializer would be the most highly rated, in terms of perceived directional precision when utilizing head-related transfer functions without reverb or room simulations. The second was a comparative test studying if a spatialized audio rendering would increase immersion compared to non-spatialized audio rendering, when tested with the Virtual Heritage demonstration. The central aim in the showcase demonstration was to create an immersive virtual environment where users would feel as if they were travelling from the present, back to the Bronze Age, in order to understand and learn about the location’s unique history via auditory storytelling. The project was implemented utilising the Unity game engine. The research on music and other sound content used in the project’s sonic environment is explained. Finally, results of the project work are discussed.Tämä opinnäytetyö perustuu Virtual Heritage projektityöhön, joka on tehty Systems of Representation tutkimusryhmälle. Projektin tavoite on luoda malliesimerkki siitä, miten virtuaalitodellisuutta voitaisiin käyttää hyväksi turismisovelluksissa. Esimerkkikohteeksi projektille oli valittu Sammallahdenmäen hautaröykkiöt, joka on hyväksytty mukaan UNESCON maailmanperintöluetteloon. Tehtäväni oli toteuttaa Unity pelimoottorilla prototyyppi, jossa kartoitetaan virtuaalisen tilaäänen käyttömahdollisuuksia kyseisen teeman ympärillä. Opinnäyte on kirjoitettu äänisuunnittelijan näkökulmasta keskittyen kolmeen projektityöhön liittyvään keskeiseen osaan: prototyypin äänisuunnitteluun, immersion käsitteeseen sekä spatialisointi liitännäisten (plug-in) toimintaan ja laatuun. Virtuaalitodellisuuksiin sisältöä tuottavana äänisuunnittelijana tarvitsin parhaat mahdolliset työkalut uskottavan 3D äänimaailman luomiseen. Virtual Heritage projektia työstettäessä kävi kuitenkin ilmi, että ajankohtaista vertailevaa tutkimusta spatialisointi liitännäisten laatueroista ei ollut löydettävissä. Oli mahdotonta määritellä yksilöllisesti, mikä liitännäisistä on toimivin ratkaisu suurimmalle käyttäjäkunnalle. Täten oli tarpeellista toteuttaa kaksi tutkimusta. Ensimmäinen oli empiirinen verkkotutkimus, jolla arvioitiin spatialisointi liitännäisten suorituskykyä, kun mitataan subjektiivisesti äänen tilallisen sijoittumisen tarkkuutta kuulokekuuntelussa ilman kaikuprosessointeja. Tutkimuksessa parhaiten suoriutunut liitännäinen implementoitiin prototyyppiin. Toisessa kokeessa tutkittiin kuinka paljon implementoidun spatialisointi liitännäisen käyttäminen lisää virtuaalitodellisuuden immersiota verrattaessa spatialisoimattomaan ääneen, kun testialustana toimii Virtual Heritage prototyyppi. Projektin keskeisin tavoite oli luoda immersiivinen virtuaalitodellisuus, jossa käyttäjä voi kokea matkaavansa nykyajasta pronssikautiselle Sammallahdenmäelle ja oppia tällä tavoin kohteen ainutlaatuisesta historiasta äänikerronnan keinoin. Opinnäytetyössä esitellään äänikerronnan sisältöön ja toteutustapaan johtavat tutkimukset, tuotanto sekä ajatukset lopputuloksesta

Retroactive spatializer

The aim of this project is increasing the research carried out by Sylvain Marchand, researcher of the LaBRI (Laboratoire Bordelais de Recherche en Informatique), about the RetroSpat System. RetroSpat stands for "Retroactive Spatializer" and it is a free software project for the spatialization of sounds represented in the spectral domain. This system is intended to be a spatializer with perceptive feedback. More precisely, RetroSpat can guess positions of physical sound sources (e.g. loudspeakers) from binaural inputs, and can then output multichannel signals to the loudspeakers while controlling the spatial location of virtual sound sources. So, the project will be based practically on improving the localization method as a result of the localization corresponding to a sound source, changing the software architecture to merge the localization and spatialization parts and on an improvement of the graphical user interface. This project is organized as follows. In the first section is presented some history of acousmatic music. Next a description of the model in Section 2 is done including the spatialization and localization methods. Section 3 is dedicated to the Source code. The problems with the actual software and its possible reasons of appearance are discussed in Section 4. Finally there are some conclusions of the work done

SPATIAL SOUND SYSTEM TO AID INTERACTIVITY IN A HUMAN CENTRED DESIGN EVALUATION OF AN AIRCRAFT CABIN ENVIRONMENT

There is a lot of research towards the concept of 3D sound in virtual reality environments. With the incipient growth in the significance of designing more realistic and immersive experiences for a Human Centred Design (HCD) approach, sound perception is believed to add an interactive element in maximizing the human perspective. In this context, the concept of an audio-visual interaction model between a passenger and a crew member in an immersive aircraft cabin environment is studied and presented in this paper. The study focuses on the design and usability of spatial sources as an interactive component in a regional aircraft cabin design for Human in the Loop evaluation. Sound sources are placed among the virtual manikins acting as passengers with the aim of building a realistic virtual environment for the user enacting the role of a crew member. The crew member, while walking throughthe cabin can orient and identify the position of the sound source inside the immersive Cabin environment. We review the 3D sound approaches and cues for sound spatialization in a virtual environment and propose that audio-visual interactivity aids the immersive Human centred design analysis

Audio Compression using a Modified Vector Quantization algorithm for Mastering Applications

Audio data compression is used to reduce the transmission bandwidth and storage requirements of audio data. It is the second stage in the audio mastering process with audio equalization being the first stage. Compression algorithms such as BSAC, MP3 and AAC are used as standards in this paper. The challenge faced in audio compression is compressing the signal at low bit rates. The previous algorithms which work well at low bit rates cannot be dominant at higher bit rates and vice-versa. This paper proposes an altered form of vector quantization algorithm which produces a scalable bit stream which has a number of fine layers of audio fidelity. This modified form of the vector quantization algorithm is used to generate a perceptually audio coder which is scalable and uses the quantization and encoding stages which are responsible for the psychoacoustic and arithmetical terminations that are actually detached as practically all the data detached during the prediction phases at the encoder side is supplemented towards the audio signal at decoder stage. Therefore, clearly the quantization phase which is modified to produce a bit stream which is scalable. This modified algorithm works well at both lower and higher bit rates. Subjective evaluations were done by audio professionals using the MUSHRA test and the mean normalized scores at various bit rates was noted and compared with the previous algorithms

The Sound of the hallmarks of cancer

The objective of this research is to create a mixed portfolio of data-driven composition and performance interfaces, fixed Electroacoustic/Computer music compositions, and live-improvised musical and audiovisual works reflecting cancer as a disease. The main methodology in generating the raw sonic material is the sonification of high-throughput, protein/RNA fold-change data, derived from the bio- molecular research of cancer cells. This data and relevant insight into the field are obtained as part of a collaboration with Barts Cancer Institute, in London, UK. Furthermore, for the purpose of musical effectiveness and reaching wider audiences, a focus has been placed on balancing the use of data-driven sonic material with composer-driven musical choices, by drawing upon the narrative of the Hallmarks of Cancer (Hanahan and Weinberg, 2011) which is a widely accepted conceptual framework in the field of cancer research for understanding the various biomolecular processes responsible for causing cancer. This method is adopted in order to inspire musical form, and guide some of the syntactic and aesthetic choices within both fixed and improvised works. In addition, this research also reflects upon the use of data sonification as an artistic tool and practice, while also addressing the contradictions and contention that arise as a result of scientific aims and expectations regarding sonification, resulting in a proposed original model for framing and classifying artistic works incorporating this approach

Leveraging eXtented Reality & Human-Computer Interaction for User Experi- ence in 360◦ Video

EXtended Reality systems have resurged as a medium for work and entertainment. While 360o video has been characterized as less immersive than computer-generated VR, its realism, ease of use and affordability mean it is in widespread commercial use. Based on the prevalence and potential of the 360o video format, this research is focused on improving and augmenting the user experience of watching 360o video. By leveraging knowledge from Extented Reality (XR) systems and Human-Computer Interaction (HCI), this research addresses two issues affecting user experience in 360o video: Attention Guidance and Visually Induced Motion Sickness (VIMS). This research work relies on the construction of multiple artifacts to answer the de- fined research questions: (1) IVRUX, a tool for analysis of immersive VR narrative expe- riences; (2) Cue Control, a tool for creation of spatial audio soundtracks for 360o video, as well as enabling the collection and analysis of captured metrics emerging from the user experience; and (3) VIMS mitigation pipeline, a linear sequence of modules (including optical flow and visual SLAM among others) that control parameters for visual modi- fications such as a restricted Field of View (FoV). These artifacts are accompanied by evaluation studies targeting the defined research questions. Through Cue Control, this research shows that non-diegetic music can be spatialized to act as orientation for users. A partial spatialization of music was deemed ineffective when used for orientation. Addi- tionally, our results also demonstrate that diegetic sounds are used for notification rather than orientation. Through VIMS mitigation pipeline, this research shows that dynamic restricted FoV is statistically significant in mitigating VIMS, while mantaining desired levels of Presence. Both Cue Control and the VIMS mitigation pipeline emerged from a Research through Design (RtD) approach, where the IVRUX artifact is the product of de- sign knowledge and gave direction to research. The research presented in this thesis is of interest to practitioners and researchers working on 360o video and helps delineate future directions in making 360o video a rich design space for interaction and narrative.Sistemas de Realidade EXtendida ressurgiram como um meio de comunicação para o tra- balho e entretenimento. Enquanto que o vídeo 360o tem sido caracterizado como sendo menos imersivo que a Realidade Virtual gerada por computador, o seu realismo, facili- dade de uso e acessibilidade significa que tem uso comercial generalizado. Baseado na prevalência e potencial do formato de vídeo 360o, esta pesquisa está focada em melhorar e aumentar a experiência de utilizador ao ver vídeos 360o. Impulsionado por conhecimento de sistemas de Realidade eXtendida (XR) e Interacção Humano-Computador (HCI), esta pesquisa aborda dois problemas que afetam a experiência de utilizador em vídeo 360o: Orientação de Atenção e Enjoo de Movimento Induzido Visualmente (VIMS). Este trabalho de pesquisa é apoiado na construção de múltiplos artefactos para res- ponder as perguntas de pesquisa definidas: (1) IVRUX, uma ferramenta para análise de experiências narrativas imersivas em VR; (2) Cue Control, uma ferramenta para a criação de bandas sonoras de áudio espacial, enquanto permite a recolha e análise de métricas capturadas emergentes da experiencia de utilizador; e (3) canal para a mitigação de VIMS, uma sequência linear de módulos (incluindo fluxo ótico e SLAM visual entre outros) que controla parâmetros para modificações visuais como o campo de visão restringido. Estes artefactos estão acompanhados por estudos de avaliação direcionados para às perguntas de pesquisa definidas. Através do Cue Control, esta pesquisa mostra que música não- diegética pode ser espacializada para servir como orientação para os utilizadores. Uma espacialização parcial da música foi considerada ineficaz quando usada para a orientação. Adicionalmente, os nossos resultados demonstram que sons diegéticos são usados para notificação em vez de orientação. Através do canal para a mitigação de VIMS, esta pesquisa mostra que o campo de visão restrito e dinâmico é estatisticamente significante ao mitigar VIMS, enquanto mantem níveis desejados de Presença. Ambos Cue Control e o canal para a mitigação de VIMS emergiram de uma abordagem de Pesquisa através do Design (RtD), onde o artefacto IVRUX é o produto de conhecimento de design e deu direcção à pesquisa. A pesquisa apresentada nesta tese é de interesse para profissionais e investigadores tra- balhando em vídeo 360o e ajuda a delinear futuras direções em tornar o vídeo 360o um espaço de design rico para a interação e narrativa

On binaural spatialization and the use of GPGPU for audio processing

3D recordings and audio, namely techniques that aim to create the perception of sound sources placed anywhere in 3 dimensional space, are becoming an interesting resource for composers, live performances and augmented reality. This thesis focuses on binaural spatialization techniques. We will tackle the problem from three different perspectives. The first one is related to the implementation of an engine for audio convolution, this is a real implementation problem where we will confront with a number of already available systems trying to achieve better results in terms of performances. General Purpose computing on Graphic Processing Units (GPGPU) is a promising approach to problems where a high parallelization of tasks is desirable. In this thesis the GPGPU approach is applied to both offline and real-time convolution having in mind the spatialization of multiple sound sources which is one of the critical problems in the field. Comparisons between this approach and typical CPU implementations are presented as well as between FFT and time domain approaches. The second aspect is related to the implementation of an augmented reality system having in mind an “off the shelf” system available to most home computers without the need of specialized hardware. A system capable of detecting the position of the listener through a head-tracking system and rendering a 3D audio environment by binaural spatialization is presented. Head tracking is performed through face tracking algorithms that use a standard webcam, and the result is presented over headphones, like in other typical binaural applications. With this system users can choose audio files to play, provide virtual positions for sources in an Euclidean space, and then listen as if they are coming from that position. If users move their head, the signals provided by the system change accordingly in real-time, thus providing the realistic effect of a coherent scene. The last aspect covered by this work is within the field of psychoacoustic, long term research where we are interested in understanding how binaural audio and recordings are perceived and how then auralization systems can be efficiently designed. Considerations with regard to the quality and the realism of such sounds in the context of ASA (Auditory Scene Analysis) are propose

Scanning Spaces: Paradigms for Spatial Sonification and Synthesis

In 1962 Karlheinz Stockhausen’s “Concept of Unity in Electronic Music” introduced a connection between the parameters of intensity, duration, pitch, and timbre using an accelerating pulse train. In 1973 John Chowning discovered that complex audio spectra could be synthesized by increasing vibrato rates past 20Hz. In both cases the notion of acceleration to produce timbre was critical to discovery. Although both composers also utilized sound spatialization in their works, spatial parameters were not unified with their synthesis techniques. This dissertation examines software studies and multimedia works involving the use of spatial and visual data to produce complex sound spectra. The culmination of these experiments, Spatial Modulation Synthesis, is introduced as a novel, mathematical control paradigm for audio-visual synthesis, providing unified control of spatialization, timbre, and visual form using high-speed sound trajectories.The unique, visual sonification and spatialization rendering paradigms of this disser- tation necessitated the development of an original audio-sample-rate graphics rendering implementation, which, unlike typical multimedia frameworks, provides an exchange of audio-visual data without downsampling or interpolation