An audio-visual system for object-based audio : from recording to listening

Object-based audio is an emerging representation for audio content, where content is represented in a reproduction format-agnostic way and, thus, produced once for consumption on many different kinds of devices. This affords new opportunities for immersive, personalized, and interactive listening experiences. This paper introduces an end-to-end object-based spatial audio pipeline, from sound recording to listening. A high-level system architecture is proposed, which includes novel audiovisual interfaces to support object-based capture and listenertracked rendering, and incorporates a proposed component for objectification, that is, recording content directly into an object-based form. Text-based and extensible metadata enable communication between the system components. An open architecture for object rendering is also proposed. The system’s capabilities are evaluated in two parts. First, listener-tracked reproduction of metadata automatically estimated from two moving talkers is evaluated using an objective binaural localization model. Second, object-based scene capture with audio extracted using blind source separation (to remix between two talkers) and beamforming (to remix a recording of a jazz group) is evaluate

Spatial Audio Mixing in Virtual Reality

The development of Virtual Reality (VR) systems and multimodal simulations presents possibilities in spatial music mixing, be it in virtual spaces, for ensembles and orchestral compositions, or for surround sound in film and music. Traditionally, user interfaces for mixing music have employed the channel strip metaphor for controlling volume, panning, and other audio effects, which are aspects that also have grown into the culture of mixing music spatially. Simulated rooms and two-dimensional panning systems are simply implemented on computer screens to facilitate the placement of sound sources within space. This paper presents design aspects for mixing in VR, investigating already existing virtual music mixing products, and creating a framework from which a virtual spatial-music mixing tool can be implemented. Finally, the tool will be tested against a similar computer version to examine whether or not the sensory benefits and palpable spatial proportions of a VE can improve the process of mixing 3D sound

Conflicts, integration, hybridization of subcultures: An ecological approach to the case of queercore

This paper investigates the case study of queercore, providing a socio-historical analysis of its subcultural production, in the terms of what Michel Foucault has called archaeology of knowledge (1969). In particular, we will focus on: the self-definition of the movement; the conflicts between the two merged worlds of punk and queer culture; the \u201cinternal-subcultural\u201d conflicts between both queercore and punk, and between queercore and gay\lesbian music culture; the political aspects of differentiation. In the conclusion, we will offer an innovative theoretical proposal about the interpretation of subcultures in ecological and semiotic terms, combining the contribution of the American sociologist Andrew Abbot and of the Russian semiologist Jurij Michajlovi\u10d Lotma

Evaluation of an Audio-haptic Sensory Substitution Device for Enhancing Spatial Awareness for the Visually Impaired

SIGNIFICANCE: Visually impaired participants were surprisingly fast in learning a new sensory substitution device, which allows them to detect obstacles within a 3.5-m radius and to find the optimal path in between. Within a few hours of training, participants successfully performed complex navigation as well as with the white cane. PURPOSE: Globally, millions of people live with vision impairment, yet effective assistive devices to increase their independence remain scarce. A promising method is the use of sensory substitution devices, which are human-machine interfaces transforming visual into auditory or tactile information. The Sound of Vision (SoV) system continuously encodes visual elements of the environment into audio-haptic signals. Here, we evaluated the SoV system in complex navigation tasks, to compare performance with the SoV system with the white cane, quantify training effects, and collect user feedback. METHODS: Six visually impaired participants received eight hours of training with the SoV system, completed a usability questionnaire, and repeatedly performed assessments, for which they navigated through standardized scenes. In each assessment, participants had to avoid collisions with obstacles, using the SoV system, the white cane, or both assistive devices. RESULTS: The results show rapid and substantial learning with the SoV system, with less collisions and higher obstacle awareness. After four hours of training, visually impaired people were able to successfully avoid collisions in a difficult navigation task as well as when using the cane, although they still needed more time. Overall, participants rated the SoV system's usability favorably. CONCLUSIONS: Contrary to the cane, the SoV system enables users to detect the best free space between objects within a 3.5-m (up to 10-m) radius and, importantly, elevated and dynamic obstacles. All in all, we consider that visually impaired people can learn to adapt to the haptic-auditory representation and achieve expertise in usage through well-defined training within acceptable time

User-Influenced/Machine-Controlled Playback: The variPlay Music App Format for Interactive Recorded Music

This paper concerns itself with an autoethnography of the five-year ‘variPlay’ project. This project drew from three consecutive rounds of research funding to develop an app format that could host both user interactivity to change the sound of recorded music in real-time, and a machine-driven mode that could autonomously remix, playing back a different version of a song upon every listen, or changing part way on user demand. The final funded phase involved commercialization, with the release of three apps using artists from the roster of project partner, Warner Music Group. The concept and operation of the app is discussed, alongside reflection on salient matters such as product development, music production, mastering, and issues encountered through the commercialization itself. The final apps received several thousand downloads around the world, in territories such as France, USA, and Mexico. Opportunities for future development are also presented

Virtual Heritage: Audio design for immersive virtual environments using researched spatializers.

This thesis work is based on a Virtual Heritage project being developed by the Systems of Representation research group. The objective of the project is to create a showcase demonstration on how the virtual reality (VR) could be used as an application for tourism in the heritage sector. In this context, my task was to develop a concept and prototype of how 'spatialized' sound could be used in a VR application. The initial location chosen for the concept was the ancient heritage burial site of Sammallahdenmäki, one of the Finnish heritage sites listed in the UNESCO register of World Heritage Sites. The thesis, that is written from an audio designer's perspective, focuses on three aspects of this project. First is the sound design for the Virtual Heritage project and the second is the quality of currently available 'spatializer' plug-ins used for headphone listening. In order to evaluate the process of designing 3D audio for virtual environments, the methods and principles within binaural rendering, sound design and immersion must be understood. Therefore, functions and theories within audio spatialization and 3D audio design are reviewed. Audio designers working on virtual reality content need the best possible solutions for creating believable 3D audio experiences. However, while working on the Virtual Heritage project, we did not find any comparative studies made about commercially available spatializer plug-ins for Unity. Thus, it was unknown what plug-in would have been the best possible solution for 3D audio spatialization. Consequently, two tests were conducted during this thesis work. First was an online test measuring which spatializer would be the most highly rated, in terms of perceived directional precision when utilizing head-related transfer functions without reverb or room simulations. The second was a comparative test studying if a spatialized audio rendering would increase immersion compared to non-spatialized audio rendering, when tested with the Virtual Heritage demonstration. The central aim in the showcase demonstration was to create an immersive virtual environment where users would feel as if they were travelling from the present, back to the Bronze Age, in order to understand and learn about the location’s unique history via auditory storytelling. The project was implemented utilising the Unity game engine. The research on music and other sound content used in the project’s sonic environment is explained. Finally, results of the project work are discussed.Tämä opinnäytetyö perustuu Virtual Heritage projektityöhön, joka on tehty Systems of Representation tutkimusryhmälle. Projektin tavoite on luoda malliesimerkki siitä, miten virtuaalitodellisuutta voitaisiin käyttää hyväksi turismisovelluksissa. Esimerkkikohteeksi projektille oli valittu Sammallahdenmäen hautaröykkiöt, joka on hyväksytty mukaan UNESCON maailmanperintöluetteloon. Tehtäväni oli toteuttaa Unity pelimoottorilla prototyyppi, jossa kartoitetaan virtuaalisen tilaäänen käyttömahdollisuuksia kyseisen teeman ympärillä. Opinnäyte on kirjoitettu äänisuunnittelijan näkökulmasta keskittyen kolmeen projektityöhön liittyvään keskeiseen osaan: prototyypin äänisuunnitteluun, immersion käsitteeseen sekä spatialisointi liitännäisten (plug-in) toimintaan ja laatuun. Virtuaalitodellisuuksiin sisältöä tuottavana äänisuunnittelijana tarvitsin parhaat mahdolliset työkalut uskottavan 3D äänimaailman luomiseen. Virtual Heritage projektia työstettäessä kävi kuitenkin ilmi, että ajankohtaista vertailevaa tutkimusta spatialisointi liitännäisten laatueroista ei ollut löydettävissä. Oli mahdotonta määritellä yksilöllisesti, mikä liitännäisistä on toimivin ratkaisu suurimmalle käyttäjäkunnalle. Täten oli tarpeellista toteuttaa kaksi tutkimusta. Ensimmäinen oli empiirinen verkkotutkimus, jolla arvioitiin spatialisointi liitännäisten suorituskykyä, kun mitataan subjektiivisesti äänen tilallisen sijoittumisen tarkkuutta kuulokekuuntelussa ilman kaikuprosessointeja. Tutkimuksessa parhaiten suoriutunut liitännäinen implementoitiin prototyyppiin. Toisessa kokeessa tutkittiin kuinka paljon implementoidun spatialisointi liitännäisen käyttäminen lisää virtuaalitodellisuuden immersiota verrattaessa spatialisoimattomaan ääneen, kun testialustana toimii Virtual Heritage prototyyppi. Projektin keskeisin tavoite oli luoda immersiivinen virtuaalitodellisuus, jossa käyttäjä voi kokea matkaavansa nykyajasta pronssikautiselle Sammallahdenmäelle ja oppia tällä tavoin kohteen ainutlaatuisesta historiasta äänikerronnan keinoin. Opinnäytetyössä esitellään äänikerronnan sisältöön ja toteutustapaan johtavat tutkimukset, tuotanto sekä ajatukset lopputuloksesta

Ir hacia arriba es siempre bueno: análisis multimodal de metáforas en un anuncio televisivo con Filmip, el método de identificación de metáforas fílmicas

Conceptual Metaphor Theory developed by Lakoff & Johnson (1980) suggested that we use metaphors to evaluate and communicate in our various environments. Although metaphors encompass a large variety of taxonomies, orientational metaphors are those that rely on spatial position to map concepts into other ones, referring to a relation of valence and verticality. Stated by Kövecses (2010) conceptual metaphors such orientational ones draw ‘upward’ and ‘downward’ spatial positions in which ‘upward’ is usually referred to as having positive connotations, whereby their opposites, ‘downwards’, are understood as negative. This paper seeks to unveil how the orientational metaphor good is up is employed in a filmic narrative of a language learning application for technological devices named Babbel. The present analysis is developed under the application of FILMIP (Filmic Metaphor Identification Procedure, Bort-Mir 2019). In the analyzed narrative, the orientational metaphor good is up is represented in the Babbel TV commercial (2018) as a tool for persuading customers that the best way of escalating positions at work is by learning new languages. This analysis demonstrates how orientational metaphors in multimodal media emerge as a convenient device for marketing campaigns in the context of social status improvement.La Teoría de la Metáfora Conceptual (Lakoff & Johnson 1980) sugiere que utilizamos metáforas para comunicarnos en diversos ambientes. Aunque las metáforas abarcan una gran variedad de taxonomías, las metáforas orientacionales son aquellas que se basan en la posición espacial para mapear ciertos conceptos en otros, refiriéndose a una relación de valencia y verticalidad. Las metáforas orientacionales dibujan posiciones espaciales “hacia arriba” y “hacia abajo” en las que se suele decir que “hacia arriba” tiene connotaciones positivas, mientras que su opuesto, “hacia abajo”, se entiende como negativo (Kövecses 2010). Este artículo pretende desvelar cómo la metáfora orientacional bueno es arriba se emplea en la narración fílmica sobre una aplicación móvil de aprendizaje de idiomas llamada Babbel. El presente análisis se desarrolla bajo la aplicación de FILMIP (Filmic Metaphor Identification Procedure, Bort-Mir 2019). En la narrativa analizada, la metáfora orientacional bueno es arriba se representa como una herramienta para persuadir a los clientes de que la mejor manera de escalar posiciones en el trabajo es mediante el aprendizaje de nuevos idiomas. Este análisis demuestra cómo las metáforas orientacionales en los medios multimodales emergen como un dispositivo brillante para las campañas de marketing en el contexto de la mejora del estatus social

Three Solitudes and a DJ: A Mashed-up Study of Counterpoint in a Digital Realm

This dissertation is primarily concerned with developing an understanding of how the use of pre-recorded digital audio shapes and augments conventional notions of counterpoint. It outlines a theoretical framework for analyzing the contrapuntal elements in electronically and digitally composed musics, specifically music mashups, and Glenn Gould’s Solitude Trilogy ‘contrapuntal radio’ works. Conventional studies of counterpoint encompass sixteenth- through early twentieth-century modernist and neo- classical materials but stop there. Composition by magnetic tape and computer software using pre-existing recorded audio offers the potential for a new study of music that displays clear contrapuntal elements but lacks the analytical models to outline the underlying musical systems. Central to these investigations is the assertion that counterpoint operates not only within the sphere of art music but also in the compositional logic of non-musical sound works (radio documentary) and in the harmonic and melodic underpinnings of popular music. The first chapter examines technological and cultural developments that contribute to the formation of digital contrapuntal music. The second and third chapters outline the traditional musical elements—harmony, form, and texture—of contrapuntal radio and mashups, respectively. Chapter Four explores how counterpoint exists in the sonic space of the stereo or mono sound field. Chapter Five presents the notion of program as a useful concept for analyzing interaction between lyric samples to form original narratives. These two final chapters present the original contributions from contrapuntal radio and mashups to a study of counterpoint. In each of these chapters, counterpoint forms the basis for how we perceive the underlying systems of musical works composed by traditional counterpoint or by assembling pre-existing recorded audio. The connection between the old and new is important, as one does not supplant but augment the other. As such, counterpoint is a fluid musical concept, rather than a fixed system of rules governing composition in a narrow musical palette

Computational Methods for Assisting Radio Drama Production.

PhD Theses.Radio Drama is a theatrical form of art that usually exists solely in the acoustic domain consisting of music, speech, and sound effects and is most often consumed through broadcast radio. This thesis proposes methods for assisting a human creator in producing radio dramas. Much research has been done to aiding creativity using artificial intelligence techniques in storytelling, music composition, the visual arts, and fi lm. Despite that, radio drama is under-represented in such research. Radio drama consists of both literary aspects, such as plot, story characters, or environments, as well as production aspects, such as speech, music, and sound effects. While plenty of research has been examining each of those aspects individually there is currently no research that combines such studies in the context of radio drama production. In this thesis, an interdisciplinary approach to assisting a human creator in radio drama production is developed. The task is explored through the joint prism of natural language processing, music information retrieval, and automatic mixing. We show that individual literary aspects of radio drama can be automatically extracted from a story draft provided by a human creator, by using natural language processing methods. Formal rules can be used to express the aforementioned elements in the form of a script able to be read and altered by both the human creator and the computer. We devise recommender systems for sound, music, and audio effects to retrieve the assets required for production. Rules derived from radio drama literature can then use those recorded assets to produce a radio drama mix in a semi-automatic way. Furthermore, an adaptive reverberation effect suggests reverberation settings for each track based on track content and past user choices. The degree of success for individual tasks in aiding production is demonstrated using examples of radio drama production from raw stories and validated through objective evaluation metrics, and listening tests