‘A Swarm of Sound’ : Audiovisual Immersion in Björk’s VR Video ‘Family’

Author's accepted manuscript.This article explores the idea of audiovisual immersion through the portal of the virtual reality music video. Our focus falls on a close reading of Björk’s video, ‘Family’, which addresses questions of immersion in relation to user-experience, staging, and technological innovation. This article draws on the authors’ responses to the video by considering the implications of VR immersion in a new generation of music video productions. As part of the methodology on offer, a model for music analysis is devised for conceptualising virtual audiovisual space (VAVS) and the inextricable relationships between production and compositional design.acceptedVersio

Redefining Access to Large Audiovisual Archives through Embodied Experiences in Immersive Environments: Creativity & Cognition 2022 -- Graduate Student Symposium

Audiovisual archives are the mnemonic archives of the 21st century, with important cultural institutions increasingly digitizing their video collections. However, these remain mostly inaccessible, due to the sheer amount of content combined with the lack of innovative forms of engagement through compelling frameworks for their exploration. The present research therefore aims at redefining access to large video collections through embodied experiences in immersive environments. The author claims that, once users are empowered to be actors of the experience rather than mere spectators, their creativity is stimulated and narrative can emerge.Comment: Proceedings of the 14th Conference on Creativity and Cognitio

3D (embodied) projection mapping and sensing bodies : a study in interactive dance performance

This dissertation identifies the synergies between physical and virtual environments when designing for immersive experiences in interactive dance performances. The integration of virtual information in physical space is transforming our interactions and experiences with the world. By using the body and creative expression as the interface between real and virtual worlds, dance performance creates a privileged framework to research and design interactive mixed reality environments and immersive augmented architectures. The research is primarily situated in the fields of visual art and interaction design. It combines performance with transdisciplinary fields and intertwines practice with theory. The theoretical and conceptual implications involved in designing and experiencing immersive hybrid environments are analyzed using the reality–virtuality continuum. These theories helped frame the ways augmented reality architectures are achieved through the integration of dance performance with digital software and reception displays. They also helped identify the main artistic affordances and restrictions in the design of augmented reality and augmented virtuality environments for live performance. These pervasive media architectures were materialized in three field experiments, the live dance performances. Each performance was created in three different stages of conception, design and production. The first stage was to “digitize” the performer’s movement and brain activity to the virtual environment and our system. This was accomplished through the use of depth sensor cameras, 3D motion capture, and brain computer interfaces. The second stage was the creation of the computational architecture and software that aggregates the connections and mapping between the physical body and the spatial dynamics of the virtual environment. This process created real-time interactions between the performer’s behavior and motion and the real-time generative computer 3D graphics. Finally, the third stage consisted of the output modality: 3D projector based augmentation techniques were adopted in order to overlay the virtual environment onto physical space. This thesis proposes and lays out theoretical, technical, and artistic frameworks between 3D digital environments and moving bodies in dance performance. By sensing the body and the brain with the 3D virtual environments, new layers of augmentation and interactions are established, and ultimately this generates mixed reality environments for embodied improvisational self-expression.Radio-Television-Fil

Towards immersive generosity: The need for a novel framework to explore large audiovisual archives through embodied experiences in immersive environments

This article proposes an innovative framework to explore large audiovisual archives using Immersive Environments to place users inside a dataset and create an embodied experience. It starts by outlining the need for such a novel interface to meet the needs of archival scholars and the GLAM sector, and discusses issues in the current modes of access, mostly restrained to traditional information retrieval systems based on metadata. The paper presents the concept of ``generous interfaces" as a preliminary approach to address these issues, and argues some of the key reasons why employing Immersive Visual Storytelling might benefit such frameworks. The theory of embodiment is leveraged to justify this claim, showing how a more embodied understanding of a collection can result in a stronger engagement for the public. By placing users as actors in the experience rather than mere spectators, the emergence of narrative is driven by their interactions, with benefits in terms of engagement with the public and understanding of the cultural component. The framework we propose is applied to two existing installations to analyze them in-depth and critique them, highlighting the key directions to pursue for further development.Comment: This is the pre-published version (after peer-review

Immersed in Pop! Excursions into Compositional Design

Recent changes in consumer audio and music technology and distribution - for example the addition of 3D audio formats such as Dolby Atmos to music streaming services, the recent release of “Spatial Audio” on Apple and Beats products, the proliferation of musical content in virtual reality and 360º videos, etc. - have reignited a public discourse on concepts of immersion and interactivity in popular music and media. This raises questions and necessitates a deepening of popular musicological discourse in these areas. This thesis thus asks: what is the relationship between so-called immersive media and immersive experience? How are immersive and interactive experiences of audiovisual popular music compositionally designed? And to what degree do interpretations of immersion and interactivity in popular music imply agency on part of the listener/viewer? To address these questions, Bresler has authored or co-authored four articles and book chapters on music in immersive and interactive media with a focus on compositional design and immersion in pop music. In the framing chapter, these articles are contextualized through the coining of the term immersive staging, which is a framework for understanding how the perceived relationship between the performer and listener is mediated through technology, performativity, audiovisual compositional design, and aesthetics. Additionally, the chapter makes a case for the hermeneutic methodologies employed throughout.publishedVersio

A Broken Mirror Held to History’s Face On the Narrative Use of Computer Screens, Multi Screen Experiences, and a Transmedia Theoretical Console in the Popular Assassin’s Creed Series

This paper presents media theorist Nanna Verhoeff’s concept of the theoretical console, as a popular and overt form of transmedia narrative. The theoretical console is taken to be a transmedia assemblage that draws attention to itself, as comprising diverse and meaningful media objects, that can be connected in a shared narrative. My main examples of this concept here are those popular video games that spatially juxtapose several types of computer screens and computer uses, with a narrative emphasis. With extensive references to theory on screened media and on transmedia narratives, Assassin’s Creed IV: Black Flag is my main example case study. Specifically, this game and its series peers encourage historiographic contemplations, by assembling a theoretical console across several media forms. Other popular video games from that series provide variations of this same transmedia constellation, this “theoretical console”. In its transmedia constellation, with a second screen mobile phone app, and other complementary screen media, the fictionalized history of Assassin’s Creed IV: Black Flag implicates elements from its actual reality, across various forms of engagement. Assassin’s Creed IV: Black Flag presents a high fidelity historical fiction, to be comprehensively enacted. It mirrors the player’s use of differing computer screens diegetically within playable frame story sections. In addition, the complementary affordances of the mobile phone app, and integrated social media websites, all encourage its player to stay involved in this fictional world, even outside immediate play. With this, the game draws many activities into a single transmedial fiction constellation. Moreover, the game diegetically references online repositories for both its fictionalized history, and actual history. This use of computer screens, to form a transmedia constellation in the form of an overt theoretical console, is shown to complement this popular game’s hypermediative narrative of a fictional shadow war secretly driving actual human history, which then meaningfully posits how to theorize history in our everyday lives

Sonic interactions in virtual environments

This book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments

Sonic Interactions in Virtual Environments

This open access book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments

Embodiment and embodied design

Picture this. A preverbal infant straddles the center of a seesaw. She gently tilts her weight back and forth from one side to the other, sensing as each side tips downward and then back up again. This child cannot articulate her observations in simple words, let alone in scientific jargon. Can she learn anything from this experience? If so, what is she learning, and what role might such learning play in her future interactions in the world? Of course, this is a nonverbal bodily experience, and any learning that occurs must be bodily, physical learning. But does this nonverbal bodily experience have anything to do with the sort of learning that takes place in schools - learning verbal and abstract concepts? In this chapter, we argue that the body has everything to do with learning, even learning of abstract concepts. Take mathematics, for example. Mathematical practice is thought to be about producing and manipulating arbitrary symbolic inscriptions that bear abstract, universal truisms untainted by human corporeality. Mathematics is thought to epitomize our species’ collective historical achievement of transcending and, perhaps, escaping the mundane, material condition of having a body governed by haphazard terrestrial circumstance. Surely mathematics is disembodied