Embodied Musical Interaction

Music is a natural partner to human-computer interaction, offering tasks and use cases for novel forms of interaction. The richness of the relationship between a performer and their instrument in expressive musical performance can provide valuable insight to human-computer interaction (HCI) researchers interested in applying these forms of deep interaction to other fields. Despite the longstanding connection between music and HCI, it is not an automatic one, and its history arguably points to as many differences as it does overlaps. Music research and HCI research both encompass broad issues, and utilize a wide range of methods. In this chapter I discuss how the concept of embodied interaction can be one way to think about music interaction. I propose how the three “paradigms” of HCI and three design accounts from the interaction design literature can serve as a lens through which to consider types of music HCI. I use this conceptual framework to discuss three different musical projects—Haptic Wave, Form Follows Sound, and BioMuse

Emergent Perception and Video Games that Listen: Applying Sonic Virtuality for Creative and Intelligent NPC Behaviours

‘Non-player characters (NPCs)’ can present well-crafted behaviours and evoke engaging and immersive player experiences but such behaviour in current NPCs is illusory and only achievable within a controlled and linear/fixed video game context. NPCs struggle greatly to portray flexible or creative behaviours within an adaptive or procedurally generated environment and this is even more apparent in their relationship with sound. This paper posits that recent theoretical developments in cognitive psychology offer significant opportunity to advance NPC-AI and proposes that an intelligence framework, based upon Sonic Virtuality and integrated within an NPC, would offer distinct advantages over current systems. To illustrate this vision, a roadmap for future work is laid out using Sonic Virtuality as the foundation for a ‘synthetic listener’; an NPC capable of responding to procedurally generated and external (player-domain) audio. As a philosophical exploration, underlying principles are considered for other perception modalities, presenting an avenue of games-AI research that, ultimately, could dramatically improve NPC- ‘humanness’ and evoke a player-immersion and presence equivalent to linear/fixed AI but in much bigger, more complex virtual worlds

Understanding Music Interaction, and Why It Matters

This is the introductory chapter of a book dedicated to new research in, and emerging new understandings of, music and human-computer interaction—known for short as music interaction. Music interaction research plays a key role in innovative approaches to diverse musical activities, including performance, composition, education, analysis, production and collaborative music making. Music interaction is pivotal in new research directions in a range of activities, including audience participation, interaction between music and dancers, tools for algorithmic music, music video games, audio games, turntablism and live coding. More generally, music provides a powerful source of challenges and new ideas for human-computer interaction (HCI). This introductory chapter reviews the relationship between music and human-computer interaction and outlines research themes and issues that emerge from the collected work of researchers and practitioners in this book

The Aural and the Quotidian: Everyday Experience in Listening and Practice

The research herein comprises an examination of the following question: in what ways do our experiences of the everyday inhere in our experiences of the aural as aesthetic and meaningful? It is not concerned with forging a definition of everyday sound as a category of sonic effects, but instead an analysis of the ways that the everyday, aural and otherwise, is interpenetrating with our perceptual capacities and the cultural practices encompassing aural aesthetic production and experience. This thesis extends extant discourses surrounding the notion that the experience of sound as meaningful and aesthetic is connected to our general experience as embodied beings in the material world. The following analysis encompasses aspects of auditory perception, music aesthetics, and sound art production from the perspective of the body, as it is the locus of the listening subject situated within the domain of everyday experience. This includes an investigation of sound transduction technologies, as the devices that enable aural aesthetic practice are central to its analysis in the context of the everyday. Listening attitudes are transformed through cultural practice, structuring the relationship between the domain of the everyday, the embodied listening subject, sound recordings as cultural artefacts, and the attendant process of transduction. Discourses that attribute non-material, disembodied understandings to aesthetic experience are examined and challenged. From this, a fundamentally material, embodied approach to auditory experience is proposed, and with it a consideration of the ways that sound art and acousmatic music engage with the process of human understanding and the constitution of meaning in sound. Self-reflexive methodologies in aural aesthetic practice are exemplified, with the aim of promoting an expanded conception of aural context that includes the technological, cultural, and phenomenal aspects of its production

A General Theory Of Composition: Cross-modal Perception, Sound, Synergy and Meaning in contemporary composition practice.

Music extends beyond audibility. We perceive a performance – the work-on-stage – not only sonically but also visually and physically. Bridging science, phenomenology, eastern philosophy and music, A General Theory of Composition explores cross-modal perception and new synergies between musical and non-musical modes of creative expression through the media of sound, composition as a whole, and interdisciplinary research by posing the key-questions: What is SOUND? Where does SOUND begin, and where does it end? A General Theory of Composition, as a phenomenological investigation, reconfirms that the nature of both artistic practice and our sensory perception is holistic and reciprocally cross-modal. ‘Multi-sensory fluidity’ (Coessens), ‘sonic sensibility’ (Voegelin), ‘sensory substitution’ (McGann), naturally embedded in our modes of perception, suggest that our perception is always ‘embodied’ (Johnson) within its complex interdisciplinary Gestalt – composition as a whole. From this perspective the multisensory human body can be seen to engage meaningfully with the world via cross-modal listening, that promotes multi-sensory, psychosomatic reciprocal exchange between audible-visual-physical modes of perception. From the observation of my own artistic praxis, the holistic perspective of my theory brings forward the view on musical and/or any practice as a ‘lived experience’ (Stein) through theories of The Empathy Theory (Stein), Spheres of Human Essence (Walther), and The Speech Act Theory (Austin). It shares insights that: 1) sound/music, as language, is a phenomenon rooted in the philosophical domain; 2) there is no pure medium – all media are multi-modal, inter-subjective and reciprocally interconnected; 3) through the mode of ‘active listening’, as a means of communication, sound naturally crosses into other domains; 4) our perceptual modes intertwine, synthesise, and co-exist in a synergetic relationship within the complex chamber of our multi-sensory, psychosomatic, physical bodies. The exegesis is accompanied by two creative works, LIBERATO and SKETCHES, presented in three contextual subdivisions that collectively illustrate the workings of my theory and creative practice: the work-on-stage; cross-modal perception; expanded musical notation