End-user action-sound mapping design for mid-air music performance

How to design the relationship between a performer’s actions and an instrument’s sound response has been a consistent theme in Digital Musical Instrument (DMI) research. Previously, mapping was seen purely as an activity for DMI creators, but more recent work has exposed mapping design to DMI musicians, with many in the field introducing soware to facilitate end-user mapping, democratising this aspect of the DMI design process. This end-user mapping process provides musicians with a novel avenue for creative expression, and offers a unique opportunity to examine how practising musicians approach mapping design.Most DMIs suffer from a lack of practitioners beyond their initial designer, and there are few that are used by professional musicians over extended periods. The Mi.Mu Gloves are one of the few examples of a DMI that is used by a dedicated group of practising musicians, many of whom use the instrument in their professional practice, with a significant aspect of creative practice with the gloves being end-user mapping design. The research presented in this dissertation investigates end-user mapping practice with the Mi.Mu Gloves, and what influences glove musicians’ design decisions based on the context of their music performance practice, examining the question: How do end-users of a glove-based mid-air DMI design action–sound mapping strategies for musical performance?In the first study, the mapping practice of existing members of the Mi.Mu Glove community is examined. Glove musicians performed a mapping design task, which revealed marked differences in the mapping designs of expert and novice glove musicians, with novices designing mappings that evoked conceptual metaphors of spatial relationships between movement and music, while more experienced musicians focused on designing ergonomic mappings that minimised performer error.The second study examined the initial development period of glove mapping practice. A group of novice glove musicians were tracked in a longitudinal study. The findings supported the previous observation that novices designed mappings using established conceptual metaphors, and revealed that transparency and the audience’s ability to perceive their mappings was important to novice glove musicians. However, creative mapping was hindered by system reliability and the novices’ poorly trained posture recognition.The third study examined the mapping practice of expert glove musicians, who took part in a series of interviews. Findings from this study supported earlier observations that expert glove musicians focus on error minimisation and ergonomic, simple controls, but also revealed that the expert musicians embellished these simple controls with performative ancillary gestures to communicate aesthetic meaning. The expert musicians also suffered from system reliability, and had developed a series of gestural techniques to mitigate accidental triggering.The fourth study examined the effects of system-related error in depth. A laboratory study was used to investigate how system-related errors impacted a musician’s ability to acquire skill with the gloves, finding that a 5% rate of system error had a significant effect on skill acquisition.Learning from these findings, a series of design heuristics are presented, applicable for use in the fields of DMI design, mid-air interaction design and end-user mapping design

Multiparametric interfaces for fine-grained control of digital music

Digital technology provides a very powerful medium for musical creativity, and the way in which we interface and interact with computers has a huge bearing on our ability to realise our artistic aims. The standard input devices available for the control of digital music tools tend to afford a low quality of embodied control; they fail to realise our innate expressiveness and dexterity of motion. This thesis looks at ways of capturing more detailed and subtle motion for the control of computer music tools; it examines how this motion can be used to control music software, and evaluates musicians’ experience of using these systems. Two new musical controllers were created, based on a multiparametric paradigm where multiple, continuous, concurrent motion data streams are mapped to the control of musical parameters. The first controller, Phalanger, is a markerless video tracking system that enables the use of hand and finger motion for musical control. EchoFoam, the second system, is a malleable controller, operated through the manipulation of conductive foam. Both systems use machine learning techniques at the core of their functionality. These controllers are front ends to RECZ, a high-level mapping tool for multiparametric data streams. The development of these systems and the evaluation of musicians’ experience of their use constructs a detailed picture of multiparametric musical control. This work contributes to the developing intersection between the fields of computer music and human-computer interaction. The principal contributions are the two new musical controllers, and a set of guidelines for the design and use of multiparametric interfaces for the control of digital music. This work also acts as a case study of the application of HCI user experience evaluation methodology to musical interfaces. The results highlight important themes concerning multiparametric musical control. These include the use of metaphor and imagery, choreography and language creation, individual differences and uncontrol. They highlight how this style of interface can fit into the creative process, and advocate a pluralistic approach to the control of digital music tools where different input devices fit different creative scenarios

Audio-haptic relationships as compositional and performance strategies

As a performer of firstly acoustic and latterly electronic and electro-instrumental music, I constantly seek to improve my mode of interaction with the digital realm: that is, to achieve a high level of sensitivity and expression. This thesis illustrates reasons why making use of haptic interfaces—which offer physical feedback and resistance to the performer—may be viewed as an important approach in addressing the shortcomings of some the standard systems used to mediate the performer’s engagement with various sorts of digital musical information. By examining the links between sound and touch, and the performer-instrument relationship, various new compositional and performance strategies start to emerge. I explore these through a portfolio of original musical works, which span the continuum of composition and improvisation, largely based around performance paradigms for piano and live electronics. I implement new haptic technologies, using vibrotactile feedback and resistant interfaces, as well as exploring more metaphorical connections between sound and touch. I demonstrate the impact that the research brings to the creative musical outcomes, along with the implications that these techniques have on the wider field of live electronic musical performance

Subtlety and detail in digital musical instrument design

PhD thesisSubtlety and detail are fundamental to what makes musical instruments special, and worth dedicating a life’s practice to, for designer, maker, player and listener alike. While instruments are recognised and classified by form, it is the nuances of individual instruments that constitute their power to say what could not be said any other way. Digital musical instruments (DMI) have long been criticised as lacking expressive depth, but technology of sufficient fidelity now exists, which raises compelling questions. What can contemporary DMI designers learn from heritage practices about mastering subtlety and detail? What forms does this mastery take, and how can it be elucidated, compared and shared? Using DMI design tools, kits and activities as probes, this thesis addresses these questions from the perspectives of design, embodiment and craft. In a preliminary study, violin luthiers were asked about subtlety and detail in their practice and culture. The outcomes suggested that subtle details originate in the tacit and embodied realms, which are facilitated to develop by specific contexts, environments and materials. In the first study, attendees of a DMI research conference participated in a workshop reflecting on subtlety and detail. Attendees were divided into groups and explored the physical details of a DMI design kit, in an activity book ended by discussion. Responses focused on re-interpretations of instrumental identity, suggesting that the provided context motivated in the opposite direction to the original brief. In the second study, the same kit was deployed with single rather than co-located groups of digital luthiers, modifying instead the sound of the instrument via a Pure Data patch, and responses focused less on instrumental identity and more on gesture-sound mapping strategies. Provocatively, neither studies resulted in sustained focus on details, motivating a novel DMI probe and activity for individuals. In the third study, digital and traditional instrument makers, musicians and other creatives, were invited to handcraft the resonance models of a digital tuned percussion instrument using sculpting clay, responding to constrained briefs. Participants’ backgrounds deeply influenced their responses, and distinctive themes emerged related to aesthetics, tacit and embodied knowledge, and algorithmic pattern. This thesis introduces a scale-based ontology of DMI design, dividing detail into macro, meso and micro levels. Focusing on the micro scale, a series of reflections and suggestions are provided based on the investigations, for how DMI design practitioners, technologists and researchers can illuminate this domain, for the benefit of subtle and detailed digital musical expression

Being sound: FLOSS, flow and event in the composition and ensemble performance of free open computer music

This commentary describes my recent approach to writing compositions for the ensemble performance of computer music. Drawing on experimental music and improvisation, I contend that such music is best considered in terms of people’s situated and relational interplay. The compositional and performative question that permeates this thesis is ‘what can we do, in this time and space, with these tools available to us?’. As themes of equality and egalitarian access underpin this work throughout, I highlight my engagement with Free Libre Open Source Software (FLOSS) ideology and community, reflecting on how this achieves my aims. I describe my writing of text score compositions, making use of the term bounded improvisation, whose purposeful requirements for indeterminate realisation extends most current computer-based performance practice. Though no single strand of this research is perhaps unusual by itself, such an assemblage as that outlined above (incorporating composition, computer coding and ensemble performance practice) is, when allied to an understanding of electronic and computer music praxis, currently an underdeveloped approach. Such an approach I have thus chosen to term free open computer music. I incorporate two further pre-existing conceptual formulations to present a framework for constructing, reflecting on, and developing my work in this field. Firstly flow or 'immersed experience' is useful to explicate difficult to capture aspects of instrumental engagement and ensemble performance. Secondly, this portfolio of scores aims to produce well-constructed situations, facilitating spaces of flow which contain within their environments the opportunity for an event to take place. I present the outcomes of my practice as place-forming tactics that catalyse something to do, but not what to do, in performative spaces such as those described above. Such intentions define my aims for composition. These theoretical concerns, together with an allied consideration of the underpinning themes highlighted above, is a useful framework for refection and evaluation of this work

Designing Sound for Social Robots: Advancing Professional Practice through Design Principles

Sound is one of the core modalities social robots can use to communicate with the humans around them in rich, engaging, and effective ways. While a robot's auditory communication happens predominantly through speech, a growing body of work demonstrates the various ways non-verbal robot sound can affect humans, and researchers have begun to formulate design recommendations that encourage using the medium to its full potential. However, formal strategies for successful robot sound design have so far not emerged, current frameworks and principles are largely untested and no effort has been made to survey creative robot sound design practice. In this dissertation, I combine creative practice, expert interviews, and human-robot interaction studies to advance our understanding of how designers can best ideate, create, and implement robot sound. In a first step, I map out a design space that combines established sound design frameworks with insights from interviews with robot sound design experts. I then systematically traverse this space across three robot sound design explorations, investigating (i) the effect of artificial movement sound on how robots are perceived, (ii) the benefits of applying compositional theory to robot sound design, and (iii) the role and potential of spatially distributed robot sound. Finally, I implement the designs from prior chapters into humanoid robot Diamandini, and deploy it as a case study. Based on a synthesis of the data collection and design practice conducted across the thesis, I argue that the creation of robot sound is best guided by four design perspectives: fiction (sound as a means to convey a narrative), composition (sound as its own separate listening experience), plasticity (sound as something that can vary and adapt over time), and space (spatial distribution of sound as a separate communication channel). The conclusion of the thesis presents these four perspectives and proposes eleven design principles across them which are supported by detailed examples. This work contributes an extensive body of design principles, process models, and techniques providing researchers and designers with new tools to enrich the way robots communicate with humans

Mobile sound: media art in hybrid spaces

The thesis explores the relationships between sound and mobility through an examination of sound art. The research engages with the intersection of sound, mobility and art through original empirical work and theoretically through a critical engagement with sound studies. In dialogue with the work of De Certeau, Lefebvre, Huhtamo and Habermas in terms of the poetics of walking, rhythms, media archeology and questions of publicness, I understand sound art as an experimental mobile and public space. The thesis establishes and situates the emerging field of mobile sound art by mapping three key traditions of mobile sound art - locative art, sound art and public art - and creates a taxonomy of mobile sound art by defining four categories: 'placing sounds', 'sound platforms', 'sonifying mobility' and 'musical instruments' (each represented by one case study). In doing so it develops a methodology that is attentive to the specifics of the sonic and mobile of media experience. I demonstrate how sonic interactions and embodied mobility are designed and experienced in specific ways in each of the four case studies - 'Aura' by Symons (UK), 'Pophorns' by Torstensson and Sandelin (Sweden), 'SmSage' by Redfern and Borland (US) and 'Core Sample' by Rueb (US) (all 2007). In tracing the topos of the musical telephone, discussing the making and breaking of relevant micro publics, accounting for the polyphonies of footsteps and unwrapping bundles of rhythms, this thesis contributes to understanding complex media experiences in hybrid spaces. In doing so it critically sheds light on the quality of sonic artistic experiences, the audience engagement with urban, public and networked spaces and the relationship between sound art and everyday media experience. My thesis provides valuable insight into auditory ways of mobilising and making public spaces, non-verbal and embodied media practices, and rhythms and scales of mobile media experiences

Design Strategies for Adaptive Social Composition: Collaborative Sound Environments

In order to develop successful collaborative music systems a variety of subtle interactions need to be identified and integrated. Gesture capture, motion tracking, real-time synthesis, environmental parameters and ubiquitous technologies can each be effectively used for developing innovative approaches to instrument design, sound installations, interactive music and generative systems. Current solutions tend to prioritise one or more of these approaches, refining a particular interface technology, software design or compositional approach developed for a specific composition, performer or installation environment. Within this diverse field a group of novel controllers, described as ‘Tangible Interfaces’ have been developed. These are intended for use by novices and in many cases follow a simple model of interaction controlling synthesis parameters through simple user actions. Other approaches offer sophisticated compositional frameworks, but many of these are idiosyncratic and highly personalised. As such they are difficult to engage with and ineffective for groups of novices. The objective of this research is to develop effective design strategies for implementing collaborative sound environments using key terms and vocabulary drawn from the available literature. This is articulated by combining an empathic design process with controlled sound perception and interaction experiments. The identified design strategies have been applied to the development of a new collaborative digital instrument. A range of technical and compositional approaches was considered to define this process, which can be described as Adaptive Social Composition. Dan Livingston