Ten-Hand Piano: A Networked Music Installation

This paper presents the latest developments of the Public Sound Objects (PSOs) system, an experimental framework to implement and test new concepts for Networked Music. The project of a Public interactive installation using the PSOs system was commissioned in 2007 by Casa da Musica, the main concert hall space in Porto. It resulted in a distributed musical structure with up to ten interactive performance terminals distributed along the Casa da Musica’s hallways, collectively controlling a shared acoustic piano. The installation allows the visitors to collaborate remotely with each other, within the building, using a software interface custom developed to facilitate collaborative music practices and with no requirements in terms previous knowledge of musical performance

Primate drum kit: A system for studying acoustic pattern production by non-human primates using acceleration and strain sensors

The possibility of achieving experimentally controlled, non-vocal acoustic production in non-human primates is a key step to enable the testing of a number of hypotheses on primate behavior and cognition. However, no device or solution is currently available, with the use of sensors in non-human animals being almost exclusively devoted to applications in food industry and animal surveillance. Specifically, no device exists which simultaneously allows: (i) spontaneous production of sound or music by non-human animals via object manipulation, (ii) systematical recording of data sensed from these movements, (iii) the possibility to alter the acoustic feedback properties of the object using remote control. We present two prototypes we developed for application with chimpanzees (Pan troglodytes) which, while fulfilling the aforementioned requirements, allow to arbitrarily associate sounds to physical object movements. The prototypes differ in sensing technology, costs, intended use and construction requirements. One prototype uses four piezoelectric elements embedded between layers of Plexiglas and foam. Strain data is sent to a computer running Python through an Arduino board. A second prototype consists in a modified Wii Remote contained in a gum toy. Acceleration data is sent via Bluetooth to a computer running Max/MSP. We successfully pilot tested the first device with a group of chimpanzees. We foresee using these devices for a range of cognitive experiments. © 2013 by the authors; licensee MDPI, Basel, Switzerland

Transforming musical performance: activating the audience as digital collaborators

Digital technologies have transformed the performance practice, recording and distribution technologies, economy and sonic landscape of music in a process of change that began in the early 1980s. Recent technological developments have opened up the possibility of embodied interaction between audiences and performers, reframing music performance as a collaborative improvisatory space that affords Interactive Musical Participation. The research in this practice-based thesis looks at the relationship and experience of audience members and musicians exploring Interactive Musical Participation within the wide stylistic framework of contemporary jazz. It also studies the potential for the creation of compositional, technological and performance protocols to enable successful Interactive Musical Participation. This has been achieved through a process of mapping the methodology behind the composition, technical infrastructure, performances and post-performance analysis of a series of musical artefacts. Cook (2001 and 2009) suggests that researchers in this field should “Make a piece, not an instrument or controller” and this dictum has influenced the development of the technical infrastructure for this research. Easily accessible and low-cost digital audio workstations Ableton Live (2017) and Logic Pro X (Apple, 2019) as well as the digital protocols Open Sound Control (OSC) (Opensoundcontrol.org) have been utilised to deliver the programming and networking requirements. A major innovation stemming from this project has been the development of the Deeper Love Soundpad App, a sample playback app for Apple smartphones and iPads, in collaboration with Dr. Rob Toulson. The theoretical background to this research has been informed by actornetwork theory, the sociological approach developed by Bruno Latour (2005), Michel Callon (1986) and John Law (1992). Actor-network theory (ANT) provides a framework for understanding the mechanics of power and organisation within heterogeneous non-hierarchical networks. Mapping and analysing the ANT networks and connections created by the research performances has provided valuable data in the Interactive Musical Participatio

Computer Musicking: Designing for Collaborative Digital Musical Interaction.

PhDThis thesis is about the design of software which enables groups of people to make music together. Networked musical interaction has been an important aspect of Sound and Music Computing research since the early days, although collaborative music software has yet to gain mainstream popularity, and there is currently limited research on the design of such interfaces. This thesis draws on research from Computer Supported Cooperative Work (CSCW) to explore the design of systems for Collaborative Digital Musical Interaction (CDMI). A central focus of this research is the concept of Awareness: a person’s understanding of what is happening, and of who is doing what. A novel software interface is developed and used over three experimental studies to investigate the effects different interface designs have on the way groups of musicians collaborate. Existing frameworks from CSCW are extended to accommodate the properties of music as an auditory medium, and theories of conventional musical interaction are used to elaborate on the nature of music making as a collaborative and social activity which is focused on process-oriented creativity. This research contributes to the fields of Human-Computer Interaction (HCI), Computer Supported Cooperative Work, and Sound and Music Computing through the identification of empirically derived design implications and recommendations for collaborative musical environments. These guidelines are demonstrated through the design of a hypothetical collaborative music system. This thesis also contributes towards the methodology for evaluating such systems, and considers the distinctions between CDMI and the forms of collaboration traditionally studied within CSCW.Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Account Award (DTA)

Apps, Agents, and Improvisation: Ensemble Interaction with Touch-Screen Digital Musical Instruments

This thesis concerns the making and performing of music with new digital musical instruments (DMIs) designed for ensemble performance. While computer music has advanced to the point where a huge variety of digital instruments are common in educational, recreational, and professional music-making, these instruments rarely seek to enhance the ensemble context in which they are used. Interaction models that map individual gestures to sound have been previously studied, but the interactions of ensembles within these models are not well understood. In this research, new ensemble-focussed instruments have been designed and deployed in an ongoing artistic practice. These instruments have also been evaluated to find out whether, and if so how, they affect the ensembles and music that is made with them. Throughout this thesis, six ensemble-focussed DMIs are introduced for mobile touch-screen computers. A series of improvised rehearsals and performances leads to the identification of a vocabulary of continuous performative touch-gestures and a system for tracking these collaborative performances in real time using tools from machine learning. The tracking system is posed as an intelligent agent that can continually analyse the gestural states of performers, and trigger a response in the performers' user interfaces at appropriate moments. The hypothesis is that the agent interaction and UI response can enhance improvised performances, allowing performers to better explore creative interactions with each other, produce better music, and have a more enjoyable experience. Two formal studies are described where participants rate their perceptions of improvised performances with a variety of designs for agent-app interaction. The first, with three expert performers, informed refinements for a set of apps. The most successful interface was redesigned and investigated further in a second study with 16 non-expert participants. In the final interface, each performer freely improvised with a limited number of notes; at moments of peak gestural change, the agent presented users with the opportunity to try different notes. This interface is shown to produce performances that are longer, as well as demonstrate improved perceptions of musical structure, group interaction, enjoyment and overall quality. Overall, this research examined ensemble DMI performance in unprecedented scope and detail, with more than 150 interaction sessions recorded. Informed by the results of lab and field studies using quantitative and qualitative methods, four generations of ensemble-focussed interface have been developed and refined. The results of the most recent studies assure us that the intelligent agent interaction does enhance improvised performances

Sonic interactions in virtual environments

This book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments

Sonic Interactions in Virtual Environments

This open access book tackles the design of 3D spatial interactions in an audio-centered and audio-first perspective, providing the fundamental notions related to the creation and evaluation of immersive sonic experiences. The key elements that enhance the sensation of place in a virtual environment (VE) are: Immersive audio: the computational aspects of the acoustical-space properties of Virutal Reality (VR) technologies Sonic interaction: the human-computer interplay through auditory feedback in VE VR systems: naturally support multimodal integration, impacting different application domains Sonic Interactions in Virtual Environments will feature state-of-the-art research on real-time auralization, sonic interaction design in VR, quality of the experience in multimodal scenarios, and applications. Contributors and editors include interdisciplinary experts from the fields of computer science, engineering, acoustics, psychology, design, humanities, and beyond. Their mission is to shape an emerging new field of study at the intersection of sonic interaction design and immersive media, embracing an archipelago of existing research spread in different audio communities and to increase among the VR communities, researchers, and practitioners, the awareness of the importance of sonic elements when designing immersive environments

Moving sounds and sonic moves : exploring interaction quality of embodied music mediation technologies through a user-centered perspective

This research project deals with the user-experience related to embodied music mediation technologies. More specifically, adoption and policy problems surrounding new media (art) are considered, which arise from the usability issues that to date pervade new interfaces for musical expression. Since the emergence of new wireless mediators and control devices for musical expression, there is an explicit aspiration of the creative industries and various research centers to embed such technologies into different areas of the cultural industries. The number of applications and their uses have exponentially increased over the last decade. Conversely, many of the applications to date still suffer from severe usability problems, which not only hinder the adoption by the cultural sector, but also make culture participants take a rather cautious, hesitant, or even downright negative stance towards these technologies. Therefore, this thesis takes a vantage point that is in part sociological in nature, yet has a link to cultural studies as well. It combines this with a musicological frame of reference to which it introduces empirical user-oriented approaches, predominantly taken from the field of human-computer-interaction studies. This interdisciplinary strategy is adopted to cope with the complex nature of digital embodied music controlling technologies. Within the Flanders cultural (and creative) industries, opportunities of systems affiliated with embodied interaction are created and examined. This constitutes an epistemological jigsaw that looks into 1) “which stakeholders require what various levels of involvement, what interactive means and what artistic possibilities?”, 2) “the way in which artistic aspirations, cultural prerequisites and operational necessities of (prospective) users can be defined?”, 3) “how functional, artistic and aesthetic requirements can be accommodated?”, and 4) “how quality of use and quality of experience can be achieved, quantified, evaluated and, eventually, improved?”. Within this multi-facetted problem, the eventual aim is to assess the applicability of the foresaid technology, both from a theoretically and empirically sound basis, and to facilitate widening and enhancing the adoption of said technologies. Methodologically, this is achieved by 1) applied experimentation, 2) interview techniques, 3) self-reporting and survey research, 4) usability evaluation of existing devices, and 5) human-computer interaction methods applied – and attuned – to the specific case of embodied music mediation technologies. Within that scope, concepts related to usability, flow, presence, goal assessment and game enjoyment are scrutinized and applied, and both task- and experience-oriented heuristics and metrics are developed and tested. In the first part, covering three chapters, the general context of the thesis is given. In the first chapter, an introduction to the topic is offered and the current problems are enumerated. In the second chapter, a broader theoretical background is presented of the concepts that underpin the project, namely 1) the paradigm of embodiment and its connection to musicology, 2) a state of the arts concerning new interfaces for musical expression, 3) an introduction into HCI-usability and its application domain in systematic musicology, 4) an insight into user-centered digital design procedures, and 5) the challenges brought about by e-culture and digitization for the cultural-creative industries. In the third chapter, the state of the arts concerning the available methodologies related to the thesis’ endeavor is discussed, a set of literature-based design guidelines are enumerated and from this a conceptual model is deduced which is gradually presented throughout the thesis, and fully deployed in the “SoundField”-project (as described in Chapter 9). The following chapters, contained in the second part of the thesis, give a quasi-chronological overview of how methodological concepts have been applied throughout the empirical case studies, aimed specifically at the exploration of the various aspects of the complex status quaestionis. In the fourth chapter, a series of application-based tests, predominantly revolving around interface evaluation, illustrate the complex relation between gestural interfaces and meaningful musical expression, advocating a more user-centered development approach to be adopted. In the fifth chapter, a multi-purpose questionnaire dubbed “What Moves You” is discussed, which aimed at creating a survey of the (prospective) end-users of embodied music mediation technologies. Therefore, it primarily focused on cultural background, musical profile and preferences, views on embodied interaction, literacy of and attitudes towards new technology and participation in digital culture. In the sixth chapter, the ethnographical studies that accompanied the exhibition of two interactive art pieces, entitled "Heart as an Ocean" & "Lament", are discussed. In these studies, the use of interview and questionnaire methodologies together with the presentation and reception of interactive art pieces, are probed. In the seventh chapter, the development of the collaboratively controlled music-game “Sync-In-Team” is presented, in which interface evaluation, presence, game enjoyment and goal assessment are the pivotal topics. In the eighth chapter, two usability studies are considered, that were conducted on prototype systems/interfaces, namely a heuristic evaluation of the “Virtual String” and a usability metrics evaluation on the “Multi-Level Sonification Tool”. The findings of these two studies in conjunction with the exploratory studies performed in association with the interactive art pieces, finally gave rise to the “SoundField”-project, which is recounted in full throughout the ninth chapter. The integrated participatory design and evaluation method, presented in the conceptual model is fully applied over the course of the “SoundField”-project, in which technological opportunities and ecological validity and applicability are investigated through user-informed development of numerous use cases. The third and last part of the thesis renders the final conclusions of this research project. The tenth chapter sets out with an epilogue in which a brief overview is given on how the state of the arts has evolved since the end of the project (as the research ended in 2012, but the research field has obviously moved on), and attempts to consolidate the implications of the research studies with some of the realities of the Flemish cultural-creative industries. Chapter eleven continues by discussing the strengths and weaknesses of the conceptual model throughout the various stages of the project. Also, it comprises the evaluation of the hypotheses, how the assumptions that were made held up, and how the research questions eventually could be assessed. Finally, the twelfth and last chapter concludes with the most important findings of the project. Also, it discusses some of the implications on cultural production, artistic research policy and offers an outlook on future research beyond the scope of the “SoundField” project

Timbre from Sound Synthesis and High-level Control Perspectives

International audienceExploring the many surprising facets of timbre through sound manipulations has been a common practice among composers and instrument makers of all times. The digital era radically changed the approach to sounds thanks to the unlimited possibilities offered by computers that made it possible to investigate sounds without physical constraints. In this chapter we describe investigations on timbre based on the analysis by synthesis approach that consists in using digital synthesis algorithms to reproduce sounds and further modify the parameters of the algorithms to investigate their perceptual relevance. In the first part of the chapter timbre is investigated in a musical context. An examination of the sound quality of different wood species for xylophone making is first presented. Then the influence of instrumental control on timbre is described in the case of clarinet and cello performances. In the second part of the chapter, we mainly focus on the identification of sound morphologies, so called invariant sound structures responsible for the evocations induced by environmental sounds by relating basic signal descriptors and timbre descriptors to evocations in the case of car door noises, motor noises, solid objects, and their interactions

Evaluating Embodiment in Musical Instrument Modification and Augmentation

This PhD seeks to identify key aspects that optimise the learning process of new musical interfaces by professional musicians. Modifying or extending an existing musical instrument can impact players' skills. Fluency of execution or pitch accuracy can deteriorate due to demands on the performer's attention from the unfamiliarity of the instrument. As a result, players may require additional training on a modified instrument before they regain their fluency. The problem is that performers, especially professional players who have already invested many years in the unmodified musical instrument, might prefer to start from a high level. Thus, designing a new instrument that builds upon existing skills can be appealing. However, which design strategies might support such a goal? Which aspects of the original design should be preserved? How can we assess whether the resulting modified instrument allows the performer to retain their skills? This research presents four studies that tackle these questions. Results from the first two studies suggest that the design strategy should focus on participants’ sensorimotor imagery rather than the instrument's auditory feedback. During these studies, participants were still able to retain their fluency and pitch accuracy even in the presence of disrupting or irrelevant auditory feedback. Two additional studies propose quantitative methods to evaluate skill retention in instrument modification. This research can advise designers on whether they are on the right track in crafting an interface that builds upon existing skills. This challenge may apply to augmented instruments, the modification of existing musical instruments, or new digital instruments