A Framework to Evaluate the Adoption Potential of Interactive Performance Systems for Popular Music

Popular music plays a central role in the lives of millions of people. It motivates beginners, engages experienced musicians, and plays both functional (e.g. churches) and non-functional (e.g. music festivals) roles in many contexts. Forming and maintaining a popular music ensemble can be challenging, particularly for part-time musicians who face other demands on their time. Where an ensemble has a functional role, performing music of consistent style and quality becomes imperative yet the demands of everyday life mean that it is not always possible to have a full complement of musicians. Interactive music technology has the potential to substitute for absent musicians to give a consistent musical output. However, the technology to achieve this (for popular music) is not yet mature, or in a suitable form for adoption and use by musicians who are not experienced with interactive music systems, or who are unprepared to work in experimental music or with experimental systems (a particular concern for functional ensembles). This paper proposes a framework of issues to be considered when developing interactive music technologies for popular music ensemble performance. It explores aspects that are complementary to technological concerns, focusing on adoption and practice to guide future technological developments

Interactive Sound in Performance Ecologies: Studying Connections among Actors and Artifacts

This thesis’s primary goal is to investigate performance ecologies, that is the compound of humans, artifacts and environmental elements that contribute to the result of a per- formance. In particular, this thesis focuses on designing new interactive technologies for sound and music. The goal of this thesis leads to the following Research Questions (RQs): • RQ1 How can the design of interactive sonic artifacts support a joint expression across different actors (composers, choreographers, and performers, musicians, and dancers) in a given performance ecology? • RQ2 How does each different actor influence the design of different artifacts, and what impact does this have on the overall artwork? • RQ3 How do the different actors in the same ecology interact, and appropriate an interactive artifact? To reply to these questions, a new framework named ARCAA has been created. In this framework, all the Actors of a given ecology are connected to all the Artifacts throughout three layers: Role, Context and Activity. This framework is then applied to one systematic literature review, two case studies on music performance and one case study in dance performance. The studies help to better understand the shaded roles of composers, per- formers, instrumentalists, dancers, and choreographers, which is relevant to better design interactive technologies for performances. Finally, this thesis proposes a new reflection on the blurred distinction between composing and designing a new instrument in a context that involves a multitude of actors. Overall, this work introduces the following contributions to the field of interaction design applied to music technology: 1) ARCAA, a framework to analyse the set of inter- connected relationship in interactive (music) performances, validated through 2 music studies, 1 dance study and 1 systematic literature analysis; 2) Recommendations for de- signing music interactive system for performance (music or dance), accounting for the needs of the various actors and for the overlapping on music composition and design of in- teractive technology; 3) A taxonomy of how scores have shaped performance ecologies in NIME, based on a systematic analysis of the literature on score in the NIME proceedings; 4) Proposal of a methodological approach combining autobiographical and idiographical design approaches in interactive performances.O objetivo principal desta tese é investigar as ecologias performativas, conjunto formado pelos participantes humanos, artefatos e elementos ambientais que contribuem para o resultado de uma performance. Em particular, esta tese foca-se na conceção de novas tecnologias interativas para som e música. O objetivo desta tese originou as seguintes questões de investigação (Research Questions RQs): • RQ1 Como o design de artefatos sonoros interativos pode apoiar a expressão con- junta entre diferentes atores (compositores, coreógrafos e performers, músicos e dançarinos) numa determinada ecologia performativa? • RQ2 Como cada ator influencia o design de diferentes artefatos e que impacto isso tem no trabalho artístico global? • RQ3 Como os diferentes atores de uma mesma ecologia interagem e se apropriam de um artefato interativo? Para responder a essas perguntas, foi criado uma nova framework chamada ARCAA. Nesta framework, todos os atores (Actores) de uma dada ecologia estão conectados a todos os artefatos (Artefacts) através de três camadas: Role, Context e Activity. Esta framework foi então aplicada a uma revisão sistemática da literatura, a dois estudos de caso sobre performance musical e a um estudo de caso em performance de dança. Estes estudos aju- daram a comprender melhor os papéis desempenhados pelos compositores, intérpretes, instrumentistas, dançarinos e coreógrafos, o que é relevante para melhor projetar as tec- nologias interativas para performances. Por fim, esta tese propõe uma nova reflexão sobre a distinção entre compor e projetar um novo instrumento num contexto que envolve uma multiplicidade de atores. Este trabalho apresenta as seguintes contribuições principais para o campo do design de interação aplicado à tecnologia musical: 1) ARCAA, uma framework para analisar o conjunto de relações interconectadas em performances interativas, validado através de dois estudos de caso relacionados com a música, um estudo de caso relacionado com a dança e uma análise sistemática da literatura; 2) Recomendações para o design de sistemas interativos musicais para performance (música ou dança), tendo em conta as necessidades dos vários atores e a sobreposição entre a composição musical e o design de tecnologia interactiva; 3) Uma taxonomia sobre como as partituras musicais moldaram as ecologias performativas no NIME, com base numa análise sistemática da literatura dos artigos apresentados e publicados nestas conferência; 4) Proposta de uma aborda- gem metodológica combinando abordagens de design autobiográfico e idiográfico em performances interativas

Methods and Technologies for the Analysis and Interactive Use of Body Movements in Instrumental Music Performance

List of related publications: http://www.federicovisi.com/publications/A constantly growing corpus of interdisciplinary studies support the idea that music is a complex multimodal medium that is experienced not only by means of sounds but also through body movement. From this perspective, musical instruments can be seen as technological objects coupled with a repertoire of performance gestures. This repertoire is part of an ecological knowledge shared by musicians and listeners alike. It is part of the engine that guides musical experience and has a considerable expressive potential. This thesis explores technical and conceptual issues related to the analysis and creative use of music-related body movements in instrumental music performance. The complexity of this subject required an interdisciplinary approach, which includes the review of multiple theoretical accounts, quantitative and qualitative analysis of data collected in motion capture laboratories, the development and implementation of technologies for the interpretation and interactive use of motion data, and the creation of short musical pieces that actively employ the movement of the performers as an expressive musical feature. The theoretical framework is informed by embodied and enactive accounts of music cognition as well as by systematic studies of music-related movement and expressive music performance. The assumption that the movements of a musician are part of a shared knowledge is empirically explored through an experiment aimed at analysing the motion capture data of a violinist performing a selection of short musical excerpts. A group of subjects with no prior experience playing the violin is then asked to mime a performance following the audio excerpts recorded by the violinist. Motion data is recorded, analysed, and compared with the expert’s data. This is done both quantitatively through data analysis xii as well as qualitatively by relating the motion data to other high-level features and structures of the musical excerpts. Solutions to issues regarding capturing and storing movement data and its use in real-time scenarios are proposed. For the interactive use of motion-sensing technologies in music performance, various wearable sensors have been employed, along with different approaches for mapping control data to sound synthesis and signal processing parameters. In particular, novel approaches for the extraction of meaningful features from raw sensor data and the use of machine learning techniques for mapping movement to live electronics are described. To complete the framework, an essential element of this research project is the com- position and performance of études that explore the creative use of body movement in instrumental music from a Practice-as-Research perspective. This works as a test bed for the proposed concepts and techniques. Mapping concepts and technologies are challenged in a scenario constrained by the use of musical instruments, and different mapping ap- proaches are implemented and compared. In addition, techniques for notating movement in the score, and the impact of interactive motion sensor systems in instrumental music practice from the performer’s perspective are discussed. Finally, the chapter concluding the part of the thesis dedicated to practical implementations describes a novel method for mapping movement data to sound synthesis. This technique is based on the analysis of multimodal motion data collected from multiple subjects and its design draws from the theoretical, analytical, and practical works described throughout the dissertation. Overall, the parts and the diverse approaches that constitute this thesis work in synergy, contributing to the ongoing discourses on the study of musical gestures and the design of interactive music systems from multiple angles

Interactive music: Balancing creative freedom with musical development.

PhDThis thesis is about interactive music, a musical experience that involves participation from the listener but is itself a composed piece of music and the Interactive Music Systems (IMSs) that create these experiences, such as a sound installation that responds to the movements of its audience. Some IMSs are brief marvels commanding only a few seconds of attention. Others engage those who participate for considerably longer. Our goal here is to understand why this difference arises and how we may then apply this understanding to create better interactive music experiences. I present a refined perspective of interactive music as an exploration into the relationship between action and sound. Reasoning about IMSs in terms of how they are subjectively perceived by a participant, I argue that fundamental to creating a captivating interactive music is the evolving cognitive process of making sense of a system through interaction. I present two new theoretical tools that provide complementary contributions to our understanding of this process. The first, the Emerging Structures model, analyses how a participant's evolving understanding of a system's behaviour engages and motivates continued involvement. The second, a framework of Perceived Agency, refines the notion of `creative control' to provide a better understanding of how the norms of music establish expectations of how skill will be demonstrated. I develop and test these tools through three practical projects: a wearable musical instrument for dancers created in collaboration with an artist, a controlled user study investigating the effects of constraining the functionality of a screen-based IMS, and an interactive sound installation that may only be explored through coordinated movement with another participant. This final work is evaluated formally through discourse analysis. Finally, I show how these tools may inform our understanding of an oft-cited goal within the field: conversational interaction with an interactive music system.Platform Grant (EPSRC EP/E045235/1)

Computer Musicking: Designing for Collaborative Digital Musical Interaction.

PhDThis thesis is about the design of software which enables groups of people to make music together. Networked musical interaction has been an important aspect of Sound and Music Computing research since the early days, although collaborative music software has yet to gain mainstream popularity, and there is currently limited research on the design of such interfaces. This thesis draws on research from Computer Supported Cooperative Work (CSCW) to explore the design of systems for Collaborative Digital Musical Interaction (CDMI). A central focus of this research is the concept of Awareness: a person’s understanding of what is happening, and of who is doing what. A novel software interface is developed and used over three experimental studies to investigate the effects different interface designs have on the way groups of musicians collaborate. Existing frameworks from CSCW are extended to accommodate the properties of music as an auditory medium, and theories of conventional musical interaction are used to elaborate on the nature of music making as a collaborative and social activity which is focused on process-oriented creativity. This research contributes to the fields of Human-Computer Interaction (HCI), Computer Supported Cooperative Work, and Sound and Music Computing through the identification of empirically derived design implications and recommendations for collaborative musical environments. These guidelines are demonstrated through the design of a hypothetical collaborative music system. This thesis also contributes towards the methodology for evaluating such systems, and considers the distinctions between CDMI and the forms of collaboration traditionally studied within CSCW.Engineering and Physical Sciences Research Council (EPSRC) Doctoral Training Account Award (DTA)

Designing expressive engagement with electronic and hyper instruments.The Electrumpet a case study

Using values-led participatory design (Iversen, Halskov, & Leong, 2013) as a foundation, this thesis argues the importance of values in the design decisions that steer the conception and development of new electroacoustic musical instruments. A model is introduced that defines seven distinct Personas, as different perspectives on the creation process of ‘performing instrument designers’. Second Order Virtuosity (Hildebrand, Lopes, Hoelzl, & Campo, 2016) and the persona model are cross-examined, substantiated by the virtuoso practice of peer ‘performing instrument designers’ (PIDs). The Electrumpet, a hyper instrument, is used as a case study for the application of the model in relation to its improved design and the evaluation of its progress as a musical instrument in general. The thesis concludes with a description of the technical implementation of the improved Electrumpet system and the compositional and improvisational strategies implemented

The Gestural Control of Audio Processing

Gesture enabled devices have become so ubiquitous in recent years that commands such as ‘pinch to zoom-in on an image’ are part of most people’s gestural vocabulary. Despite this, gestural interfaces have been used sparingly within the audio industry. The aim of this research project is to evaluate the effectiveness of a gestural interface for the control of audio processing. In particular, the ability of a gestural system to streamline workflow and rationalise the number of control parameters, thus reducing the complexity of Human Computer Interaction (HCI). A literature review of gestural technology explores the ways in which it can improve HCI, before focussing on areas of implementation in audio systems. Case studies of previous research projects were conducted to evaluate the benefits and pitfalls of gestural control over audio. The findings from these studies concluded that the scope of this project should be limited to two-dimensional gestural control. An elicitation of gestural preferences was performed to identify expert-user’s gestural associations. This data was used to compile a taxonomy of gestures and their most widely-intuitive parameter mappings. A novel interface was then produced using a popular tablet-computer. This facilitated the control of equalisation, compression and gating. Objective testing determined the performance of the gestural interface in comparison to traditional WIMP (Windows, Icons, Menus, Pointer) techniques, thus producing a benchmark for the system under test. Further testing is carried out to observe the effects of graphic user interfaces (GUIs) in a gestural system, in particular the suitability of skeuomorphic (knobs and faders) designs in modern DAWs (Digital Audio Workstations). A novel visualisation method, deemed more suitable for gestural interaction, is proposed and tested. Semantic descriptors are explored as a means of further improving the speed and usability of gestural interfaces, through the simultaneous control of multiple parameters. This rationalisation of control moves towards the implementation of gestural shortcuts and ‘continuous pre-sets’