Augmenting a guitar with its digital footprint

We explore how to digitally augment musical instruments by connecting them to their social histories. We describe the use of Internet of Things technologies to connect an acoustic guitar to its digital footprint – a record of how it was designed, built and played. We introduce the approach of crafting interactive decorative inlay into the body of an instrument that can then be scanned using mobile devices to reveal its digital footprint. We describe the design and construction of an augmented acoustic guitar called Carolan alongside activities to build its digital footprint through documented encounters with twenty-seven players in a variety of settings. We reveal the design challenge of mapping the different surfaces of the instrument to various facets of its footprint so as to afford appropriate experiences to players, audiences and technicians. We articulate an agenda for further research on the topic of connecting instruments to their social histories, including capturing and performing digital footprints and creating personalized and legacy experiences

Augmenting a guitar with its digital footprint

We explore how to digitally augment musical instruments by connecting them to their social histories. We describe the use of Internet of Things technologies to connect an acoustic guitar to its digital footprint – a record of how it was designed, built and played. We introduce the approach of crafting interactive decorative inlay into the body of an instrument that can then be scanned using mobile devices to reveal its digital footprint. We describe the design and construction of an augmented acoustic guitar called Carolan alongside activities to build its digital footprint through documented encounters with twenty-seven players in a variety of settings. We reveal the design challenge of mapping the different surfaces of the instrument to various facets of its footprint so as to afford appropriate experiences to players, audiences and technicians. We articulate an agenda for further research on the topic of connecting instruments to their social histories, including capturing and performing digital footprints and creating personalized and legacy experiences

Considering musical structure in location-based experiences

Locative music experiences are often non-linear and as such the final structure of the music heard is guided by the movements of the user. We note an absence of principles regarding how composers should approach the structuring of such locative soundtracks. For instance, how does one compose for a non-linear soundtrack using linear, pre-composed placed sounds, where fixed musical time is placed into the indeterminate time of the user’s experience? Furthermore, how does one create a soundtrack that is suitable for the location, but also functions as a coherent musical structure? We explore these questions by analyzing an existing ‘placed sound’ work from a traditional music theory perspective and in doing so reveal some structural principals from ‘fixed’ musical forms can also support the composition of contemporary locative music experiences

Considering musical structure in location-based experiences

Locative music experiences are often non-linear and as such the final structure of the music heard is guided by the movements of the user. We note an absence of principles regarding how composers should approach the structuring of such locative soundtracks. For instance, how does one compose for a non-linear soundtrack using linear, pre-composed placed sounds, where fixed musical time is placed into the indeterminate time of the user’s experience? Furthermore, how does one create a soundtrack that is suitable for the location, but also functions as a coherent musical structure? We explore these questions by analyzing an existing ‘placed sound’ work from a traditional music theory perspective and in doing so reveal some structural principals from ‘fixed’ musical forms can also support the composition of contemporary locative music experiences

Improvisation Pedagogy: An Epistemological Perspective of the 4‘E’ Model within Digital Musical Instruments

Recent years have witnessed the appearance of many new digital musical instruments (DMIs) and other interfaces for musical expression (NIME). This paper highlights a well-established music educational background theory that we believe may help DMI developers and users better understand DMIs in the context of music cognition and education. From an epistemological perspective, we present the paradigm of enactive music cognition related to improvisation in the context of the skills and needs of 21st century music learners. We hope this can lead to a deeper insertion of DMIs into music education, as well as to new DMIs to be ideated, prototyped and developed within these concepts and theories in mind. We specifically address the theory generally known as the 4E model of cognition (embodied, embedded, extended and enactive) within DMIs. The concept of autopoiesis is also described. Finally, we present some concrete cases of DMIs and NIMEs, and we describe how the experience of musical improvisation with them may be seen through the prism of such theories

Interactive Sound in Performance Ecologies: Studying Connections among Actors and Artifacts

This thesis’s primary goal is to investigate performance ecologies, that is the compound of humans, artifacts and environmental elements that contribute to the result of a per- formance. In particular, this thesis focuses on designing new interactive technologies for sound and music. The goal of this thesis leads to the following Research Questions (RQs): • RQ1 How can the design of interactive sonic artifacts support a joint expression across different actors (composers, choreographers, and performers, musicians, and dancers) in a given performance ecology? • RQ2 How does each different actor influence the design of different artifacts, and what impact does this have on the overall artwork? • RQ3 How do the different actors in the same ecology interact, and appropriate an interactive artifact? To reply to these questions, a new framework named ARCAA has been created. In this framework, all the Actors of a given ecology are connected to all the Artifacts throughout three layers: Role, Context and Activity. This framework is then applied to one systematic literature review, two case studies on music performance and one case study in dance performance. The studies help to better understand the shaded roles of composers, per- formers, instrumentalists, dancers, and choreographers, which is relevant to better design interactive technologies for performances. Finally, this thesis proposes a new reflection on the blurred distinction between composing and designing a new instrument in a context that involves a multitude of actors. Overall, this work introduces the following contributions to the field of interaction design applied to music technology: 1) ARCAA, a framework to analyse the set of inter- connected relationship in interactive (music) performances, validated through 2 music studies, 1 dance study and 1 systematic literature analysis; 2) Recommendations for de- signing music interactive system for performance (music or dance), accounting for the needs of the various actors and for the overlapping on music composition and design of in- teractive technology; 3) A taxonomy of how scores have shaped performance ecologies in NIME, based on a systematic analysis of the literature on score in the NIME proceedings; 4) Proposal of a methodological approach combining autobiographical and idiographical design approaches in interactive performances.O objetivo principal desta tese é investigar as ecologias performativas, conjunto formado pelos participantes humanos, artefatos e elementos ambientais que contribuem para o resultado de uma performance. Em particular, esta tese foca-se na conceção de novas tecnologias interativas para som e música. O objetivo desta tese originou as seguintes questões de investigação (Research Questions RQs): • RQ1 Como o design de artefatos sonoros interativos pode apoiar a expressão con- junta entre diferentes atores (compositores, coreógrafos e performers, músicos e dançarinos) numa determinada ecologia performativa? • RQ2 Como cada ator influencia o design de diferentes artefatos e que impacto isso tem no trabalho artístico global? • RQ3 Como os diferentes atores de uma mesma ecologia interagem e se apropriam de um artefato interativo? Para responder a essas perguntas, foi criado uma nova framework chamada ARCAA. Nesta framework, todos os atores (Actores) de uma dada ecologia estão conectados a todos os artefatos (Artefacts) através de três camadas: Role, Context e Activity. Esta framework foi então aplicada a uma revisão sistemática da literatura, a dois estudos de caso sobre performance musical e a um estudo de caso em performance de dança. Estes estudos aju- daram a comprender melhor os papéis desempenhados pelos compositores, intérpretes, instrumentistas, dançarinos e coreógrafos, o que é relevante para melhor projetar as tec- nologias interativas para performances. Por fim, esta tese propõe uma nova reflexão sobre a distinção entre compor e projetar um novo instrumento num contexto que envolve uma multiplicidade de atores. Este trabalho apresenta as seguintes contribuições principais para o campo do design de interação aplicado à tecnologia musical: 1) ARCAA, uma framework para analisar o conjunto de relações interconectadas em performances interativas, validado através de dois estudos de caso relacionados com a música, um estudo de caso relacionado com a dança e uma análise sistemática da literatura; 2) Recomendações para o design de sistemas interativos musicais para performance (música ou dança), tendo em conta as necessidades dos vários atores e a sobreposição entre a composição musical e o design de tecnologia interactiva; 3) Uma taxonomia sobre como as partituras musicais moldaram as ecologias performativas no NIME, com base numa análise sistemática da literatura dos artigos apresentados e publicados nestas conferência; 4) Proposta de uma aborda- gem metodológica combinando abordagens de design autobiográfico e idiográfico em performances interativas

Artefact, Participant and Interaction in Auditory Experiences

Presented at the 27th International Conference on Auditory Display (ICAD 2022) 24-27 June 2022, Virtual conference.The act of sound perception and its subjective dimensions, from physical to psychoacoustics, from semantic to affective, carry an inherent challenge for the conception and evaluation of every audio-based artefact. Starting from a previous framework of evaluation approaches, we seek to deconstruct the configuring elements of these processes, searching for theoretical foundations informing Sound Design and possible applications for Auditory Displays. This work is a first step into identifying a body of knowledge on the listener’s experience, how the act of listening takes place and how the sequence of listening actions can evolve as forms of dialogue, creating dialogical spaces for making sense of auditory information. With this work, practitioners can gain new insights into how existing techniques for creating auditory artefacts can be configured and transformed into new, alternative approaches

Design and Application of the BiVib Audio-Tactile Piano Sample Library

A library of piano samples composed of binaural recordings and keyboard vibrations has been built, with the aim of sharing accurate data that in recent years have successfully advanced the knowledge on several aspects about the musical keyboard and its multimodal feedback to the performer. All samples were recorded using calibrated measurement equipment on two Yamaha Disklavier pianos, one grand and one upright model. This paper documents the sample acquisition procedure, with related calibration data. Then, for sound and vibration analysis, it is shown how physical quantities such as sound intensity and vibration acceleration can be inferred from the recorded samples. Finally, the paper describes how the samples can be used to correctly reproduce binaural sound and keyboard vibrations. The library has potential to support experimental research about the psycho-physical, cognitive and experiential effects caused by the keyboard’s multimodal feedback in musicians and other users, or, outside the laboratory, to enable an immersive personal piano performance

Being sound: FLOSS, flow and event in the composition and ensemble performance of free open computer music

This commentary describes my recent approach to writing compositions for the ensemble performance of computer music. Drawing on experimental music and improvisation, I contend that such music is best considered in terms of people’s situated and relational interplay. The compositional and performative question that permeates this thesis is ‘what can we do, in this time and space, with these tools available to us?’. As themes of equality and egalitarian access underpin this work throughout, I highlight my engagement with Free Libre Open Source Software (FLOSS) ideology and community, reflecting on how this achieves my aims. I describe my writing of text score compositions, making use of the term bounded improvisation, whose purposeful requirements for indeterminate realisation extends most current computer-based performance practice. Though no single strand of this research is perhaps unusual by itself, such an assemblage as that outlined above (incorporating composition, computer coding and ensemble performance practice) is, when allied to an understanding of electronic and computer music praxis, currently an underdeveloped approach. Such an approach I have thus chosen to term free open computer music. I incorporate two further pre-existing conceptual formulations to present a framework for constructing, reflecting on, and developing my work in this field. Firstly flow or 'immersed experience' is useful to explicate difficult to capture aspects of instrumental engagement and ensemble performance. Secondly, this portfolio of scores aims to produce well-constructed situations, facilitating spaces of flow which contain within their environments the opportunity for an event to take place. I present the outcomes of my practice as place-forming tactics that catalyse something to do, but not what to do, in performative spaces such as those described above. Such intentions define my aims for composition. These theoretical concerns, together with an allied consideration of the underpinning themes highlighted above, is a useful framework for refection and evaluation of this work

La hipernormalización tecnológica de la creatividad: propuesta de evaluación del aprendizaje creativo en las interacciones tecnológicas musicales

[ES] El paradigma tecnosimbólico y positivista de la cuarta revolución industrial hipernormaliza progresivamente nuestras diversas formas de creatividad. Su estandarización y automatización afectan a ciencia, educación y salud a través del uso y la apropiación de la tecnología. La dialéctica del dataísmo, la conectividad, la inteligencia artificial, el aprendizaje automático o el pensamiento computacional promueve una onto-epistemología transhumanista y tecnócrata que acapara el debate pedagógico de la sociedad globalizada. Actualmente modelos computarizados intentan emular creatividad, aprendizaje o expresividad comunicativa, generando disonancias cognitivas producidas por la normalización y la estandarización sistémica connatural a la tecnología digital. Analizando este contexto e integrando los hallazgos y aportes alternativos de la educación y la terapia musical, esta investigación propone un modelo evaluativo alternativo respetuoso con el bienestar y salud de cada usuario, aportando información crucial para una optimización de la tecnología creativa y expresiva más humanizada, singular e individualizada, facilitando así un adecuado aprendizaje creativo e intuitivo dentro de un proceso creativo y expresivo musical. La investigación presente plantea el The Individualized Music Therapy Assessment Profile (IMTAP), como base metodológica evaluativa que incorpora, dentro de su propio marco, las formas metodológicas del rizoanálisis junto a los planteamientos de las Alfabetizaciones Múltiples (Kalantzis et al., 2020; Masny, 2014), integrando el trabajo de evaluación experimental que aportan Wanderley y Mackay (2019). Los resultados concluyen que es posible evaluar las interacciones entre humanos e interfaces tecnológicas musicales a través de la experiencia interactiva, singular, proactiva, creativa y expresiva de cada usuario tecnológico, integrando aspectos conjuntos de la educación, la salud y el bienestar de los usuarios dentro de los procesos de diseño de las tecnologías creativas y expresivas musicales. Además se aconseja la participación de arteterapeutas y musicoterapeutas profesionales dentro de los equipos de diseño de las interacciones y las interfaces cognitivas, creativas y expresivas, más allá del desarrollo de los diseños tecnológicos en etapas tempranas, y no sólo dentro de los marcos en los que se configuran las tecnologías musicales, sino durante los diferentes procesos iterativos que deben evaluar y optimizar las tecnologías ideadas en beneficio de la educación, la salud y el bienestar de cada usuario. [EN] The technosymbolic and positivist paradigm of the fourth industrial revolution progressively hypernormalizes our various forms of creativity. Its standardization and automation affect science, education and health through the use and appropriation of technology. The dialectic of dataism, connectivity, artificial intelligence, automatic learning or computational thinking promotes a transhumanist and technocratic onto-epistemology that monopolizes the pedagogical debate of the globalized society. Currently computerized models try to emulate creativity, learning or communicative expressiveness, generating cognitive dissonances produced by normalization and systemic standardization natural to digital technology. Analyzing this context and integrating the findings and alternative contributions of music education and therapy, this research proposes an alternative evaluative model respectful of the well-being and health of each user, providing crucial information for a more humanized optimization of creative and expressive technology, unique and individualized, thus facilitating adequate creative and intuitive learning within a creative and expressive musical process. The present investigation proposes The Individualized Music Therapy Assessment Profile (IMTAP), as an evaluative methodological basis that incorporates, within its own framework, the methodological forms of rhizoanalysis together with the approaches of Multiple Literacies (Kalantzis et al., 2020; Masny, 2014), integrating the experimental evaluation work provided by Wanderley and Mackay (2019). The results conclude that it is possible to evaluate the interactions between humans and musical technological interfaces through the interactive, singular, proactive, creative and expressive experience of each technological user, integrating joint aspects of education, health and well-being of users within of the design processes of creative and expressive musical technologies. In addition, the participation of professional art therapists and music therapists is advised within the design teams of cognitive, creative and expressive interactions and interfaces, beyond the development of technological designs in early stages, and not only within the frameworks in which musical technologies are configured, but during the different iterative processes that must evaluate and optimize the technologies designed for the benefit of education, health and well-being of each user