Interaction and the Art of User-Centered Digital Musical Instrument Design

This thesis documents the formulation of a research-based practice in multimedia art, technology and digital musical instrument design. The primary goal of my research was to investigate the principles and methodologies involved in the structural design of new interactive digital musical instruments aimed at performance by members of the general public, and to identify ways that the design process could be optimized to increase user adoption of these new instruments. The research was performed over three years and moved between studies at the University of Maine, internships in New York, and specialized research at the Input Devices and Music Interaction Laboratory at McGill University. My work is presented in two sections. The first covers early studies in user interaction and exploratory works in web and visual design, sound art, installation, and music performance. While not specifically tied to the research topic of user adoption of digital musical instruments, this work serves as the conceptual and technical background for the dedicated work to follow. The second section is dedicated to focused research on digital musical instrument design through two major projects carried out as a Graduate Research Trainee at McGill University. The first was the design and prototype of the Noisebox, a new digital musical instrument. The purpose of this project was to learn the various stages of instrument design through practical application. A working prototype has been presented and tested, and a second version is currently being built. The second project was a user study that surveyed musicians about digital musical instrument use. It asked questions about background, instrument choice, music styles played, and experiences with and attitudes towards new digital musical instruments. Based on the results of the two research projects, a model of digital musical instrument design is proposed that adopts a user-centered focus, soliciting user input and feedback throughout the design process from conception to final testing. This approach aims to narrow the gap between conceptual design of new instruments and technologies and the actual musicians who would use them

The Seaboard: discreteness and continuity in musical interface design

The production of acoustic music bridges two senses—touch and hearing—by connecting physical movements, gestures, and tactile interactions with the creation of sound. Mastery of acoustic music depends on the development and refinement of muscle memory and ear training in concert. This process leads to a capacity for great depth of expression even though the actual timbral palette of each given acoustic instrument is relatively limited. By contrast, modern modes of music creation involving recorded music and digital sound manipulation sacrifice this immediate bridge and substitute more abstract processes that enable sonic possibilities extending far beyond the acoustic palette. Mastery in abstract approaches to music making doesn’t necessarily rely on muscle memory or ear training, as many key processes do not need to happen in realtime. This freedom from the limits of time and practiced physical manipulation radically increases the range of achievable sounds, rhythms and effects, but sometimes results in a loss of subtlety of expressiveness. This practice-based PhD asks whether it is possible, and if so how, to achieve an integration of relevant sensor technologies, design concepts, and formation techniques to create a new kind of musical instrument and sound creation tool that bridges this gap with a satisfying result for musicians and composers. In other words, can one create new, multi-dimensional interfaces which provide more effective ways to control the expressive capabilities of digital music creation in real-time? In particular, can one build on the intuitive, logical, and well-known layout of the piano keyboard to create a new instrument that more fully enables both continuous and discrete approaches to music making? My research practice proposes a new musical instrument called the Seaboard, documents its invention, development, design, and refinement, and evaluates the extent to which it positively answers the above question. The Seaboard is a reinterpretation of the piano keyboard as a soft, continuous wavelike surface that places polyphonic pitch bend, vibrato and continuous touch right at the musician’s fingertips. The addition of new realtime parameters to a familiar layout means it combines the intuitiveness of the traditional instrument with some of the versatility of digital technology. Designing and prototyping the Seaboard to the point of successfully proving that a new synthesis between acoustic techniques and digital technologies is possible is shown to require significant coordination and integration of a range of technical disciplines. The research approach has been to build and refine a series of prototypes that successively grapple with the integration of these elements, whilst rigorously documenting the design issues, engineering challenges, and ultimate decisions that determine whether an intervention in the field of musical instrumentation is fruitful

3D Composer: A Software for Micro-composition

The aim of this compositional research project is to find new paradigms of expression and representation of musical information, supported by technology. This may further our understanding of how artistic intention materialises during the production of a musical work. A further aim is to create a software device, which will allow the user to generate, analyse and manipulate abstract musical information within a multi-dimensional environment. The main intent of this software and composition portfolio is to examine the process involved during the development of a compositional tool to verify how transformations applied to the conceptualisation of musical abstraction will affect musical outcome, and demonstrate how this transformational process would be useful in a creative context. This thesis suggests a reflection upon various technological and conceptual aspects within a dynamic multimedia framework. The discussion situates the artistic work of a composer within the technological sphere, and investigates the role of technology and its influences during the creative process. Notions of space are relocated in the scope of a personal compositional direction in order to develop a new framework for musical creation. The author establishes theoretical ramifications and suggests a definition for micro-composition. The main aspect focuses on the ability to establish a direct conceptual link between visual elements and their correlated musical output, ultimately leading to the design of a software called 3D-Composer, a tool for the visualisation of musical information as a means to assist composers to create works within a new methodological and conceptual realm. Of particular importance is the ability to transform musical structures in three-dimensional space, based on the geometric properties of micro-composition. The compositions Six Electroacoustic Studies and Dada 2009 display the use of the software. The formalisation process was derived from a transposition of influences of the early twentieth century avant-garde period, to a contemporary digital studio environment utilising new media and computer technologies for musical expression

Landscapes of the invisible: sounds, cosmologies and poetics of space

In this PhD by Publication I revisit and contextualize art works and essays I have collaboratively created under the name Flow Motion between 2004-13, in order to generate new insights on the contributions they have made to diverse and emerging fields of contemporary arts practice/research, including digital, virtual, sonic and interdisciplinary art. The works discussed comprise the digital multimedia installation and sound art performance Astro Black Morphologies/Astro Dub Morphologies (2004-5), the sound installation and performance Invisible (2006-7), the web art archive and performance presentation project promised lands (2008-10), and two related texts, Astro Black Morphologies: Music and Science Lovers (2004) and Music and Migration (2013). I show how these works map new thematic constellations around questions of space and diaspora, music and cosmology, invisibility and spectrality, the body and perception. I also show how the works generate new connections between and across contemporary avant-garde, experimental and popular music, and visual art and cinema traditions. I describe the methodological design, approaches and processes through which the works were produced, with an emphasis on transversality, deconstruction and contemporary black music forms as key tools in my collaborative artistic and textual practice. I discuss how, through the development of methods of data translation and transformation, and distinctive visual approaches for the re-elaboration of archival material, the works produced multiple readings of scientific narratives, digital X-ray data derived from astronomical research on black holes and dark energy, and musical, photographic and textual material related to historical and contemporary accounts of migration. I also elaborate on the relation between difference and repetition, the concepts of multiplicity and translation, and the processes of collective creation which characterize my/Flow Motion’s work. The art works and essays I engage with in this commentary produce an idea of contemporary art as the result of a fluid, open and mutating assemblage of diverse and hybrid methods and mediums, and as an embodiment of a cross-cultural, transversal and transdisciplinary knowledge shaped by research, process, creative dialogues, collaborative practice and collective signature

Music with Plants: Cultivating Bonds Between Grade-Schoolers and Nature Through Sound Design

Uma necessidade crescente para a consciencialização ambiental requer estratégias sensíveis que nos ajudem a repensar a forma como as interacções transformam o nosso habitat. As cidades modernas enfrentam agora o desaparecimento progressivo dos espaços verdes, do qual resulta a diminuição de oportunidades para as crianças brincarem (Costa & Costa, 2012; Ruggles, 2017). Esta dissertação propõe uma abordagem musical para nutrir os laços entre estudantes do ensino primário e a natureza.Como as plantas e os humanos experiência a vida em tempos muito diferentes, os últimos podem percepcionar as plantas como seres silenciosos e imóveis (Ruggles, 2017; Mancuso & Viola, 2016). A sua capacidade para sentir os elementos circundantes, para os quais não estamos despertos, tornam as plantas aliadas inestimáveis no caminho para uma relação mais consciente com o planeta que partilhamos. Campos de estudo como a ecologia acústica e a bioacústica das plantas começaram recentemente a florescer, inspirando artistas sonoros a explorar o mundo vegetal, combinando ciência e as artes (Patrão, Harvard University: Ex-centric Music Studies Conference, 2018).Na educação, por meio da brincadeira, as plantas poderão ter um papel significativo ao estimularem os sentidos e a ligação entre crianças que habitam os meios urbanos cada vez mais artificiais e o mundo dos seres vivos que as rodeiam (Costa & Costa, 2012).Este trabalho sugere que a música tem o potencial para intermediar estas interacções, proporcionando experiência significantes aos jovens participantes, trazendo para o mesmo espaço tecnologias digitais, música e plantas. Sendo que a música em si poderá ter a capacidade de fortalecer laços entre pessoas e promover e manter coesão de grupo. A música é "a cola social que amplia a cooperação e fortalece os sentimentos de união" (Honing, 2018).A growing need for environmental awareness requires sensitive strategies to help us rethink the way interactions transform our habitat. Modern cities are now faced with the progressive vanishing of green spaces, resulting in fewer opportunities for children to play (Costa & Costa, 2012; Ruggles, 2017). This work proposes a musical approach to nourishing the bonds between elementary-school students and nature. Since plants and humans experience life in very different time frames, humans might mistakenly perceive plants as quiet and motionless living-beings (Ruggles, 2017; Mancuso & Viola, 2016). The ability to sense elements of their surroundings to which we are unaware of, makes plants invaluable allies towards a more conscious relationship with the planet we share. Fields like acoustic ecology and plant bioacoustics have recently started to bloom, inspiring sound artists to discover the vegetal world combining science and the arts (Patrão, Harvard University: Ex-centric Music Studies Conference, 2018).In education, by the means of play, plants might perform a significant role in stimulating the senses and the connection between children inhabiting increasingly artificial cities and the world of living things that surrounds them (Costa & Costa, 2012). This dissertation suggests that music has the potential to intermediate these interactions and provide meaningful experiences to the young participants by bringing to common ground digital technology, music and plants. Being that music itself is thought to have capability to strengthen bonds between people, promote and maintain group cohesion. It is "the social glue that enhances cooperation and strengthens feelings of unity" (Honing, 2018)

The Visualization and Representation of Electroacoustic Music

In Chapters 1 and 2 there are definitions and a review of electroacoustic music, and then visualization generally and as applied to music. Chapter 3 is a review of specific and relevant literature as regards to the visualization of electroacoustic music. Chapter 4 introduces the concepts of imagining as opposed to discovering new sound, and what is important to this research about these terms; in addition what is meant and indicated by them. Chapter 5 deals with the responses that composers currently working have made to the enquiry concerning visualization. In this chapter these responses are dealt with as case studies. In a similar way, Chapter 6 looks at some examples of historical work in electroacoustic music, again as case studies. In Chapter 7 a taxonomical structure for the use of visualization in electroacoustic composition is established and derived from the case study results. Chapter 8 looks at relevant examples of software and how they offer visualization case studies. Chapter 9 looks at the place of the archive in various stages of the compositional process. Chapter 10 investigates the problems of visualizing musical timbre as possible evidence for future strategies. Chapter 11 offers some conclusions and implications as to the main research questions, as well as more specific outlines of potential strategies for the visualization of electroacoustic music

How do electronic musicians make their music? : creative practice through informal learning resources

The paper will explore the creative practices of professional electronic musicians through their interviews published in consumer materials, commonly perceived as informal learning resources. Drawing on data gathered from music technology and music culture-related publications including magazines, newspapers, online sources, video, and radio, the study will consolidate 154 existing interviews, given over a seventeen-year period. The aim of this study is not to provide a complete illustration of electronic music production, but rather offer points of significance that hold meaning in the context of creative practice