Improving Data-driven Design and Exploration of Digital Musical Instruments

We present Gesture Mapper, an application for digital musical instrument designers to rapidly prototype mappings from performer gestures to sound synthesis parameters. Prior work [2] has shown that using interactive supervised learning to generate mappings from user-generated examples can be more efficient and effective than users writing mapping functions in code. In this work, we explore new ways to improve on data-driven design of interactive systems, specifically by proposing new mechanisms for rapid exploration and comparison of multiple alternative mappings. We present a conceptual structure for interactive mappings, a basic framework for generating mappings from more diverse types of user-specified constraints than are supported by supervised learning, and the new Gesture Mapper user interface for mapping exploration and comparison

Multiple Media Interfaces for Music Therapy

This article describes interfaces (and the supporting technological infrastructure) to create audiovisual instruments for use in music therapy. In considering how the multidimensional nature of sound requires multidimensional input control, we propose a model to help designers manage the complex mapping between input devices and multiple media software. We also itemize a research agenda

PIWeCS: enhancing human/machine agency in an interactive composition system

This paper focuses on the infrastructure and aesthetic approach used in PIWeCS: a Public Space Interactive Web-based Composition System. The concern was to increase the sense of dialogue between human and machine agency in an interactive work by adapting Paine's (2002) notion of a conversational model of interaction as a ‘complex system’. The machine implementation of PIWeCS is achieved through integrating intelligent agent programming with MAX/MSP. Human input is through a web infrastructure. The conversation is initiated and continued by participants through arrangements and composition based on short performed samples of traditional New Zealand Maori instruments. The system allows the extension of a composition through the electroacoustic manipulation of the source material

Affordance of vibrational excitation for music composition and performance

Mechanical vibrations have typically been used in the performance domain within feedback systems to inform musicians of system states or as communication channels between performers. In this paper, we propose the addi- tional taxonomic category of vibrational excitation of mu- sical instruments for sound generation. To explore the va- riety of possibilities associated with this extended taxon- omy, we present the Oktopus, a multi-purpose wireless sys- tem capable of motorised vibrational excitation. The sys- tem can receive up to eight inputs and generates vibrations as outputs through eight motors that can be positioned ac- cordingly to produce a wide range of sounds from an ex- cited instrument. We demonstrate the usefulness of the proposed system and extended taxonomy through the de- velopment and performance of Live Mechanics, a compo- sition for piano and interactive electronics

Exploring the Digital Music Instrument Trombosonic with Extreme Users and at a Participatory Performance

We introduce the “Trombosonic” as a new digital music instrument inspired by the slide trombone. An ultrasonic sensor combined with a red laser allows the performer to play the instrument using similar movements to playing a trombone to change the pitch, despite the absence of a physical slider, by moving one hand back and forth. Additional sensors enhance the potential for musical expression by movement of the whole interface and by using the breath. We identify and discuss a variety of design issues arising from the Trombosonic. Due to its compact size and the lack of a slider, the Trombosonic can be played in many different ways. In order to explore varied potential uses of the Trombosonic, we carried out a series of informal evaluations. These included experts in new musical instruments, an older user, a younger user, an interaction design expert, and the audience at an experimental concert with audience participation. Future work is also discussed. Further technical development might include a built-in microphone to use the human voice and an expansion of the synthesiser’s features

Designing and evaluating the usability of a machine learning API for rapid prototyping music technology

To better support creative software developers and music technologists' needs, and to empower them as machine learning users and innovators, the usability of and developer experience with machine learning tools must be considered and better understood. We review background research on the design and evaluation of application programming interfaces (APIs), with a focus on the domain of machine learning for music technology software development. We present the design rationale for the RAPID-MIX API, an easy-to-use API for rapid prototyping with interactive machine learning, and a usability evaluation study with software developers of music technology. A cognitive dimensions questionnaire was designed and delivered to a group of 12 participants who used the RAPID-MIX API in their software projects, including people who developed systems for personal use and professionals developing software products for music and creative technology companies. The results from the questionnaire indicate that participants found the RAPID-MIX API a machine learning API which is easy to learn and use, fun, and good for rapid prototyping with interactive machine learning. Based on these findings, we present an analysis and characterization of the RAPID-MIX API based on the cognitive dimensions framework, and discuss its design trade-offs and usability issues. We use these insights and our design experience to provide design recommendations for ML APIs for rapid prototyping of music technology. We conclude with a summary of the main insights, a discussion of the merits and challenges of the application of the CDs framework to the evaluation of machine learning APIs, and directions to future work which our research deems valuable

Data Driven Analysis of Tiny Touchscreen Performance with MicroJam

The widespread adoption of mobile devices, such as smartphones and tablets, has made touchscreens a common interface for musical performance. New mobile musical instruments have been designed that embrace collaborative creation and that explore the affordances of mobile devices, as well as their constraints. While these have been investigated from design and user experience perspectives, there is little examination of the performers' musical outputs. In this work, we introduce a constrained touchscreen performance app, MicroJam, designed to enable collaboration between performers, and engage in a novel data-driven analysis of more than 1600 performances using the app. MicroJam constrains performances to five seconds, and emphasises frequent and casual music making through a social media-inspired interface. Performers collaborate by replying to performances, adding new musical layers that are played back at the same time. Our analysis shows that users tend to focus on the centre and diagonals of the touchscreen area, and tend to swirl or swipe rather than tap. We also observe that while long swipes dominate the visual appearance of performances, the majority of interactions are short with limited expressive possibilities. Our findings are summarised into a set of design recommendations for MicroJam and other touchscreen apps for social musical interaction

Body as Instrument – Performing with Gestural Interfaces

This paper explores the challenge of achieving nuanced control and physical engagement with gestural interfaces in performance. Performances with a prototype gestural performance system, Gestate, provide the basis for insights into the application of gestural systems in live contexts. These reflections stem from a performer's perspective, summarising the experience of prototyping and performing with augmented instruments that extend vocal or instrumental technique through gestural control. Successful implementation of rapidly evolving gestural technologies in real-time performance calls for new approaches to performing and musicianship, centred on a growing understanding of the body's physical and creative potential. For musicians hoping to incorporate gestural control seamlessly into their performance practice, a balance of technical mastery and kinaesthetic awareness is needed to adapt existing approaches to their own purposes. Within non-tactile systems, visual feedback mechanisms can support this process by providing explicit visual cues that compensate for the absence of haptic feedback. Experience gained through prototyping and performance can yield a deeper understanding of the broader nature of gestural control and the way in which performers inhabit their own bodies.4 page(s