Tabletop Tangible Interfaces for Music Performance: Design and Evaluation

This thesis investigates a new generation of collaborative systems: tabletop tangible interfaces (TTIs) for music performance or musical tabletops. Musical tabletops are designed for professional musical performance, as well as for casual interaction in public settings. These systems support co-located collaboration, offered by a shared interface. However, we still know little about their challenges and opportunities for collaborative musical practice: in particular, how to best support beginners or experts or both. This thesis explores the nature of collaboration on TTIs for music performance between beginners, experts, or both. Empirical work was done in two stages: 1) an exploratory stage; and 2) an experimental stage. In the exploratory stage we studied the Reactable, a commercial musical tabletop designed for beginners and experts. In particular, we explored its use in two environments: a multi-session study with expert musicians in a casual lab setting; and a field study with casual visitors in a science centre. In the experimental stage we conducted a controlled experiment for mixed groups using a bespoke musical tabletop interface, SoundXY4. The design of this study was informed by the previous stage about a need to support better real-time awareness of the group activity (workspace awareness) in early interactions. For the three studies, groups musical improvisation was video-captured unobtrusively with the aim of understanding natural uses during group musical practice. Rich video data was carefully analysed focusing on the nature of social interaction and how workspace awareness was manifested. The findings suggest that musical tabletops can support peer learning during multiple sessions; fluid between-group social interaction in public settings; and a democratic and ecological approach to music performance. The findings also point to how workspace awareness can be enhanced in early interactions with TTIs using auditory feedback with ambisonics spatialisation. The thesis concludes with theoretical, methodological, and practical implications for future research in New Interfaces for Musical Expression (NIME), tabletop studies, and Human-Computer Interaction (HCI)

Collaborative music interaction on tabletops: an HCI approach

With the advent of tabletop interaction, collaborative activities are better supported than they are on single-user PCs because there exists a physical shareable space, and interaction with digital data is more embodied and social. In sound and music computing, collaborative music making has traditionally been done using interconnected networks, but using separated computers. Musical tabletops introduce opportunities of playing in collaboration through sharing physically the same musical interface. However, few tabletop musical interfaces exploit this collaborative potential (e.g. the Reactable). We are interested in looking into how collaboration can be fully supported by means of musical tabletops for music performance in contrast with more traditional settings. We are also looking at whether collective musical engagement can be enhanced by providing more suitable interfaces to collaboration. In HCI and software development, we find an iterative process approach of design and evaluation—where evaluation allows us to identify key issues that can be addressed in the next design iteration of the system. Using a similar iterative approach, we plan to design and evaluate some tabletop musical interfaces. The aim is to understand what design choices can enhance and enrich collaboration and collective musical engagement on these systems. In this paper, we explain the evaluation methodologies we have undertaken in three preliminary pilot studies, and the lessons we have learned. Initial findings indicate that evaluating tabletop musical interfaces is a complex endeavour which requires an approach as close as possible to a real context, with an interdisciplinary approach provided by interaction analysis techniques

Multi-touch interaction principles for collaborative real-time music activities: towards a pattern language

In this paper we give an analysis of the literature on a set of problems that can arise when undertaking the interaction design of multi-touch applications for collaborative real-time music activities, which are designed for multitouch technologies (e.g. smartphones, tablets, interactive tabletops, among others). Each problem is described, and a candidate design pattern (CDP) is suggested in the form of a short sentence and a diagram—an approach inspired by Christopher Alexander’s A Pattern Language. These solutions relate to the fundamental collaborative principles of democratic relationships, identities and collective interplay. We believe that this approach might disseminate forms of best design practice for collaborative music applications, in order to produce real-time musical systems which are collaborative and expressive

Factors in human recognition of timbre lexicons generated by data clustering

Since the development of sound recording technologies, the palette of sound timbres available for music creation was extended way beyond traditional musical instruments. The organization and categorization of timbre has been a common endeavor. The availability of large databases of sound clips provides an opportunity for obtaining datadriven timbre categorizations via content-based clustering. In this article we describe an experiment aimed at understanding what factors influence the process of learning a given clustering of sound samples. We clustered a large database of short sound clips, and analyzed the success of participants in assigning sounds to the “correct” clusters after listening to a few examples of each. The results of the experiment suggest a number of relevant factors related both to the strategies followed by users and to the quality measures of the clustering solution, which can guide the design of creative applications based on audio clip clustering

Hyperconnected Action Painting

This performance invites the audience to participate in an immersive experience using their mobile devices. The aim is at capturing their actions on a digital painting inspired by Jackson Pollock’s action painting technique. The audience is connected to a wireless network and a Web Audio application that recognizes a number of gestures through the mobile accelerometer sensor, which trigger different sounds. Gestures will be recognized and mapped to a digital canvas. A set of loudspeakers will complement the audience’s actions with ambient sounds. The performance explores audio spatialization using both loudspeakers and mobile phone speakers, that combined with the digital painting provides an immersive audiovisual experience. The final digital canvas will be available online as a memory of the performance

Loop-aware Audio Recording for the Web

Music loops are audio recordings used as basic building blocks in many types of music. The use of pre-recorded loops facilitates engagement into music creation to users regardless of their background in music theory. Using online loop databases also affords simple collaboration and exchange. Hence, music loops are particularly attractive for web audio applications. However, traditional musical audio recording typically relies on complex DAW software. Recording loops usually requires consideration of musical meter and tempo, and withstanding metronome sounds. In this paper, we propose loop-aware audio recording as a use case for web audio technologies. Our approach supports hands-free, low-stress recording of music loops in web- enabled devices. The system is able to detect repetitions in an incoming audio stream. Based on this information, it segments and ranks the repeated fragments, presenting the list to the user. We provide an example implementation, and evaluate the use of the different MIR libraries available in the web audio platform for the proposed task

Issues and techniques for collaborative music making on multi-touch surfaces

A range of systems exist for collaborative music making on multi-touch surfaces. Some of them have been highly successful, but currently there is no systematic way of designing them, to maximise collaboration for a particular user group. We are particularly interested in systems that will engage novices and experts. We designed a simple application in an initial attempt to clearly analyse some of the issues. Our application allows groups of users to express themselves in collaborative music making using pre-composed materials. User studies were video recorded and analysed using two techniques derived from Grounded Theory and Content Analysis. A questionnaire was also conducted and evaluated. Findings suggest that the application affords engaging interaction. Enhancements for collaborative music making on multi-touch surfaces are discussed. Finally, future work on the prototype is proposed to maximise engagement

Hex Player—a virtual musical controller

In this paper, we describe a playable musical interface for tablets and multi-touch tables. The interface is a generalized keyboard, inspired by the Thummer, and consists of an array of virtual buttons. On a generalized keyboard, any given interval always has the same shape (and therefore fingering); furthermore, the fingering is consistent over a broad range of tunings. Compared to a physical generalized keyboard, a virtual version has some advantages—notably, that the spatial location of the buttons can be transformed by shears and rotations, and their colouring can be changed to reflect their musical function in different scales. We exploit these flexibilities to facilitate the playing not just of conventional Western scales but also a wide variety of microtonal generalized diatonic scales known as moment of symmetry, or well-formed, scales. A user can choose such a scale, and the buttons are automatically arranged so their spatial height corresponds to their pitch, and buttons an octave apart are always vertically above each other. Furthermore, the most numerous scale steps run along rows, while buttons within the scale are light-coloured, and those outside are dark or removed. These features can aid beginners; for example, the chosen scale might be the diatonic, in which case the piano’s familiar white and black colouring of the seven diatonic and five chromatic notes is used, but only one scale fingering need ever be learned (unlike a piano where every key needs a different fingering). Alternatively, it can assist advanced composers and musicians seeking to explore the universe of unfamiliar microtonal scales

Towards a taxonomy for video analysis on collaborative musical tabletops

This position paper summarises some themes encountered when analysing video data in the context of music performance with interactive tabletops. It presents methodological approaches and coding schemes used for a set of experiments on musical tabletops and collaboration. Finally, it outlines an initial taxonomy based on the outcomes of the projects introduced, which can be used for video annotation of collaborative music interaction