Whole body interaction in abstract domains

Whole Body Interaction appears to be a good fit of interaction style for some categories of application domain, such as the motion capture of gestures for computer games and virtual physical sports. However, the suitability of whole body interaction for more abstract application domains is less apparent, and the creation of appropriate whole body interaction designs for complex abstract areas such as mathematics, programming and musical harmony remains challenging. We argue, illustrated by a detailed case study, that conceptual metaphor theory and sensory motor contingency theory offer analytic and synthetic tools whereby whole body interaction can in principle be applied usefully to arbitrary abstract application domains. We present the case study of a whole body interaction system for a highly abstract application area, tonal harmony in music. We demonstrate ways in which whole body interaction offers strong affordances for action and insight in this domain when appropriate conceptual metaphors are harnessed in the design. We outline how this approach can be applied to abstract domains in general, and discuss its limitations.Published versio

Whole body interaction in abstract domains

Whole Body Interaction appears to be a good fit of interaction style for some categories of application domain, such as the motion capture of gestures for computer games and virtual physical sports. However, the suitability of whole body interaction for more abstract application domains is less apparent, and the creation of appropriate whole body interaction designs for complex abstract areas such as mathematics, programming and musical harmony remains challenging. We argue, illustrated by a detailed case study, that conceptual metaphor theory and sensory motor contingency theory offer analytic and synthetic tools whereby whole body interaction can in principle be applied usefully to arbitrary abstract application domains. We present the case study of a whole body interaction system for a highly abstract application area, tonal harmony in music. We demonstrate ways in which whole body interaction offers strong affordances for action and insight in this domain when appropriate conceptual metaphors are harnessed in the design. We outline how this approach can be applied to abstract domains in general, and discuss its limitations

Music and HCI

Music is an evolutionarily deep-rooted, abstract, real-time, complex, non-verbal, social activity. Consequently, interaction design in music can be a valuable source of challenges and new ideas for HCI. This workshop will reflect on the latest research in Music and HCI (Music Interaction for short), with the aim of strengthening the dialogue between the Music Interaction community and the wider HCI community. We will explore recent ideas from Music Interaction that may contribute new perspectives to general HCI practice, and conversely, recent HCI research in non-musical domains with implications for Music Interaction. We will also identify any concerns of Music Interaction that may require unique approaches. Contributors engaged in research in any area of Music Interaction or HCI who would like to contribute to a sustained widening of the dialogue between the distinctive concerns of the Music Interaction community and the wider HCI community will be welcome

Song walker harmony space: embodied interaction design for complex musical skills

Tonal harmony is widely considered to be the most technical and complex part of music theory. Consequently harmonic skills can be hard to acquire. Furthermore, experience of the flexible manipulation of harmony in real time generally requires the ability to play an instrument. Even for those with instrumental skills, it can be difficult to gain clear insight into harmonic abstractions. The above state of affairs gives rise to substantial barriers not only for beginners but also for many experienced musicians. To address these problems, Harmony Space (Holland et al. 2009) is an interactive digital music system designed to give insight into a wide range of musical tasks in tonal harmony, ranging from performance and composition to analysis. Harmony Space employs a principled set of spatial mappings to offer fluid, precise, intuitive control of harmony. These mappings give rise to sensory-motor and music-theoretic affordances that are hard to obtain in any other way. As a result, harmonic abstractions are rendered amenable to concrete, visible control by simple spatial manipulation. In the language of conceptual metaphor theory, many relationships in tonal harmony become accessible to rapid, universal, low-level, robust human inference mechanisms using image schemata such as containment, contact, centre-periphery, and source-path-goal. This process is more rapid, and imposes far less cognitive load, than slow, abstract symbolic reasoning. Using the above principles, several versions of Harmony Space have been de-signed to exploit specific interaction styles for different purposes. We note some key variants, such as the desktop version (Holland 1994), the camera tracked version (Holland et al. 2009), while focusing principally on the most recent version, Song Walker (Holland et al. 2011), which employs whole body interaction. Preliminary results from a study of the Song Walker system are outlined, in which both beginners and expert musicians undertook a range of musical tasks involving the performance, composition and analysis of music. Finally, we offer a discussion of the limitations of the current system, and outline directions for future wor

Sound Spheres: A Design Study of the Articulacy of a Non-Contact Finger Tracking Virtual Musical Instrument

A key challenge in the design of Virtual Musical instruments (VMIs) is finding expressive, playable, learnable mappings from gesture to sound that progressively reward practice by performers. Designing such mappings can be particularly demanding in the case of non-contact musical instruments, where physical cues can be scarce. Unaided intuition works well for many instrument designers, but others may find design and evaluation heuristics useful when creating new VMIs. In this paper we gather existing criteria from the literature to assemble a simple set of design and evaluation heuristics that we dub articulacy. This paper presents a design case study in which an expressive non-contact finger-tracking VMI, Sound Spheres, is designed and evaluated with the support of the articulacy heuristics. The case study explores the extent to which articulacy usefully informs the design of a non-contact VMI, and we reflect on the usefulness or otherwise of heuristic approaches in this context

Exploring whole body interaction and design for museums

Museums increasingly use digital technology to enhance exhibition experiences for families, notably in relation to physically mediated installations for young children through natural user interfaces. Yet little is known about how families and children engage with such installations and the kinds of interactive experiences they engender in museum spaces. This paper addresses a pressing need for research to adopt an analytical focus on the body during such digitally mediated interactions in order to understand how bodily interaction contributes to meaning making in the museum context. It reports an observation study of families and children interacting with a whole-body interface (using Kinect) in the context of an installation in a museum exhibit on rare Chinese paintings. The study shows how the installation design engenders particular forms of bodily interaction, collaboration and meaning making. It also contributes design insights into whole-body interaction installations in museums and public spaces

MAMIC goes live: a music programming system for non-specialist delivery

The computing curriculum in England has shifted from “software training” to a model where children learn to code as a way of understanding underlying principles. This has created challenges for primary school teaching practitioners, many of whom require upskilling. The Music And Math In Collaboration (MAMIC) project addresses these issues via a custom library for the Pure Data (Pd) visual programming environment that interconnects key musical, mathematical and coding concepts within a unified environment that is able to be delivered by non-specialist teachers. Initial findings from deployment “in the wild” (i.e. in situ) are presented and future work is discussed

Exploring whole body interaction and design for museums

Museums increasingly use digital technology to enhance exhibition experiences for families, notably in relation to physically mediated installations for young children through natural user interfaces. Yet little is known about how families and children engage with such installations and the kinds of interactive experiences they engender in museum spaces. This paper addresses a pressing need for research to adopt an analytical focus on the body during such digitally mediated interactions in order to understand how bodily interaction contributes to meaning making in the museum context. It reports an observation study of families and children interacting with a whole-body interface (using Kinect) in the context of an installation in a museum exhibit on rare Chinese paintings. The study shows how the installation design engenders particular forms of bodily interaction, collaboration and meaning making. It also contributes design insights into whole-body interaction installations in museums and public spaces

Music interaction: understanding music and human-computer interaction

We introduce and review recent research in Music and Human Computer Interaction, also known as Music Interaction. After a general overview of the discipline, we analyse the themes and issues raised by the fifteen chapters of this book, each of which presents recent research in this field. The bulk of this chapter is organised as an FAQ. This enables some FAQs to focus on cross cutting issues that appear in multiple chapters, and some chapters to feature in multiple FAQs. Broad topics include: the scope of research in Music Interaction; the role of HCI in Music Interaction; and conversely, the role of Music Interaction in HCI. High-level themes include embodied cognition, spatial cognition, evolutionary interaction, gesture, formal language, affective interaction, and methodologies from social science. Musical activities of interest include performance, composition, analysis, collaborative music making, and human and machine improvisation. Specific issues include: whether Music Interaction should be easy; what can be learned from the experience of being “in the groove”, and what can be learned from the deep commitment of musical amateurs. Broader issues include: what Music Interaction can offer traditional instruments and traditional musical activities; what relevance it has for non-musical domains; and ways in which Music Interaction can enable entirely new musical activities possible