Affordance of vibrational excitation for music composition and performance

Mechanical vibrations have typically been used in the performance domain within feedback systems to inform musicians of system states or as communication channels between performers. In this paper, we propose the addi- tional taxonomic category of vibrational excitation of mu- sical instruments for sound generation. To explore the va- riety of possibilities associated with this extended taxon- omy, we present the Oktopus, a multi-purpose wireless sys- tem capable of motorised vibrational excitation. The sys- tem can receive up to eight inputs and generates vibrations as outputs through eight motors that can be positioned ac- cordingly to produce a wide range of sounds from an ex- cited instrument. We demonstrate the usefulness of the proposed system and extended taxonomy through the de- velopment and performance of Live Mechanics, a compo- sition for piano and interactive electronics

A qualitative analysis of haptic feedback in music focused exercises new interfaces for musical expression

We present the findings of a pilot-study that analysed the role of haptic feedback in a musical context. To closely examine the role of haptics in Digital Musical Instrument (DMI) design an experiment was formulated to measure the users’ perception of device usability across four separate feedback stages: fully haptic (force and tactile combined), constant force only, vibrotactile only, and no feedback. The study was piloted over extended periods with the intention of exploring the application and integration of DMIs in real-world musical contexts. Applying a music orientated analysis of this type enabled the investigative process to not only take place over a comprehensive period, but allowed for the exploration of DMI integration in everyday compositional and explorative practices. As with any investigation that involves creativity, it was important that the participants did not feel rushed or restricted. That is, they were given sufficient time to explore and assess the different feedback types without constraint. This provided an accurate and representational set of qualitative data for validating the participants’ experience with the different feedback types they were presented with

A Qualitative Analysis of Haptic Feedback in Music Focused Exercises

We present the findings of a pilot-study that analysed the role of haptic feedback in a musical context. To examine the role of haptics in Digital Musical Instrument (DMI) design an experiment was formulated to measure the users’ perception of device usability across four separate feedback stages: fully haptic (force and tactile combined), constant force only, vibrotactile only, and no feedback. The study was piloted over extended periods with the intention of exploring the application and integration of DMIs in real-world musical contexts. Applying a music orientated analysis of this type enabled the investigative process to not only take place over a comprehensive period, but allowed for the exploration of DMI integration in everyday compositional practices. As with any investigation that involves creativity, it was important that the participants did not feel rushed or restricted. That is, they were given sufficient time to explore and assess the different feedback types without constraint. This provided an accurate and representational set of qualitative data for validating the participants’ experience with the different feedback types they were presented with

Sampling the past:a tactile approach to interactive musical instrument exhibits in the heritage sector

In the last decade, the heritage sector has had to adapt to a shifting cultural landscape of public expectations and attitudes towards ownership and intellectual property. One way it has done this is to focus on each visitor’s encounter and provide them with a sense of experiential authenticity.There is a clear desire by the public to engage with music collections in this way, and a sound museological rationale for providing such access, but the approach raises particular curatorial problems, specifically how do we meaningfully balance access with the duty to preserve objects for future generations?This paper charts the development of one such project. Based at Fenton House in Hampstead, and running since 2008, the project seeks to model digitally the keyboard instruments in the Benton Fletcher Collection and provide a dedicated interactive exhibit, which allows visitors to view all of the instruments in situ, and then play them through a custom-built two-manual MIDI controller with touch-screen interface.We discuss the approach to modelling, which uses high-definition sampling, and highlight the strengths and weaknesses of the exhibit as it currently stands, with particular focus on its key shortcoming: at present, there is no way to effectively model the key feel of a historic keyboard instrument.This issue is of profound importance, since the feel of any instrument is fundamental to its character, and shapes the way performers relate to it. The issue is further compounded if we are to consider a single dedicated keyboard as being the primary mode of interface for several instrument models of different classes, each with its own characteristic feel.We conclude by proposing an outline solution to this problem, detailing early work on a real-time adaptive haptic keyboard interface that changes its action in response to sampled resistance curves, measured on a key-by-key basis from the original instruments

Musical Haptics

Haptic Musical Instruments; Haptic Psychophysics; Interface Design and Evaluation; User Experience; Musical Performanc

Towards a multi-layer architecture for multi-modal rendering of expressive actions

International audienceExpressive content has multiple facets that can be conveyed by music, gesture, actions. Different application scenarios can require different metaphors for expressiveness control. In order to meet the requirements for flexible representation, we propose a multi-layer architecture structured into three main levels of abstraction. At the top (user level) there is a semantic description, which is adapted to specific user requirements and conceptualization. At the other end are low-level features that describe parameters strictly related to the rendering model. In between these two extremes, we propose an intermediate layer that provides a description shared by the various high-level representations on one side, and that can be instantiated to the various low-level rendering models on the other side. In order to provide a common representation of different expressive semantics and different modalities, we propose a physically-inspired description specifically suited for expressive actions

Embodied Musical Interaction

Music is a natural partner to human-computer interaction, offering tasks and use cases for novel forms of interaction. The richness of the relationship between a performer and their instrument in expressive musical performance can provide valuable insight to human-computer interaction (HCI) researchers interested in applying these forms of deep interaction to other fields. Despite the longstanding connection between music and HCI, it is not an automatic one, and its history arguably points to as many differences as it does overlaps. Music research and HCI research both encompass broad issues, and utilize a wide range of methods. In this chapter I discuss how the concept of embodied interaction can be one way to think about music interaction. I propose how the three “paradigms” of HCI and three design accounts from the interaction design literature can serve as a lens through which to consider types of music HCI. I use this conceptual framework to discuss three different musical projects—Haptic Wave, Form Follows Sound, and BioMuse

Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD

Audio-haptic relationships as compositional and performance strategies

As a performer of firstly acoustic and latterly electronic and electro-instrumental music, I constantly seek to improve my mode of interaction with the digital realm: that is, to achieve a high level of sensitivity and expression. This thesis illustrates reasons why making use of haptic interfaces—which offer physical feedback and resistance to the performer—may be viewed as an important approach in addressing the shortcomings of some the standard systems used to mediate the performer’s engagement with various sorts of digital musical information. By examining the links between sound and touch, and the performer-instrument relationship, various new compositional and performance strategies start to emerge. I explore these through a portfolio of original musical works, which span the continuum of composition and improvisation, largely based around performance paradigms for piano and live electronics. I implement new haptic technologies, using vibrotactile feedback and resistant interfaces, as well as exploring more metaphorical connections between sound and touch. I demonstrate the impact that the research brings to the creative musical outcomes, along with the implications that these techniques have on the wider field of live electronic musical performance