Haptics in music: the effects of vibrotactile stimulus in low frequency auditory difference detection tasks

We present an experiment that investigated the effect of vibrotactile stimulation in auditory pitch discrimination tasks. Extra-auditory information was expected to have some influence upon the frequency discrimination of auditory Just Noticeable Difference (JND) detection levels at 160 Hz. To measure this, the potential to correctly identified positive and negative frequency changes for two randomly divided groups was measured and then compared. The first group was given an audio only JND test and the second group was given the same test, but with additional vibrotactile stimulus delivered via a vibrating glove device. The results of the experiment suggest that in musical interactions involving the selection of specific pitches, or the detection of pitch variation, vibrotactile feedback may have some advantageous effect upon a musician's ability to perceive changes when presented in synchrony with auditory stimulus

Multimodal Listening as Technologically-Mediated Embodied Musicking

This paper discusses an approach to multimodal listening that draws upon three disparate but related areas of research: the creative music practice of the first author, quantitative evaluation of vibrotactile technology within speech and hearing science aimed at improving music perception and enjoyment for cochlear implant users, and a qualitative study involving the musical experiences of people who routinely wear a variety of assistive hearing technologies. It proposes that listening is an active, embodied process, taking place within the diverse sensorimotor, sociocultural, and aesthetic relationships between listeners and their worlds. These ideas are explored through observations of musical listening experiences that are technologically-mediated in various ways: they are ones which incorporate both the sonic and the tactile, but which do not aim to simply substitute one sensory modality for another. This project is intentionally interdisciplinary in its methodologies with the goal of illuminating how research approaches that work in tension can be productively inventive

Wearable haptic systems for the fingertip and the hand: taxonomy, review and perspectives

In the last decade, we have witnessed a drastic change in the form factor of audio and vision technologies, from heavy and grounded machines to lightweight devices that naturally fit our bodies. However, only recently, haptic systems have started to be designed with wearability in mind. The wearability of haptic systems enables novel forms of communication, cooperation, and integration between humans and machines. Wearable haptic interfaces are capable of communicating with the human wearers during their interaction with the environment they share, in a natural and yet private way. This paper presents a taxonomy and review of wearable haptic systems for the fingertip and the hand, focusing on those systems directly addressing wearability challenges. The paper also discusses the main technological and design challenges for the development of wearable haptic interfaces, and it reports on the future perspectives of the field. Finally, the paper includes two tables summarizing the characteristics and features of the most representative wearable haptic systems for the fingertip and the hand

Audio-haptic relationships as compositional and performance strategies

As a performer of firstly acoustic and latterly electronic and electro-instrumental music, I constantly seek to improve my mode of interaction with the digital realm: that is, to achieve a high level of sensitivity and expression. This thesis illustrates reasons why making use of haptic interfaces—which offer physical feedback and resistance to the performer—may be viewed as an important approach in addressing the shortcomings of some the standard systems used to mediate the performer’s engagement with various sorts of digital musical information. By examining the links between sound and touch, and the performer-instrument relationship, various new compositional and performance strategies start to emerge. I explore these through a portfolio of original musical works, which span the continuum of composition and improvisation, largely based around performance paradigms for piano and live electronics. I implement new haptic technologies, using vibrotactile feedback and resistant interfaces, as well as exploring more metaphorical connections between sound and touch. I demonstrate the impact that the research brings to the creative musical outcomes, along with the implications that these techniques have on the wider field of live electronic musical performance

Enactive Sound Machines: Theatrical Strategies for Sonic Interaction Design

Embodied interaction with digital sound has been subject to much prior research, but a method of coupling simple and intuitive hand actions to the vast potential of digital soundmaking in a perceptually meaningful way remains elusive. At the same time, artistic practices centred on performative soundmaking with objects remain overlooked by researchers. This thesis explores the design and performance of theatre sound effects in Europe and the U.S. in the late nineteenth and early twentieth century in order to converge the embodied knowledge of soundmaking at the heart of this historical practice with present-day design and evaluation strategies from Sonic Interaction Design and Digital Musical Instrument design. An acoustic theatre wind machine is remade and explored as an interactive sounding object facilitating a continuous sonic interaction with a wind-like sound. Its main soundmaking components are digitally modelled in Max/MSP. A prototype digital wind machine is created by fitting the acoustic wind machine with a rotary encoder to activate the digital wind-like sound in performance. Both wind machines are then evaluated in an experiment with participants. The results show that the timbral qualities of the wind-like sounds are the most important factor in how they are rated for similarity, that the rotational speed of both wind machines is not clearly perceivable from their sounds, and that the enactive properties of the acoustic wind machine have not yet been fully captured in the digital prototype. The wind machine’s flywheel mechanism is also found to be influential in guiding participants in their performances. The findings confirm the acoustic wind machine’s ability to facilitate enactive learning, and a more complete picture of its soundmaking components emerges. The work presented in this thesis opens up the potential of mechanisms to couple simple hand actions to complex soundmaking, whether acoustic or digital, in an intuitive way

Human-Computer interaction methodologies applied in the evaluation of haptic digital musical instruments

Recent developments in interactive technologies have seen major changes in the manner in which artists, performers, and creative individuals interact with digital music technology; this is due to the increasing variety of interactive technologies that are readily available today. Digital Musical Instruments (DMIs) present musicians with performance challenges that are unique to this form of computer music. One of the most significant deviations from conventional acoustic musical instruments is the level of physical feedback conveyed by the instrument to the user. Currently, new interfaces for musical expression are not designed to be as physically communicative as acoustic instruments. Specifically, DMIs are often void of haptic feedback and therefore lack the ability to impart important performance information to the user. Moreover, there currently is no standardised way to measure the effect of this lack of physical feedback. Best practice would expect that there should be a set of methods to effectively, repeatedly, and quantifiably evaluate the functionality, usability, and user experience of DMIs. Earlier theoretical and technological applications of haptics have tried to address device performance issues associated with the lack of feedback in DMI designs and it has been argued that the level of haptic feedback presented to a user can significantly affect the user’s overall emotive feeling towards a musical device. The outcome of the investigations contained within this thesis are intended to inform new haptic interface

Vibrotactile perception of musical pitch

Previous vibrotactile research has provided little or no definitive results on the discrimination and identification of important pitch aspects for musical performance such as relative and absolute pitch. In this thesis, psychophysical experiments using participants with and without hearing impairments have been carried out to determine vibrotactile detection thresholds on the fingertip and foot, as well as assess the perception of relative and absolute vibrotactile musical pitch. These experiments have investigated the possibilities and limitations of the vibrotactile mode for musical performance. Over the range of notes between C1 (32.7Hz) and C6 (1046.5Hz), no significant difference was found between the mean vibrotactile detection thresholds in terms of displacement for the fingertip of participants with normal hearing and with severe/profound hearing impairments. These thresholds have been used to identify an optimum dynamic range in terms of frequency-weighted acceleration to safely present vibrotactile music. Assuming a practical level of stimulation ≈10dB above the mean threshold, the dynamic range was found to vary between 12 and 27dB over the three-octave range from C2 to C5. Results on the fingertip indicated that temporal cues such as the transient and continuous parts of notes are important when considering the perception of vibrotactile pitch at suprathreshold levels. No significant difference was found between participants with normal hearing and with severe/profound hearing impairments in the discrimination of vibrotactile relative pitch from C3 to C5 using the fingertip without training. For participants with normal hearing, the mean percentage of correct responses in the post-training test was greater than 70% for intervals between four and twelve semitones using the fingertip and three to twelve semitones using the forefoot. Training improved the correct responses for larger intervals on fingertips and smaller intervals on forefeet. However, relative pitch discrimination for a single semitone was difficult, particularly with the fingertip. After training, participants with normal hearing significantly improved in the discrimination of relative pitch with the fingertip and forefoot. However, identifying relative and absolute pitch was considerably more demanding and the training sessions that were used had no significant effect

On the hunt for feedback: Vibrotactile feedback in interactive electronic music performances

The expressivity of musical performance is highly dependent on the feedback relationship between the performer and the instrument. Despite current advances in music technology, performers still struggle to retain the same expressive nuances of acoustic instruments. The capacity of performative musical expression in technologically-driven music is mitigated by the limitations of controllers and other sensor-based devices used in the performance of such music. Due to their physical properties, such devices and components are unable to provide mainly the haptic and vibrotactile experience between the instrument and the user, thus breaking the link with traditional musical performance. Such limitations are apparent to performers, suggesting often the existence of an unnatural barrier between the technology and the performer. The thesis proposes the use of vibrotactile feedback as means to enhance performer’s expressivity and creativity in technology mediated performances and situate vibrotactile feedback as part of the tradition of instrumental musical playing. Achieved through the use of small controllable electric motors, vibrotactile feedback can nourish communicative pathways between the performer and technology, a relationship that is otherwise limited or non-existing. The ability to experience an instrument's communicative response can significantly improve the performer-instrument relationship, and in turn the music performed. Through a series of case studies, compositions and performances, the dissertation suggests ways in which vibrotactile feedback may be applied to enhance the experience between the technology and the performer. As a result performers are able to develop expressive nuances and have better control of the technology during performance