MusicJacket: the efficacy of real-time vibrotactile feedback for learning to play the violin

This research investigates the potential for vibrotactile feedback to enhance motor learning in the context of playing the violin. A prototype has been built which delivers vibrotactile feedback to the arms to indicate to a novice player how to correctly hold the violin and how to bow in a straight manner. This prototype was tested in a pilot user study with four complete beginners. Observations showed improvements in three of the four players whilst receiving the feedback. We also discuss the pros and cons of using negative feedback to enhance learning

Technology inspired design for pervasive healthcare

Pervasive healthcare technologies are increasingly using novel sensory devices that are able to measure phenomena that could not be measured before. To develop novel healthcare applications that use these largely untested technologies, it is important to have a design process that allows proper exploration of the capabilities of the novel technologies. We focus on the technology-inspired design process that was used in the development of a system to support posture and provide guidance by nudging people, and how this has lead us to explore pervasive healthcare applications

In Touch with the Wild: Exploring Real-time Feedback for Learning to Play the Violin

Real-time feedback has great potential for enhancing learning complex motor-skills by enabling people to correct their mistakes as they go. Multimodal real-time cues could provide reinforcement to inform players whether they are making the correct or incorrect movements at a given time. However, little is known about how best to communicate information in real-time so that people can readily perceive and apply it to improving their movement while learning complex motor-skills. This thesis addresses this gap in knowledge by investigating how real-time feedback can enhance learning to play the violin. It explores how haptic and visual feedback are perceived, understood and acted upon in real-time when engaged in the primary task of playing the violin. Prototypes were built with sensors to measure movement and either vibrations on the body or visual signals as feedback. Three in-the-wild user studies were conducted: one comparing visual and vibrotactile feedback for individual practice; one investigating shared feedback at a musical summer school; and one examining real-time feedback as part of a programme of learning at a high school. In-the-wild studies investigate users interacting with technology in a naturalistic setting, with all the demands that this entails. The findings show real-time feedback is effective at improving violin technique and can support learning in other ways such as encouraging mutual support between learners. The positive learning outcomes, however, need to be understood with respect to the complex interplay between the technology, demands of the setting and characteristics of individual learners. A conceptual framework is provided that outlines these interdependent factors. The findings are discussed regarding their applicability to learning other physical skills and the challenges and insights of using an in-the-wild methodology. The contribution of this thesis is to demonstrate empirically and theoretically how real-time vibrotactile and visual feedback can enhance learning a complex motor-skill

Postrum II: a posture aid for trumpet players

While brass pedagogy has traditionally focussed on sound output, the importance of bodily posture to both short-term performance and longer-term injury prevention is now widely recognized. Postrum II is a Linux-based system for trumpet players that performs real-time analysis of posture and uses a combination of visual and haptic feedback to try to correct any posture issues that are found. Issues underpinning the design of the system are discussed, the transition from Mac OS X to Ubuntu detailed, and some possibilities for future work suggested

POSTRUM: Developing good posture in trumpet players through directional haptic feedback

The literature of brass pedagogy has identified the typical posture problems found in trumpet players and arrived at a consensus regarding optimal body alignment. The suggestion is that poor posture may not only hinder performance but also lead to longterm injuries. This is supported by a growing body of evidence from fields as diverse as biomechanics and pervasive healthcare. After a review of the literature, we focus on the design process used to develop Postrum; a wearable system for trumpet players that uses real-time haptic feedback to encourage better posture. In response to the multifaceted nature of the activity, the design process combines two aspects from different fields: the ‘sketching in hardware’ approach developed by Moussette and Dore in the context of Interaction Design (IxD), and sensing technologies from the New Interfaces for Musical Expression (NIME) field. We follow this with a brief overview of the Postrum system. This includes a 3D camera, custom software that compares the posture of the player to an idealized model, and two vibrotactile arrays mounted on the torso. Three different types of problem can be detected, their categories based on the literature. If player posture deviates from the ideal, haptic feedback is applied. Directional pulses used to indicate the corrective action needed. Finally, we offer some remarks about our experiences in relation to player engagement and performance, discuss emerging design issues, and outline implications for what Hochenbaum and Kapur term the ‘practice room of the future.

Augmenting Sensorimotor Control Using “Goal-Aware” Vibrotactile Stimulation during Reaching and Manipulation Behaviors

We describe two sets of experiments that examine the ability of vibrotactile encoding of simple position error and combined object states (calculated from an optimal controller) to enhance performance of reaching and manipulation tasks in healthy human adults. The goal of the first experiment (tracking) was to follow a moving target with a cursor on a computer screen. Visual and/or vibrotactile cues were provided in this experiment, and vibrotactile feedback was redundant with visual feedback in that it did not encode any information above and beyond what was already available via vision. After only 10 minutes of practice using vibrotactile feedback to guide performance, subjects tracked the moving target with response latency and movement accuracy values approaching those observed under visually guided reaching. Unlike previous reports on multisensory enhancement, combining vibrotactile and visual feedback of performance errors conferred neither positive nor negative effects on task performance. In the second experiment (balancing), vibrotactile feedback encoded a corrective motor command as a linear combination of object states (derived from a linear-quadratic regulator implementing a trade-off between kinematic and energetic performance) to teach subjects how to balance a simulated inverted pendulum. Here, the tactile feedback signal differed from visual feedback in that it provided information that was not readily available from visual feedback alone. Immediately after applying this novel “goal-aware” vibrotactile feedback, time to failure was improved by a factor of three. Additionally, the effect of vibrotactile training persisted after the feedback was removed. These results suggest that vibrotactile encoding of appropriate combinations of state information may be an effective form of augmented sensory feedback that can be applied, among other purposes, to compensate for lost or compromised proprioception as commonly observed, for example, in stroke survivors

Improvising through the senses: a performance approach with the indirect use of technology

This article explores and proposes new ways of performing in a technology-mediated environment. We present a case study that examines feedback loop relationships between a dancer and a pianist. Rather than using data from sensor technologies to directly control and affect musical parameters, we captured data from a dancer's arm movements and mapped them onto a bespoke device that stimulates the pianist's tactile sense through vibrations. The pianist identifies and interprets the tactile sensory experience, with his improvised performance responding to the changes in haptic information received. Our system presents a new way of technology-mediated performer interaction through tactile feedback channels, enabling the user to establish new creative pathways. We present a classification of vibrotactile interaction as means of communication, and we conclude how users experience multi-point vibrotactile feedback as one holistic experience rather than a collection of discrete feedback points