An Implementation and Usability Study of a Natural User Interface Virtual Piano

We present an implementation of a Natural User Interface (NUI) virtual piano keyboard. Using the Unity game engine and a Leap Motion hand controller, users are able to use their hands to interact with and play music in a virtual scene. Various approaches were attempted in refining the user experience of the virtual piano, and the successes and shortcomings of each implementation method are outlined and elaborated upon. Additionally, the most successful method, which was achieved by using Unity’s physics engine to detect rigid body collisions between virtual fingers and keys, was used to complete a usability study involving both experienced and amateur pianists. Subjects were presented with a series of playability tasks designed to gauge their ability to accurately use the virtual keyboard. Both the usability test data and the subject’s personal feedback were collected and analyzed

ADEPT:Exploring the Design, Pedagogy, and Analysis of a Mixed Reality Application for Piano Training

(Abstract to follow

claVision: visual automatic piano music transcription

One significant problem in the science of Musical Information Retrieval is Automatic Music Transcription, which is an automated conversion process from played music to a symbolic notation such as sheet music. Since the accuracy of previous audio-based transcription systems is not satisfactory, an innovative visual-based automatic music transcription system named claVision is proposed to perform piano music transcription. Instead of processing the music audio, the system performs the transcription only from the video performance captured by a camera mounted over the piano keyboard. claVision can be used as a transcription tool, but it also has other applications such as music education. The software has a very high accuracy (over 95%) and a very low latency (less than 6.6 ms) in real-time music transcription, even under different illumination conditions. This technology can also be used for other musical keyboard instruments. claVision is the winner of the 2014 Microsoft Imagine Cup Competition in the category of innovation in both Canadian national finals and world semifinals. As one of the top 11 teams in the world, claVision advanced to World Finals in Seattle to be demonstrated at the University of Washington, Microsoft headquarters, and the Museum of History & Industry

Designing on subjective tolerance to approximated piano reproductions

Results from three experiments are presented, showing that the perceived acoustic and vibrotactile quality of a reproduced piano does not require models simulating every aspect of the original instrument with great accuracy. It was found that high-quality loudspeaker array passive listening at the pianist's position admits distortions of the sound field. Furthermore, pianists during playing seem to compensate for errors in the auditory scene description. Finally, they are particularly sensitive to the existence of vibrotactile feedback on their fingers meanwhile tolerant about the precision with which this feedback is reproduced. Based on these results we are currently working on a lightweight portable physics-based digital piano design, that should improve upon the experience a pianist with no keyboards at hand makes when interacting with a touch-screen piano software running on smartphones and laptops

Integrating optical finger motion tracking with surface touch events

This paper presents a method of integrating two contrasting sensor systems for studying human interaction with a mechanical system, using piano performance as the case study. Piano technique requires both precise small-scale motion of fingers on the key surfaces and planned large-scale movement of the hands and arms. Where studies of performance often focus on one of these scales in isolation, this paper investigates the relationship between them. Two sensor systems were installed on an acoustic grand piano: a monocular high-speed camera tracking the position of painted markers on the hands, and capacitive touch sensors attach to the key surfaces which measure the location of finger-key contacts. This paper highlights a method of fusing the data from these systems, including temporal and spatial alignment, segmentation into notes and automatic fingering annotation. Three case studies demonstrate the utility of the multi-sensor data: analysis of finger flexion or extension based on touch and camera marker location, timing analysis of finger-key contact preceding and following key presses, and characterization of individual finger movements in the transitions between successive key presses. Piano performance is the focus of this paper, but the sensor method could equally apply to other fine motor control scenarios, with applications to human-computer interaction

Augmenting the experience of playing the piano: controlling audio processing through ancillary gestures

Pianists spend many years practicing on their instrument. As a result they develop alongside their pianistic technique a set of gestural nuances that enable them to perform expressively and establish their own acoustic signature on the piano. This mute layer of nuanced gestures is rarely taken into consideration when developing new keyboard-based gestural interfaces. These often usually require new gestural vocabularies to be learned resulting in a disruptive experience for the pianist. The main objective of this research is to investigate how new keyboard-based gestural interfaces can enable musicians to control and transform live piano sound through the gestural nuances embedded in their technique. Specifically, how keyboard interfaces with nuanced gestural control can extend the creative possibilities available to classically trained pianists, thus stimulating new approaches to build intuitive interfaces for musical expression, and new ways of learning and playing digital instruments. Towards this goal, interviews, user tests and case studies were conducted with a range of pianists coming from different musical backgrounds, and Reach, an augmented instrument for live sound modulation controlled by gestural nuances embedded in the pianistic technique was developed

ECHOES OF THE MASTER: A MULTI-DIMENSIONAL MAPPING OF ENRIQUE GRANADOS’ PEDAGOGICAL METHOD AND PIANISTIC TRADITION

The interpretation of the piano works of Enrique Granados has challenged musicologists and performers for decades. The performances of Granados have often been described as improvisatory and spontaneous, as if his creative artistry and natural talent denied following any rigid rules. He did not like to perform the same work in the same way more than once and even his compositional process was one of continual evolution. Further, anecdotes about Granados as teacher describe the way he encouraged his students to try different approaches in the performance of his pieces. However, this description of him as a relaxed and rather free spirit might be somewhat misleading. The pedagogical methods of Granados contain painstaking descriptions and guidelines for students and his comprehensive theoretical methods are supplemented by a series of practical exercises, some of which he demanded his students follow rigidly. In an effort to reconcile both sides of his artistic temperament and to steer the student of Granados’ piano works in the right direction, attempts have been made to develop authoritative editions of his works through the analysis of his recordings and scores under the supervision of well-known interpreters of his piano music. This thesis proposes that the formulation of an authentic interpretation of Granados’ piano music involves a fresh look at the source material. It re-examines Urtext in the light of additional, more up to date analytical approaches and reveals consistencies between his pedagogical methods and his own performances. It also draws on the knowledge and reminiscences of Spanish pianists who play an incredibly important role in terms of their direct lineage to the Granados/Catalan piano school, but who are perhaps not so well-known outside of the Spanish musical community. This shared information is brought together for the first time in this thesis and helps to colour a more vivid and complete portrait of one the most important Spanish musicians of all time

Harnessing ancillary microgestures in piano technique: implementing microgestural control in to an expressive keyboard-based hyper-instrument

The chapter describes through the implementation of microgestural sound control, how performers can gain wide control over digital sound processing through their existing technique. By using radar millimeter waves to capture micromotions and microgestures, performers achieve high level of expression without the need to modify their instrument nor dev additional technique. This research builds upon existing instrumental technique and removes the steep learning curve typically found when performing digital or augmented musical instruments. We present a case study that enables pianist to retain and focus on technical control and musical freedom resulting in a less disruptive experience