Elucidating musical structure through empirical measurement of performance parameters

The differences between a musical score and an instance of that music in a performance, communicates a performer’s view of the information contained in that score. The main hypothesis in this thesis is that by measuring quantifiable parameters such as tempo, dynamics and motion from live performance, the performer’s interpretation of musical structure can be detected. This will be tested for pieces for which the structure is explicit and obvious, and then used to discover musical structure from looking at patterns of aural and visual performance parameters in performances of more ambiguously structured pieces. This thesis is in two strands. The first part covers the acquisition of multi-modal parameters in piano performance. This will explore current technologies in acquiring MIDI information such as accurate onset timings and key velocities as well as motion tracking systems for measuring general body movements. A new cheap, portable and accurate system for tracking the intricacies of pianists’ finger movement is described as well as methods and tools available for analysis and visualisation of musical data. The second strand of this thesis will explore uses of these capture systems in empirically measuring performance parameters to elucidate musical structure. Two experiments follow which test the hypothesis of detecting musical structure from parameters such as tempo, dynamics and movement, before using these patterns as a basis for discovering structure in performances of the finale of Chopin’s B flat minor sonata. Body movement is discovered as an indicator of phrasing boundaries, which when combined with the measured aural parameters provides interpretations of the performed music. Phrasing boundaries are identified correctly for the control piece (Chopin’s Prelude in A major Op.28, No.7) and consequently for the first test piece (Chopin’s Prelude in B minor Op.28 No.6). The proceeding experiment identifies performers’ style of phrase endings through performances of the control piece and tests them against patterns found in the second test piece (Chopin’s B Flat minor Sonata Finale). Five out of the six performers confirm the musicological hypothesis that bar 5 is not the entry of a new theme but the continuation of the the theme beginning in bar 1

Music technologies: ppportunities for social connection

Strategies to support psychosocial well-being in older adults are desperately needed. A developing body of research points to the relationship between continued engagement with the arts and maintaining mental health and quality of life (Wang, et al., 2020). Music is an effective, non-pharmacological tool with many social and emotional benefits particularly for older adults (Creech, et al., 2014),and technology is posited to play a role in making music interventions more accessible and cost-effective (Garrido, et al., 2018). In addition to a brief overview of how musical engagement can support older Australians' psychosocial well-being, this presentation will discuss technologies for both consuming and making music. This will focus on recent empirical research findings comparing the impact and benefit of technology-driven music opportunities designed to promote social connection. Because technologies continue to develop, it is important to consider the underpinning principles corresponding to use and engagement. These principles can guide the purchase and implementation of these technologies in aged care. We will focus on fostering musical engagement through technologies for social connection and well-being. Through this lens, we will explore: 1. Technology Types – what equipment is needed, considering price, availability, and levels of user interaction? 2. Skills & Education – how to make use of existing staff and resident knowledge, and sourcing relevant education & training. 3. Flexibility & Accessibility – how easy is it to mould or modify music technology activities to residents’ personal choices as well as their physical/ cognitive abilities? 4. Sustainability – what is reproducible and sustainable in the face of staff/resident changes? I.e. how to make sure the newly purchased technology doesn’t end up in the cupboard?! Addressing aspects of implementation relevant to practitioners creates the link between increasing awareness of the benefits of music consumption and creation and being able to translate these empirical research findings into everyday use

Revitalisation of Mangarrayi: Supporting community use of archival audio exemplars for creation of language learning resources

Mangarrayi is a critically endangered language from the western Roper River re- gion in the Northern Territory of Australia. Today the greatest concentration of Mangarrayi people live at Jilkminggan, 135 kilometres south-east of Katherine. Although several older Mangarrayi speakers remain, the language is no longer used in day-to-day communication. However, there is a desire amongst a number of young adult community members to learn some of their heritage language. In this paper we discuss the process undertaken to support these aspirations, focus- ing on the use of exemplar Mangarrayi utterances sourced from archival docu- ments as a key to developing a basic level of communicative competence in con- texts identified as important to learners. This requires a clear understanding of how and when to use the utterances. We propose using a combination of language functions, topics, and sub-topics to clarify usage and support non-specialist com- munity members in using these for learning and teaching Mangarrayi.National Foreign Language Resource Cente

Does movement amplitude of a co-performer affect individual performance in musical synchronization?

Interpersonal coordination in musical ensembles often involves multisensory cues, with visual information about body movements supplementing co-performers’ sounds. Previous research on the influence of movement amplitude of a visual stimulus on basic sensorimotor synchronization has shown mixed results. Uninstructed visuomotor synchronization seems to be influenced by amplitude of a visual stimulus, but instructed visuomotor synchronization is not. While music performance presents a special case of visually mediated coordination, involving both uninstructed (spontaneously coordinating ancillary body movements with co-performers) and instructed (producing sound on a beat) forms of synchronization, the underlying mechanisms might also support rhythmic interpersonal coordination in the general population. We asked whether visual cue amplitude would affect nonmusicians’ synchronization of sound and head movements in a musical drumming task designed to be accessible regardless of musical experience. Given the mixed prior results, we considered two competing hypotheses. H1: higher amplitude visual cues will improve synchronization. H2: different amplitude visual cues will have no effect on synchronization. Participants observed a human-derived motion capture avatar with three levels of movement amplitude, or a still image of the avatar, while drumming along to the beat of tempo-changing music. The moving avatars were always timed to match the music. We measured temporal asynchrony (drumming relative to the music), predictive timing, ancillary movement fluctuation, and cross-spectral coherence of ancillary movements between the participant and avatar. The competing hypotheses were tested using conditional equivalence testing. This method involves using a statistical equivalence test in the event that standard hypothesis tests show no differences. Our results showed no statistical differences across visual cues types. Therefore, we conclude that there is not a strong effect of visual stimulus amplitude on instructed synchronization

The use of technology for arts-based activities in older adults living with mild cognitive impairment or dementia: a scoping review

For older adults living with mild cognitive impairment or dementia, creative arts-based activities can offer many benefits from enjoyment as leisure/recreation to an avenue to maintain cognitive, social and emotional wellbeing. With growing interest and recognition that technology could have potential to assist in delivering these activities in more accessible and personalised ways, a scoping review was undertaken to systematically examine the scientific literature for technology-assisted creative arts activities for older adults living with dementia. We searched PubMed, PsychINFO, Web of Science, Scopus and ACM Digital Library databases using keywords centering on population with dementia, an intervention using technology, and a context of creative arts, with no restrictions on the type of outcome measured. We retrieved 3739 records, with an additional 22 from hand-searching. 51 full-text articles met the inclusion and exclusion criteria. Findings of the review indicate technologies principally being designed for music activities (listening, and music-making), as well as storytelling and visual arts. The majority of devices were custom-made, with studies mainly reporting on validating the success of the device/intervention. This suggests most work in the field is currently at prototyping stage, although a few devices are now commercially available. Recommendations for future research includes involvement of participants reporting on their previous experiences in the arts and how this influences co-design choices, and inclusion of different severities of dementia in the participant/co-design group. Furthering device development past prototyping stage as well as collaboration between teams would enable comparisons to be made across different types of devices used for the same activity, and comparisons across arts-based activities that could lead to cross-disciplinary outcomes for the design of creative arts-based assistive technologies

Temporally Guided Music-to-Body-Movement Generation

This paper presents a neural network model to generate virtual violinist's 3-D skeleton movements from music audio. Improved from the conventional recurrent neural network models for generating 2-D skeleton data in previous works, the proposed model incorporates an encoder-decoder architecture, as well as the self-attention mechanism to model the complicated dynamics in body movement sequences. To facilitate the optimization of self-attention model, beat tracking is applied to determine effective sizes and boundaries of the training examples. The decoder is accompanied with a refining network and a bowing attack inference mechanism to emphasize the right-hand behavior and bowing attack timing. Both objective and subjective evaluations reveal that the proposed model outperforms the state-of-the-art methods. To the best of our knowledge, this work represents the first attempt to generate 3-D violinists' body movements considering key features in musical body movement

Deciphering and Embodying Contemporary Piano Scores: A Commentary on Huisman, Gingras, Dhondt, and Leman (2017)

Rehearsing a new contemporary notated piano score often requires the performer to dedicate a large amount of time to translating the notation: tasks may include interpreting new symbols, or old symbols in new contexts, learning new extended techniques, and learning the link between symbol and physical gesture. Huisman et al's article details an experiment that studies the use of various types of music editions and their effect on the practice and performance of contemporary piano music scores. This commentary considers the following issues when discussing the interpretation of unfamiliar piano scores: 1) issues in reading notation that arise from performance practice challenges, 2) cultivating an interpretive platform, and 3) embodiment in the rehearsal of a new, contemporary score

The art and science behind piano touch : a review connecting multi-disciplinary literature

The touch used to play the piano, representing aspects of body posture, hand posture, movement, speed, force and pressure on the key, has been addressed over the centuries by pedagogues, famous pianists and composers. Piano touch not only underpins basic technique in piano playing, but is also the route through which pianists can communicate their expressive intentions. This review pulls together literature concerning touch from piano pedagogy, engineering, performance analysis and biomechanics. Studies have advanced understanding in the actions behind a typical keypress motion, the influence of training by examining differences between novice and expert pianists, and the various joint contributions that discern a struck versus a pressed touch. Although individual differences are widely identified, the influence of hand anthropometry, choice of technique and difference in training between experts remains understudied. A trade-off between accuracy and ecological validity is also identified in the use of current measurement systems: to encourage wider participation and incorporation into instrumental lessons, there is a need for the development of un-intrusive measurement systems that can be used outside the laboratory environment without restrictions concerning the instrument. Implications include furthering understanding across the arts and sciences and aiding teachers and students looking to minimise the risk of injury

A 3D Camera User Interface for Wrist Angle Monitoring in Piano Performances

Les interprètes doivent gérer des blessures tout au long de leur carrière, et chez les pianistes, ce sont surtout les blessures aux poignets qui les font particulièrement souffrir. Alors que les recommandations générales insistent sur la position « neutre » du poignet pour éviter les blessures, elles sont rarement appliquées dans la réalité. De récentes avancées technologiques dans le domaine de la capture de mouvements pourraient aider les étudiants à être plus conscients de leur propension à dépasser les limites de la position « neutre » recommandée. Ces technologies pourraient être utilisées pour mesurer les positions précises du poignet durant le jeu pianistique afin de déterminer des ensembles de seuils spécifiques qui permettront d’éviter les blessures. Cet article présente les avantages et les inconvénients des technologies de la capture de mouvement, incluant la visualisation des données et leur utilisation dans le cadre de la pédagogie instrumentale, afin d’établir une liste de critères auxquels répondrait un système d’analyse des mouvements facile à utiliser. L’article présente un prototype de système de traitement des images spécialisé accompagné d’une interface graphique pour les utilisateurs qui répond à cette liste de critères. Le système recourt à des marqueurs de couleur passifs et à une caméra 3D standard afin d’encourager une utilisation en dehors des environnements de laboratoire traditionnels. Des options de calibrage simples pour la caméra, ainsi qu’une localisation de base des mains grâce à des images aériennes permettent d’observer les flexions et extensions des poignets dans de courts enregistrements vidéo. Les données sont comparées à des seuils de flexion/extension proposés pour les dactylos afin de prévenir les problèmes de tunnel carpien, et les utilisateurs sont avertis lorsqu’ils approchent ou dépassent ces seuils à la fois pendant et après l’enregistrement. Les applications potentielles du système incluent la surveillance durant la répétition de courts passages techniques, sans restriction d’instrument et de lieu.Injuries are common over a performing musician’s career and wrist injuries are the most frequent site of pain for pianists. Although general recommendations insist on keeping wrists in a “neutral” position to avoid injury, this is rarely done in practice. Recent advances in motion capture technology may aid in raising students’ awareness of the propensity to use wrist positions outside of the recommended “neutral.” These technologies may be used to measure precise wrist positions in piano playing in order to set specific thresholds for avoiding injury. This paper discusses various advantages and limitations of motion capture technologies, including data visualization and usage within the music instrument pedagogy framework in order to define a set of requirements for an accessible motion-tracking system. A prototype of a dedicated image-processing-based system with a graphical user interface that meets these requirements is described. This system uses passive coloured markers and a standard 3D camera, encouraging use outside the traditional laboratory environment. Simple camera calibration options and basic hand tracking from aerial view images allow monitoring of wrist flexion/extension over short video recordings. Measurements are compared to flexion/extension thresholds recommended for typists to prevent carpal tunnel pressure, and moments of approaching or exceeding these thresholds are flagged to the user both in real time and in post-performance. Potential applications include monitoring the practice of short technical passages without restriction of instrument or location