Vision-based Detection of Acoustic Timed Events: a Case Study on Clarinet Note Onsets

Acoustic events often have a visual counterpart. Knowledge of visual information can aid the understanding of complex auditory scenes, even when only a stereo mixdown is available in the audio domain, \eg identifying which musicians are playing in large musical ensembles. In this paper, we consider a vision-based approach to note onset detection. As a case study we focus on challenging, real-world clarinetist videos and carry out preliminary experiments on a 3D convolutional neural network based on multiple streams and purposely avoiding temporal pooling. We release an audiovisual dataset with 4.5 hours of clarinetist videos together with cleaned annotations which include about 36,000 onsets and the coordinates for a number of salient points and regions of interest. By performing several training trials on our dataset, we learned that the problem is challenging. We found that the CNN model is highly sensitive to the optimization algorithm and hyper-parameters, and that treating the problem as binary classification may prevent the joint optimization of precision and recall. To encourage further research, we publicly share our dataset, annotations and all models and detail which issues we came across during our preliminary experiments.Comment: Proceedings of the First International Conference on Deep Learning and Music, Anchorage, US, May, 2017 (arXiv:1706.08675v1 [cs.NE]

Multimodal music information processing and retrieval: survey and future challenges

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, mid-level representations, motion, and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years

Aesthetically driven design of network based multi-user instruments.

Digital networking technologies open up a new world of possibilities for music making, allowing performers to collaborate in ways not possible before. Network based Multi-User Instruments (NMIs) are one novel method of musical collaboration that take advantage of networking technology. NMIs are digital musical instruments that exist as a single entity instantiated over several nodes in a network and are performed simultaneously by multiple musicians in realtime. This new avenue is exciting, but it begs the question of how does one design instruments for this new medium? This research explores the use of an aesthetically driven design process to guide the design, construction, rehearsal, and performance of a series of NMIs. This is an iterative process that makes use of a regularly rehearsing and performing ensemble which serves as a test-bed for new instruments, from conception, to design, to implementation, to performance. This research includes details of several NMIs constructed in accordance with this design process. These NMIs have been quantitatively analysed and empirically tested for the presence of interconnectivity and group influence during performance as a method for measuring group collaboration. Furthermore qualitative analyses are applied which test for the perceived e ectiveness of these instruments during real-world performances in front of live audiences. The results of these analyses show that an aesthetically driven method of designing NMIs produces instruments that are interactive and collaborative. Furthermore results show that audiences perceive a measurable impression of interconnectivity and liveness in the ensemble even though most of the performers in the ensemble are not physically present

Designing and Composing for Interdependent Collaborative Performance with Physics-Based Virtual Instruments

Interdependent collaboration is a system of live musical performance in which performers can directly manipulate each other’s musical outcomes. While most collaborative musical systems implement electronic communication channels between players that allow for parameter mappings, remote transmissions of actions and intentions, or exchanges of musical fragments, they interrupt the energy continuum between gesture and sound, breaking our cognitive representation of gesture to sound dynamics. Physics-based virtual instruments allow for acoustically and physically plausible behaviors that are related to (and can be extended beyond) our experience of the physical world. They inherently maintain and respect a representation of the gesture to sound energy continuum. This research explores the design and implementation of custom physics-based virtual instruments for realtime interdependent collaborative performance. It leverages the inherently physically plausible behaviors of physics-based models to create dynamic, nuanced, and expressive interconnections between performers. Design considerations, criteria, and frameworks are distilled from the literature in order to develop three new physics-based virtual instruments and associated compositions intended for dissemination and live performance by the electronic music and instrumental music communities. Conceptual, technical, and artistic details and challenges are described, and reflections and evaluations by the composer-designer and performers are documented

Disability arts and visually impaired musicians in the community

This chapter reports on a multifaceted ‘disability arts scene’ in music worldwide that comprises visually impaired (i.e., blind and partially sighted) instrumentalists, singers, composers, producers, and others across a range of musical styles and genres. Some such musicians work alone but are usually deeply involved in networks. Others join community music ensembles that can be made up of musicians with a range of disabilities including visual impairments, or that consist entirely of visually impaired people. When promoting their community music participation, some visually impaired musicians draw on the history and traditions of the blind in music across the world, and thus exists the lore concerning special dispensations in the absence of sight. Yet there are also visually impaired musicians who distance themselves from that self-identity. The chapter explores how members of this unique socio-musical group consider the aforesaid ‘scene’ and its integral community music, and how their interpretations correspond or clash; it introduces key matters of accessibility, independent mobility, identity, musical approach and media, notions of discrimination, and social inclusion

GESTURE AND THE AUDITORY IMPACT ON THE PERCEPTION OF SOUND: A PERFORMANCE GUIDE FOR UNDERSTANDING GESTURAL FUNCTION ON PERCUSSION PEDAGOGY AND PERFORMANCE

I will address the concept of physical gestures and discuss how they are applicable to the way percussionists visually and audibly perceive music. Through my research, I will focus on performance and pedagogy, seeking new possibilities for introducing percussionists at all ability levels to different articulations. Percussionists have traditionally been led to believe that they do not have the versatility over the initial musical attack and sustain that wind and string players do. This document has been developed to provide a more in-depth approach to how physical and musical gestures can function to create a more versatile and musical percussionist. This project will discuss both musical and physical gestures and the current research behind them, as well as a spectromorphology, a method developed by Dennis Smalley to analyze graphic notation from computer-generated scores. By applying spectromorphology to percussion scores, I will examine how musical gestures are present in solo and ensemble literature, and I will offer suggestions on how to effectively apply these findings to methods of study and performance. This project concludes with a thorough breakdown of standard articulations across marimba/mallet instruments, snare drum, and timpani and provides suggestions on how to functionally utilize gestures for these articulations while offering graphs for each category that highlight the focal points for continued use