Multimodal music information processing and retrieval: survey and future challenges

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, mid-level representations, motion, and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years

Digital Music Libraries: Librarian Perspectives and the Challenges Ahead

This paper reports the results of a survey targeting current members of the Canadian Association of Music Libraries, Archives and Documentation Centres (CAML) that investigated the extent to which the current designs and structures of digital music libraries meet the needs of librarians in collecting, preserving, organizing, and disseminating diverse types of music documents. The challenges and barriers experienced in hosting digital collections are discussed. The gap between the current and ideal functionalities, as well as the future possibilities, are explored.

MediaScape: towards a video, music, and sound metacreation

We present a new media work, MediaScape, which is an initial foray into a fully interdisciplinary metacreativity. This paper defines metacreation, and we present examples of metacreative art within the fields of music, sound art, the history of generative narrative, and discuss the potential of the “open-documentary” as an immediate goal of metacreative video. Lastly, we describe MediaScape in detail, and present some future directions

“Give me happy pop songs in C major and with a fast tempo”: A vocal assistant for content-based queries to online music repositories

This paper presents an Internet of Musical Things system devised to support recreational music-making, improvisation, composition, and music learning via vocal queries to an online music repository. The system involves a commercial voice-based interface and the Jamendo cloud-based repository of Creative Commons music content. Thanks to the system the user can query the Jamendo music repository by six content-based features and each combination thereof: mood, genre, tempo, chords, key and tuning. Such queries differ from the conventional methods for music retrieval, which are based on the piece's title and the artist's name. These features were identified following a survey with 112 musicians, which preliminary validated the concept underlying the proposed system. A user study with 20 musicians showed that the system was deemed usable, able to provide a satisfactory user experience, and useful in a variety of musical activities. Differences in the participants’ needs were identified, which highlighted the need for personalization mechanisms based on the expertise level of the user. Importantly, the system was seen as a concrete solution to physical encumbrances that arise from the concurrent use of the instrument and devices providing interactive media resources. Finally, the system offers benefits to visually-impaired musicians

Examining the effect of practicing with different modeling conditions on the memorization of young piano students

La modélisation est une technique d'enseignement étudiée dans les domaines de l'apprentissage moteur, des neurosciences, de l'enseignement et de la musique. Cependant, on ignore si cette technique peut être efficace pour mémoriser la notation musicale pour piano, en particulier pour les jeunes élèves. Cette étude a donc examiné l'effet de la pratique instrumentale utilisant différentes conditions de modélisation sur la mémorisation d'une pièce de piano. Ces conditions de modélisation étaient les suivantes: modélisation auditive et modélisation vidéo avec indices. L'étude comportaitune quasi-expérience avec 24 jeunes élèves de piano de 3e année du Conservatoire royal de musique (CRM) au Canada ou l'équivalent. Les participants ont pratiqué avec une condition de modélisation afin de déterminer quelle condition produirait les meilleurs résultats de rétention mnémonique. Les résultats ont montré que la modélisation vidéo avec indices était l'outil de pratique le plus efficace en termes d'erreurs de notes et de rythmes, lorsqu'elle est comparée à la modélisation audio et aux groupes de pratique libre. Ces résultats appuient les recherches en neurosciences selon lesquelles l'utilisation de techniques visuelles, auditives et motrices produisent la meilleure rétention. Cela offre un grand potentiel pour l'utilisation de la modélisation vidéo avec repères comme outil de pratique pour les élèvesen piano afin d'améliorer la mémorisation.Modeling is a teaching technique that is studied in the fields of motor learning, neuroscience, teaching, and music. Yet it is unknown whether this technique can be effective in memorizing piano music especially for young students. Therefore, this study examined the effect of practicing with different modeling conditions on memorizing a piano piece. These modeling conditions were: aural modeling, and video modeling with cues. The study conducted a quasi-experiment with 24 young piano students at Grade 3 level of the Royal Conservatory of Music (RCM) in Canada or equivalent. Participants practiced with one modeling condition in order to measure which condition would produce best retention results. Results showed that video modeling with cues seemed to be the most effective practice tool in terms of low note mistakes and rhythm mistakes compared to audio modeling and free practice groups. This finding supports neuroscience research that states that the use of visual, aural and motor techniques produce the best memory recall. This provides great potential for using video modeling with cues as a practice tool for pianostudents for better memorization

RIJOQ: Attraction in the Maintenance of the Tunjung Language from Threat of Extinction

The Tunjung language in Barong Tongkok District, West Kutai holds a wealth of knowledge and culture of its people. However, during the development, the vitality of Tunjung language has decreased and held an endangered status. Therefore, it needs revitalization efforts, one of the them is through local art of RIJOQ, that is Tunjung language folk song. The purpose of this research is to describe the form of RIJOQ and the strategy of revitalizing the Tunjung language through RIJOQ. Data collection was carried out by interview and literature study methods. Interviews were conducted by structured interviews with respondents based on prepared guidelines. The literature study was carried out using note-taking techniques in the form of RIJOQ lyrics data from documentation in the form of books and social media. The data analysis used a descriptive method with an anthropolinguistic approach. The findings from the results of this study are that the form of poetry and contents of RIJOQ has shifted from traditional to modern; there are extensions and additions to themes, such as romance, affairs, household; using everyday words that are easy to understand; with simpler structure. These findings serve as a reference for the revitalization process of the Tunjung language. The values contained in RIJOQ include belief, wise advice, mutual respect, mutual forgiveness, honesty, patience, and responsibility. The revitalization strategy is to provide space for peRIJOQ or singer of RIJOQ, especially the younger generation, by holding performances in traditional and official events, celebration by holding competitions regularly; provide guidance to RIJOQ groups in order to preserve the Tunjung culture and language; social control over the development of RIJOQ; conduct studies on RIJOQ, and popularize RIJOQ through media, such as radio, television, cd, Youtube, Facebook, Instagram, dan Google Play. Thus, the existence of RIJOQ not only provides a special attraction for the younger generation but also to preserve the Tunjung language

Linking Sheet Music and Audio - Challenges and New Approaches

Score and audio files are the two most important ways to represent, convey, record, store, and experience music. While score describes a piece of music on an abstract level using symbols such as notes, keys, and measures, audio files allow for reproducing a specific acoustic realization of the piece. Each of these representations reflects different facets of music yielding insights into aspects ranging from structural elements (e.g., motives, themes, musical form) to specific performance aspects (e.g., artistic shaping, sound). Therefore, the simultaneous access to score and audio representations is of great importance. In this paper, we address the problem of automatically generating musically relevant linking structures between the various data sources that are available for a given piece of music. In particular, we discuss the task of sheet music-audio synchronization with the aim to link regions in images of scanned scores to musically corresponding sections in an audio recording of the same piece. Such linking structures form the basis for novel interfaces that allow users to access and explore multimodal sources of music within a single framework. As our main contributions, we give an overview of the state-of-the-art for this kind of synchronization task, we present some novel approaches, and indicate future research directions. In particular, we address problems that arise in the presence of structural differences and discuss challenges when applying optical music recognition to complex orchestral scores. Finally, potential applications of the synchronization results are presented

Learning Audio–Sheet Music Correspondences for Cross-Modal Retrieval and Piece Identification

This work addresses the problem of matching musical audio directly to sheet music, without any higher-level abstract representation. We propose a method that learns joint embedding spaces for short excerpts of audio and their respective counterparts in sheet music images, using multimodal convolutional neural networks. Given the learned representations, we show how to utilize them for two sheet-music-related tasks: (1) piece/score identification from audio queries and (2) retrieving relevant performances given a score as a search query. All retrieval models are trained and evaluated on a new, large scale multimodal audio–sheet music dataset which is made publicly available along with this article. The dataset comprises 479 precisely annotated solo piano pieces by 53 composers, for a total of 1,129 pages of music and about 15 hours of aligned audio, which was synthesized from these scores. Going beyond this synthetic training data, we carry out first retrieval experiments using scans of real sheet music of high complexity (e.g., nearly the complete solo piano works by Frederic Chopin) and commercial recordings by famous concert pianists. Our results suggest that the proposed method, in combination with the large-scale dataset, yields retrieval models that successfully generalize to data way beyond the synthetic training data used for model building