Multimodal music information processing and retrieval: survey and future challenges

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, mid-level representations, motion, and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years

Multimodal Music Information Processing and Retrieval: Survey and Future Challenges

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, mid-level representations, motion and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval, and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years

Score informed tonic identification for Makam music of Turkey

Tonic is a fundamental concept in many music traditions/nand its automatic identification should be relevant for establishing/nthe reference pitch when we analyse the melodic/ncontent of the music. In this paper, we present two methodologies/nfor the identification of the tonic in audio recordings/nof makam music of Turkey, both taking advantage/nof some score information. First, we compute a prominent/npitch and a audio kernel-density pitch class distribution/n(KPCD) from the audio recording. The peaks in the/nKPCD are selected as tonic candidates. The first method/ncomputes a score KPCD from the monophonic melody extracted/nfrom the score. Then, the audio KPCD is circularshifted/nwith respect to each tonic candidate and compared/nwith the score KPCD. The best matching shift indicates the/nestimated tonic. The second method extracts the monophonic/nmelody of the most repetitive section of the score./nNormalising the audio prominent pitch with respect to each/ntonic candidate, the method attempts to link the repetitive/nstructural element given in the score with the respective/ntime-intervals in the audio recording. The result producing/nthe most confident links marks the estimated tonic./nWe have tested the methods on a dataset of makam music/nof Turkey, achieving a very high accuracy (94.9%) with/nthe first method, and almost perfect identification (99.6%)/nwith the second method. We conclude that score informed/ntonic identification can be a useful first step in the computational/nanalysis (e.g. expressive analysis, intonation analysis,/naudio-score alignment) of music collections involving/nmelody-dominant content.This work is partly supported by the European Research/nCouncil under the European Union’s Seventh Framework/nProgram, as part of the CompMusic project (ERC grant/nagreement 267583)

Score informed tonic identification for Makam music of Turkey

Tonic is a fundamental concept in many music traditions/nand its automatic identification should be relevant for establishing/nthe reference pitch when we analyse the melodic/ncontent of the music. In this paper, we present two methodologies/nfor the identification of the tonic in audio recordings/nof makam music of Turkey, both taking advantage/nof some score information. First, we compute a prominent/npitch and a audio kernel-density pitch class distribution/n(KPCD) from the audio recording. The peaks in the/nKPCD are selected as tonic candidates. The first method/ncomputes a score KPCD from the monophonic melody extracted/nfrom the score. Then, the audio KPCD is circularshifted/nwith respect to each tonic candidate and compared/nwith the score KPCD. The best matching shift indicates the/nestimated tonic. The second method extracts the monophonic/nmelody of the most repetitive section of the score./nNormalising the audio prominent pitch with respect to each/ntonic candidate, the method attempts to link the repetitive/nstructural element given in the score with the respective/ntime-intervals in the audio recording. The result producing/nthe most confident links marks the estimated tonic./nWe have tested the methods on a dataset of makam music/nof Turkey, achieving a very high accuracy (94.9%) with/nthe first method, and almost perfect identification (99.6%)/nwith the second method. We conclude that score informed/ntonic identification can be a useful first step in the computational/nanalysis (e.g. expressive analysis, intonation analysis,/naudio-score alignment) of music collections involving/nmelody-dominant content.This work is partly supported by the European Research/nCouncil under the European Union’s Seventh Framework/nProgram, as part of the CompMusic project (ERC grant/nagreement 267583)

OTMM Tonic Test Dataset v3.0.0

Each annotated recording is uniquely identified with a MusicBrainz MBID. The tonic symbol is also for each recording given in the format [letter][octave][accidental][comma], e.g. B4b1 (according to AEU theory)./n/nEach recording is annotated by at least expert musician or musicologists. The annotations are stored as a list with each annotation including the annotated frequency, source dataset, relevant publication, additional observations written by the annotator and whether the octave of the annotated value is considered (for example, the octave is ambiguous in orchestral instrumental recordings)./nThe data is stored as JSON file and organized as a dictionary of recordings. An example recording is displayed below:/n/n```json/n{/n "mbid": "http://musicbrainz.org/recording/e3a22684-d237-48b5-ac27-e9b77ddd3c18", /n "verified": true, /n "annotations": [/n {/n "source": "https://github.com/MTG/otmm_tonic_dataset/blob/7f28c1a3261b9146042155ee5e0f9e644d9ebcfa/senturk2013karar_ismir/tonic_annotations.csv", /n "citation": "Şentürk, S., Gulati, S., and Serra, X. (2013). Score Informed Tonic Identification for Makam Music of Turkey. In Proceedings of 14th International Society for Music Information Retrieval Conference (ISMIR 2013), pages 175–180, Curitiba, Brazil.", /n "octave_wrapped": true, /n "observations": "", /n "value": 296.9597/n }, /n {/n "source": "https://github.com/MTG/otmm_tonic_dataset/blob/7f28c1a3261b9146042155ee5e0f9e644d9ebcfa/atli2015tonic_fma/TD2.csv", /n "citation": "Atlı, H. S., Bozkurt, B., Şentürk, S. (2015). A Method for Tonic Frequency Identification of Turkish Makam Music Recordings. In Proceedings of 5th International Workshop on Folk Music Analysis (FMA 2015), pages 119–122, Paris, France.", /n "octave_wrapped": true, /n "observations": "", /n "value": 296/n }/n ], /n "tonic_symbol": "D4"/n}/n```/n/n/nMost of the recordings in this dataset cannot be shared due to copyright. However relevant features are already computed and they can be downloaded from the [Dunya-makam](dunya.compmusic.upf.edu/makam) after registration. Please refer to the API documentation (http://dunya.compmusic.upf.edu/docs/makam.html) to how to access the data.This repository contains datasets of annotated tonic frequencies of the audio recordings of Ottoman-Turkish makam music. The annotations are compiled from several research papers published under the CompMusic project. For more information about the original datasets, please refer to the relevant paper