Optical Music Recognition with Convolutional Sequence-to-Sequence Models

Optical Music Recognition (OMR) is an important technology within Music Information Retrieval. Deep learning models show promising results on OMR tasks, but symbol-level annotated data sets of sufficient size to train such models are not available and difficult to develop. We present a deep learning architecture called a Convolutional Sequence-to-Sequence model to both move towards an end-to-end trainable OMR pipeline, and apply a learning process that trains on full sentences of sheet music instead of individually labeled symbols. The model is trained and evaluated on a human generated data set, with various image augmentations based on real-world scenarios. This data set is the first publicly available set in OMR research with sufficient size to train and evaluate deep learning models. With the introduced augmentations a pitch recognition accuracy of 81% and a duration accuracy of 94% is achieved, resulting in a note level accuracy of 80%. Finally, the model is compared to commercially available methods, showing a large improvements over these applications.Comment: ISMIR 201

Recognition of handwritten music scores

The recognition of handwritten music scores still remains an open problem. The existing approaches can only deal with very simple handwritten scores mainly because of the variability in the handwriting style and the variability in the composition of groups of music notes (i.e. compound music notes). In this work on the one hand I study the isolated symbols (i.e half-note, quarter-note, clefs, sharps) and on the other hand the compound music notes. Firstly, I will separate the isolated symbols (i.e half-notes, quarter-notes, clefs, sharps) to the compounds and I will study each one separately. The isolated symbols will be recognized with symbol recognition methods and compounds with a primitive hierarchy and syntactic rules. The method has been tested using several handwritten music scores of the CVC-MUSCIMA database and compared with a commercial Optical Music Recognition software. Given that my method is learning-free, the obtained results are promising.El reconeixement de partitures musicals manuscrites segueix sent un problema obert. Els enfocaments existents només poden reconéixer partitures manuscrites molt simples, principalment a causa de la variabilitat en l'estil d'escriptura i la variabilitat en la composició dels grups de notes musicals (p.e. els símbols musicals compostos). En aquest treball, per començar, se separaran els símbols simples (p.e blanques, negres, claus, sostinguts) dels compostos i els estudiaré per separat. Els símbols simples mitjançant mètodes de reconeixement de símbols i els compostos a partir d'una jerarquia de primitives i regles sintàctiques. El meu mètode ha estat provat utilitzant diferents partitures de música escrita a mà de la base de dades CVC-MUSCIMA i comparat amb un programari de reconeixement òptic musical comercial. Tenint en compte que el meu mètode és d'aprenentatge lliure, els resultats obtinguts són prometedors.El reconocimiento de partituras musicales manuscritas sigue siendo un problema abierto. Los enfoques existentes sólo pueden reconocer partituras manuscritas muy simples, principalmente debido a la variabilidad en el estilo de escritura y la variabilidad en la composición de los grupos de notas musicales (p.e. los símbolos musicales compuestos). En este trabajo, para empezar, se separarán los símbolos simples (p.e blancas, negras, llaves, sostenidos) de los compuestos y los estudiaré por separado. Los símbolos simples mediante métodos de reconocimiento de símbolos y los compuestos a partir de una jerarquía de primitivas y reglas sintácticas. Mi método ha sido probado utilizando diferentes partituras de música escrita a mano de la base de datos CVC-MUSCIMA y comparado con un software de reconocimiento óptico musical comercial. Teniendo en cuenta que mi método es de aprendizaje libre, los resultados obtenidos son prometedores

Recognition of online handwritten music symbols

Paper submitted to MML 2013, 6th International Workshop on Machine Learning and Music, Prague, September 23, 2013.An effective way of digitizing a new musical composition is to use an e-pen and tablet application in which the user's pen strokes are recognized online and the digital score is created with the sole effort of the composition itself. This work aims to be a starting point for research on the recognition of online handwritten music notation. To this end, different alternatives within the two modalities of recognition resulting from this data are presented: online recognition, which uses the strokes marked by a pen, and offline recognition, which uses the image generated after drawing the symbol. A comparative experiment with common machine learning algorithms over a dataset of 3800 samples and 32 different music symbols is presented. Results show that samples of the actual user are needed if good classification rates are pursued. Moreover, algorithms using the online data, on average, achieve better classification results than the others

Finding What You Need, and Knowing What You Can Find: Digital Tools for Palaeographers in Musicology and Beyond

This chapter examines three projects that provide musicologists with a range of resources for managing and exploring their materials: DIAMM (Digital Image Archive of Medieval Music), CMME (Computerized Mensural Music Editing) and the software Gamera. Since 1998, DIAMM has been enhancing research of scholars worldwide by providing them with the best possible quality of digital images. In some cases these images are now the only access that scholars are permitted, since the original documents are lost or considered too fragile for further handling. For many sources, however, simply creating a very high-resolution image is not enough: sources are often damaged by age, misuse (usually Medieval ‘vandalism’), or poor conservation. To deal with damaged materials the project has developed methods of digital restoration using mainstream commercial software, which has revealed lost data in a wide variety of sources. The project also uses light sources ranging from ultraviolet to infrared in order to obtain better readings of erasures or material lost by heat or water damage. The ethics of digital restoration are discussed, as well as the concerns of the document holders. CMME and a database of musical sources and editions, provides scholars with a tool for making fluid editions and diplomatic transcriptions: without the need for a single fixed visual form on a printed page, a computerized edition system can utilize one editor’s transcription to create any number of visual forms and variant versions. Gamera, a toolkit for building document image recognition systems created by Ichiro Fujinaga is a broad recognition engine that grew out of music recognition, which can be adapted and developed to perform a number of tasks on both music and non-musical materials. Its application to several projects is discussed

Understanding Optical Music Recognition

For over 50 years, researchers have been trying to teach computers to read music notation, referred to as Optical Music Recognition (OMR). However, this field is still difficult to access for new researchers, especially those without a significant musical background: Few introductory materials are available, and, furthermore, the field has struggled with defining itself and building a shared terminology. In this work, we address these shortcomings by (1) providing a robust definition of OMR and its relationship to related fields, (2) analyzing how OMR inverts the music encoding process to recover the musical notation and the musical semantics from documents, and (3) proposing a taxonomy of OMR, with most notably a novel taxonomy of applications. Additionally, we discuss how deep learning affects modern OMR research, as opposed to the traditional pipeline. Based on this work, the reader should be able to attain a basic understanding of OMR: its objectives, its inherent structure, its relationship to other fields, the state of the art, and the research opportunities it affords

Handwriting styles: benchmarks and evaluation metrics

Evaluating the style of handwriting generation is a challenging problem, since it is not well defined. It is a key component in order to develop in developing systems with more personalized experiences with humans. In this paper, we propose baseline benchmarks, in order to set anchors to estimate the relative quality of different handwriting style methods. This will be done using deep learning techniques, which have shown remarkable results in different machine learning tasks, learning classification, regression, and most relevant to our work, generating temporal sequences. We discuss the challenges associated with evaluating our methods, which is related to evaluation of generative models in general. We then propose evaluation metrics, which we find relevant to this problem, and we discuss how we evaluate the evaluation metrics. In this study, we use IRON-OFF dataset. To the best of our knowledge, there is no work done before in generating handwriting (either in terms of methodology or the performance metrics), our in exploring styles using this dataset.Comment: Submitted to IEEE International Workshop on Deep and Transfer Learning (DTL 2018