An XML-based Format for Advanced Music Fruition

(Abstract to follow

Segments and Mapping for Scores and Signal Representations

We present a general theoretical framework to describe segments and the different possible mapping that can be established between them. Each segment can be related to different music representations, graphical scores, music signals or gesture signals. This theoretical formalism is general and is compatible with large number of problems found in sound and gesture computing. We describe some examples we developed in interactive score representation, superposed with signal representation, and the description of synchronization between gesture and sound signals

Sound and the City: Multi-Layer Representation and Navigation of Audio Scenarios

IEEE 1599-2008 is an XML-based standard originally intended for the multi-layer representation of music information. Nevertheless, it is versatile enough to describe also information different from traditional scores written according to the Common Western Notation (CWN) rules. This paper will discuss the application of IEEE 1599-2008 to the audio description of paths and scenarios from the urban life or other landscapes. The standard we adopt allows the multilayer integration of textual, symbolical, structural, graphical, audio and video contents within a unique synchronized environment. Besides, for each kind of media, a number of digital objects is supported. As a consequence, thanks to the features of the format the produced description will be more than a mere audio track, a slideshow made of sonified static images or a movie. Finally, an ad hoc evolution of a standard viewer for IEEE 1599 documents will be presented, in order to enjoy the results of our efforts

On the Adoption of Standard Encoding Formats to Ensure Interoperability of Music Digital Archives: The IEEE 1599 Format

With this paper, we want to stimulate the discussion about technologies for inter-operation between various music datasets and collections. Among the many standards for music representation, IEEE 1599 is the only one which was born with the exact purpose of representing the heterogeneous structures of music documents, granting full synchronization of all the different aspects of music (audio recordings, sheet music images, symbolic representations, musicological analysis, etc). We propose the adoption of IEEE 1599 as an interoperability framework between different collections for advanced music experience, musicological applications, and Music Information Retrieval (MIR). In the years to come, the format will undergo a review process aimed at providing an updated/improved version. It is now the perfect time, for all the stakeholders, to come together and discuss how the format can evolve to better support their requirements, enhancing its descriptive strength and available tools. Moreover, this standard can be profitably applied to any field that requires multi-layer and synchronized descriptions

Music Synchronization, Audio Matching, Pattern Detection, and User Interfaces for a Digital Music Library System

Over the last two decades, growing efforts to digitize our cultural heritage could be observed. Most of these digitization initiatives pursuit either one or both of the following goals: to conserve the documents - especially those threatened by decay - and to provide remote access on a grand scale. For music documents these trends are observable as well, and by now several digital music libraries are in existence. An important characteristic of these music libraries is an inherent multimodality resulting from the large variety of available digital music representations, such as scanned score, symbolic score, audio recordings, and videos. In addition, for each piece of music there exists not only one document of each type, but many. Considering and exploiting this multimodality and multiplicity, the DFG-funded digital library initiative PROBADO MUSIC aimed at developing a novel user-friendly interface for content-based retrieval, document access, navigation, and browsing in large music collections. The implementation of such a front end requires the multimodal linking and indexing of the music documents during preprocessing. As the considered music collections can be very large, the automated or at least semi-automated calculation of these structures would be recommendable. The field of music information retrieval (MIR) is particularly concerned with the development of suitable procedures, and it was the goal of PROBADO MUSIC to include existing and newly developed MIR techniques to realize the envisioned digital music library system. In this context, the present thesis discusses the following three MIR tasks: music synchronization, audio matching, and pattern detection. We are going to identify particular issues in these fields and provide algorithmic solutions as well as prototypical implementations. In Music synchronization, for each position in one representation of a piece of music the corresponding position in another representation is calculated. This thesis focuses on the task of aligning scanned score pages of orchestral music with audio recordings. Here, a previously unconsidered piece of information is the textual specification of transposing instruments provided in the score. Our evaluations show that the neglect of such information can result in a measurable loss of synchronization accuracy. Therefore, we propose an OCR-based approach for detecting and interpreting the transposition information in orchestral scores. For a given audio snippet, audio matching methods automatically calculate all musically similar excerpts within a collection of audio recordings. In this context, subsequence dynamic time warping (SSDTW) is a well-established approach as it allows for local and global tempo variations between the query and the retrieved matches. Moving to real-life digital music libraries with larger audio collections, however, the quadratic runtime of SSDTW results in untenable response times. To improve on the response time, this thesis introduces a novel index-based approach to SSDTW-based audio matching. We combine the idea of inverted file lists introduced by Kurth and Müller (Efficient index-based audio matching, 2008) with the shingling techniques often used in the audio identification scenario. In pattern detection, all repeating patterns within one piece of music are determined. Usually, pattern detection operates on symbolic score documents and is often used in the context of computer-aided motivic analysis. Envisioned as a new feature of the PROBADO MUSIC system, this thesis proposes a string-based approach to pattern detection and a novel interactive front end for result visualization and analysis

Music Description and Processing: An Approach Based on Petri Nets and XML

Music description and processing require formal tools which are suitable for the representation of iteration, concurrency, ordering, hierarchy, causality, timing, synchrony, non-determinism. Petri Nets are a tool which allows to describe and process musical objects within both analysis/composition and performing environments. To accomplish this objective, a specific extension known as Music Petri Nets was developed

Managing Multiple Media Streams in HTML5: The IEEE 1599-2008 Case Study

This paper deals with the problem of managing multiple multimedia streams in a Web environment. Multimedia types to support are pure audio, video with no sound, and audio/video. Data streams refer to the same event or performance, consequently they both have and should maintain mutual synchronization. Besides, a Web player should be able to play different multimedia streams simultaneously, as well as to switch from one to another in real time. The clarifying example of a music piece encoded in IEEE 1599 format will be presented as a case study

A Web-Oriented Multi-layer Model to Interact with Theatrical Performances

This paper presents an innovative approach to online fruition of theater performances. Web applications like traditional viewers are already available for the wide audience of Internet users. Our proposal aims at adding both interactivity and multi-layer fruition, and a way to manipulate and create new media. The premise to reach these goals is digitizing a number of heterogeneous materials in order to describe a single performance comprehensively, e.g. different video and audio-takes from different perspectives, and a number of related materials such as scripts, fashion plates, playbills, etc. The format we adopt to encode such information is based on the XML international standard known as IEEE 1599. Finally, an advanced Web player supporting search and play functions for synchronized materials must be designed. This work describes the whole process, from the acquisition of materials directly on the stage to their publishing on a Web portal

Development of serious games for music education

Serious games have proved to be an effective educational tool in many fields. The first goal of this paper is to illustrate some possible applications to music and their advantages. Moreover, music can be characterized by heterogeneous multimedia contents. Among the different facets music information is made of it is worth citing music symbols, their graphical representations as scores, their audio renderings as tracks, etc. The international standard known as IEEE 1599 is an XML multilayer format for heterogeneous music contents, and describes such different aspects in an integrated and synchronized context. Making relationships among music contents explicit provides a potentially rich educational environment. Consequently, this paper discusses the concept of multilayer learning object, introduces the IEEE 1599 standard, and finally shows some applications and case studies