In this paper we present algorithms for the automatic time-synchronization of score-, MIDI- or PCM-data streams which represent the same polyphonic piano piece

Frank Kurth

Meinard Müller

Michael Clausen

Vlora Arifi

JScholarship

Automatic Synchronization of Music Data in Score-, MIDI- and PCM-FormatVlora Arifi, Michael Clausen, Frank Kurth, and Meinard MüllerUniversität Bonn, Institut für Informatik IIIRömerstr. 164, D-53117 Bonn, Germany{vlora, clausen, kurth, meinard}@cs.uni-bonn.deAbstractIn this paper we present algorithms for the auto-matic time-synchronization of score-, MIDI- or PCM-data streams representing the same polyphonic pianopiece.1 IntroductionModern digital music libraries consist of large collections ofdocuments containing music data of diverse characteristics andformats. For example, for one and the same piece of music, thelibrary may contain the corresponding score in Capella or Scoreformat, some MIDI-files, and several interpretations in form ofCD recordings. Inhomogeneity and complexity of such musicdata make content-based browsing and retrieval in digital mu-sic libraries a difficult task with many yet unsolved problems.One important step towards a solution are synchronization algo-rithms which automatically link data streams of different dataformats representing a similar kind of information. In partic-ular, in the framework of audio by synchronization we meansome procedure which, for a given position in some represen-tation of a given piece of music (e.g., given in score format),determines the corresponding position within some other repre-sentation (e.g., given in PCM-format).Such synchronization algorithms have applications in many dif-ferent scenarios: following some score-based music retrieval,linking structures can be used to access some suitable audioCD accurately to listen to the desired part of the interpretation.A further application is the automatic annotation of a piece ofmusic in different data formats as a basis for content-based re-trieval. As another example, musicologists can use synchro-nization algorithms for the investigation of agogic and tempostudies. Furthermore, temporal linking of score and audio datacan be useful for automatic tracking of the score positions dur-ing a performance.In our work, we concentrate on three representative data for-mats used for music data: the symbolic score format, the phys-ical PCM-format and the MIDI-format which may be thoughtPermission to make digital or hard copies of all or part of this workfor personal or classroom use is granted without fee provided thatcopies are not made or distributed for profit or commercial advan-tage and that copies bear this notice and the full citation on the firstpage. c©2003 Johns Hopkins University.of as a hybrid of the last two data formats. We have devel-oped synchronization algorithms for two data streams in any ofthese three data formats which will be referred to as Score-to-MIDI (SM) synchronization, Score-to-PCM (SP) synchroniza-tion, and MIDI-to-PCM (MP) synchronization. Especially, SP-and MP-synchronization constitute a difficult problem since thewaveform-based PCM-format does not contain any explicit in-formation on the notes. Therefore, in these cases some score-like parameters such as onset times and pitches have to be ex-tracted from the PCM-data prior to the actual synchronization.Due to space limitations, we are not able to give an overview ofthe related work. Links to the relevant literature can be foundin the full paper version Arifi et al. (2003a) available on ourwebsite or in Arifi et al. (2003b).2 Feature ExtractionIn this section we summarize our system for extracting note pa-rameters from the PCM-data stream. Using several establishedtools from audio signal processing, our main contributions area refined template matching algorithm for polyphonic pitch ex-traction and a two-step algorithm for note onset detection.AudioGenerate notepitches andonset positionsparameters fromSubbandanalysisComputation ofenergy vectorsPitch extractionvia templatematchingnovelity curve:onset detectionMap onset positionsvia LPC methodRefined onset detectionto subband positionsComputation ofobjectsnoteonsetspitchesdata baseTemplatesignalFigure 1: Diagram of the feature extraction algorithm.Figure 1 shows the overall feature extraction algorithm. Similarto Bobrek et al. (1998) an input PCM-signal is transformed to asubband representation using a multirate filter bank. Simultane-ously, a two-stage peak-picking algorithm detects probable noteonset positions. According to those onset positions, the subbandrepresentation is split into time intervals. For each interval, wecalculate an energy vector with components corresponding tothe subbands: each component contains the total energy withinthe interval of the respective subband. Then, for each energyvector a pitch extraction based on a template matching algo-rithm is performed. The pitch extraction yields a set of notesfor the corresponding time interval. The feature extraction al-gorithm outputs a note object for each note in each time interval,brought to you by COREView metadata, citation and similar papers at core.ac.ukprovided by JScholarshipwhere a note object consists of an onset time and a pitch infor-mation. For implementation details of the overall system werefer to Arifi (2002).3 Synchronization AlgorithmsIn this section we describe the actual synchronization algo-rithms. Here, we only consider the case of a score- and a PCM-data stream (SP-synchronization). The other cases such as SM-or MP-synchronization are even easier or can be done in a sim-ilar fashion (see Arifi (2002)).We first preprocess the score data stream where we distin-guish between two kinds of note objects: explicit and implicitnotes. For explicit objects all note parameters such as measure,beat, duration, and pitch are given explicitly. In view of thesynchronization algorithm we only use the musical onset timeand pitch. We represent each explicit note object by a tuple(e, p) ∈ Q × [0 : 127], where the rational number e encodesthe onset time and the integer p the pitch of the correspondingnote. By an implicit note object we understand notes or a groupof notes with some additional specification such as a trill, anarpeggio or grace notes. Implicit objects allow different real-izations. To get this ambiguity under control we introduce theconcept of a fuzzy note, which is defined to be a tuple (e, H)consisting of some onset time e ∈ Q and some set of alternativepitches H ⊂ [0 : 127]. In this model, the preprocessed score isgiven by some subset S ⊂ Q× 2[0:127]× 2[0:127], where 2[0:127]denotes the set of all subsets of [0 : 127]. Here, in a triple(e, H0, H1) ∈ S the subset H0 ⊂ 2[0:127] consists of all pitchesof explicit note objects having musical onset time e and simi-larly the subset H1 ⊂ 2[0:127] consists of all pitches of implicitnote objects.Next, the extracted note parameters from the PCM data stream(see Section 2) are further preprocessed by quantizing the onsettime candidates with some suitable chosen quantizer resolution∆ > 0. The resulting data stream is denoted by P∆ which canbe regarded as a subset of Q × 2[0:127]. Note that in the PCM-case there are only explicit note objects.Altogether, we may assume that the score and the ∆-quantized extracted PCM-data are given by the setsS = [(s1, S01, S11), . . . , (ss, S0s, S1s)] and P∆ =[(p1, P01), . . . , (pp, P0p)]. Here, the si, 1 ≤ i ≤ s, denotethe musical onset times and the pj , 1 ≤ j ≤ p, the quantizedphysical onset times. Furthermore, S0i, S1i, P0j ⊂ [0 : 127]are the respective sets of pitches for the explicit and implicitobjects.We now accomplish the SP-synchronization by matching thesets S and P∆ in the following sense.Definition 3.1. A score-PCM-match (SP-match) of S and P∆ isdefined to be a partial map µ : [1 : s] → [1 : p], which is strictlymonotonously increasing on its domain satisfying (S0i ∪S1i)∩P0µ(i) 6= ∅ for all i ∈ Domain(µ).This definition needs some explanations. The fact that objectsin S or P∆ may not have a counterpart in the other data streamis modeled by the requirement that µ is only a partial functionand not a total one. The monotony of µ reflects the requirementof faithful timing: if a note in S precedes a second one thisalso should hold for the µ-images of these notes. Finally, therequirement (S0i ∪ S1i) ∩ P0µ(i) 6= ∅ prevents that onset timesare linked which are completely unrelated with respect to theirpitches.Obviously, there are many possible SP-matches between Sand P∆. However, introducing a cost function makes differ-ent matches comparable. For an explicit formula of our costfunction, which is crucial in view of the quality of the solu-tion for the synchronization problems, we have to refer to Ar-ifi et al. (2003a). Summarizing, our cost function penalizesnon-matched explicit and implicit note objects in S (where theweighting of the penalty depends on the type of note objects),non-matched note-objects in P∆, as well as large relative onsettime deviations thus preventing large global deviations in thesynchronization.The solution of the SP-synchronization is then realized by anoptimal SP-match minimizing the cost function. This optimalmatch can be efficiently computed by means of dynamic pro-gramming.4 Experimental Results and ConclusionsWe have implemented a prototype of the extraction algorithmsfrom Section 2 and the synchronization algorithms in the MAT-LAB programming language and tested our algorithms for SM-,SP-, and MP-synchronization on a variety of classical poly-phonic piano pieces of different complexity and length (rangingfrom 10 to 60 seconds) played on various instruments. Fur-thermore, we have systematically generated a library of morethan one hundred test pieces both in MIDI- and PCM-formatplayed on a MIDI-piano, a Steinway grand piano, and a Schim-mel piano. In some of those pieces our performer has delib-erately built in excessive accelerandi, ritartandi, rhythmic dis-tortions, and wrong notes. Even in these extreme situations,where one unsurprisingly has many “erroneously” extractednote objects which considerably differ from the score-data, ourSP-synchronization algorithm resulted in good overall globalmatches which are sufficient for the applications mentioned inthe introduction. Even more, in case of rather accurate extractednote parameters our synchronization algorithms could resolvesubtle local time variations in some interpreted version of thepiano piece. For further details and results of our experimentswe refer to Arifi (2002).ReferencesArifi, V. Algorithmen zur Synchronisation von Musikdatenim Paritur-, MIDI- und PCM-Format. PhD thesis, Uni-versität Bonn, Institut für Informatik (2002). Retriev-able from http://hss.ulb.uni-bonn.de:90/ulb bonn/diss online/math nat fak/2002/arifi vlora/Arifi, V., Clausen, M., Kurth, F., Müller, M. (2003a).Automatic Synchronization of Music Data in Score-,MIDI-, and PCM-Format. Technical Report, Univer-sität Bonn, Institut für Informatik. Retrievable fromhttp://www-mmdb.iai.uni-bonn.de/eng-public.htmlArifi, V., Clausen, M., Kurth, F., Müller, M. (2003b). Auto-matic Synchronization of Music Data. To appear in Computingin Musicology, MIT Press.Bobrek, M., Koch, D. (1998). Music Signal Segmentation Us-ing Tree-Structured Filter Banks. Journal of Audio EngineeringSociety, 46(5), 412–427.

Automatic Synchronization of Music Data in Score-, MIDI- and PCM-Format

In this paper we present algorithms for the automatic time-synchronization of score-, MIDI- or PCM-data streams representing the same polyphonic piano piece

CiteSeerX

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.9.6707

Automatic Synchronization of Music Data in Score-, MIDI- and PCM-Format

Abstract

Similar works

Full text

Available Versions

JScholarship

CiteSeerX