A simple framework for evaluating OMR at the symbol
level is presented. While a true evaluation of an OMR
system requires a high-level analysis, the automation of
which is a largely unsolved problem, many high-level errors
are correlated to these more tractably-analyzed lowerlevel
errors

Droettboom, Michael

Fujinaga, Ichiro

JScholarship

SYMBOL-LEVEL GROUNDTRUTHING ENVIRONMENT FOR OMRMichael DroettboomDigital Knowledge Center, Sheridan LibrariesThe Johns Hopkins University3400 North Charles StreetBaltimore, MD 21218, USAIchiro FujinagaCIRMMT, Faculty of MusicMcGill University555 Sherbrooke Street WestMontréal, QC H3A 1E3, CanadaABSTRACTA simple framework for evaluating OMR at the symbollevel is presented. While a true evaluation of an OMRsystem requires a high-level analysis, the automation ofwhich is a largely unsolved problem, many high-level er-rors are correlated to these more tractably-analyzed lower-level errors.1. INTRODUCTIONComplex document understanding systems are difficult toevaluate [5]. On the other hand, pixel-level and symbol-level analysis is much more feasible to implement and stillquite useful for improving the accuracy of document un-derstanding systems [11].We demonstrate that, at least at the level of symbolrecognition, the test driven software development tech-nique [1] can be used to develop robust optical music recog-nition (OMR) systems. We argue that developing a perfor-mance evaluation metric for a pattern recognition softwaresystem is similar to the test-driven software developmentmethodology. The goal in both cases is to minimize er-rors.In many classification problems the evaluation metricis fairly straightforward. For example at the characterlevel of OCR, it is simply a matter of finding the ratio be-tween correctly identified characters and the total numberof characters. In other classification domains, this is notso simple, for example document segmentation [7], recog-nition of maps [4], mathematical equations [9], graphi-cal drawings, and music scores. In these domains, thereare often multiple correct output representations, whichmakes the problem of comparing a given output to high-level groundtruth very difficult [5]. In fact, it could be ar-gued that a complete and robust system to evaluate OMRoutput would be almost as complex and error-prone as anOMR system itself.Symbol-level analysis may not be directly suitable forcomparing commercial OMR products [2, 8], because suchPermission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copiesare not made or distributed for profit or commercial advantage and thatcopies bear this notice and the full citation on the first page.c© 2004 Universitat Pompeu Fabra.systems are usually “black boxes” that take an image asinput and produce a score-level representation as output.Nevertheless, higher-level evaluations such as [8] are of-ten correlated to lower-level errors, thus studying the ef-fects of lower-level errors to higher-level errors with open-source systems should provide insights for evaluating com-mercial products.2. EVALUATIONSTesting is useful for developing new approaches to recog-nition, comparing the relative performances (error rate,efficiency) of alternative approaches and validating incre-mental modifications to algorithms (e.g. regression test-ing). In general, there are three types of evaluation strate-gies [3]: 1) symbol-level evaluation, 2) combination ofsymbol-level evaluations, and 3) an edit cost function.1The symbol-level evaluation involves calculating therecognition rate of basic graphic symbols, which may beline segments, arcs, or characters. Sophisticated metricswill refine the recognition rate by counting separately thecorrectly identified symbols, missed symbols, and falsepositive (added) symbols.In many document-understanding systems, includingOMR, metrics are required at higher-levels. This is oftencalculated as some linear combination of weighted lower-level errors. The assignments of weights are non-trivial,however [10]. Ng, et al., 2003 [8] provides a first attemptat developing evaluation weights for OMR. However, asmusic is perceptual, some errors are “more important thanothers” and can rely on levels of complex context. There-fore any fully complete evaluation metric must rely, atleast on some level, on human studies.Another popular metric for the evaluation of complexdocument understanding systems is to calculate the costof manually correcting and redrawing graphical objects.This is achieved by calculating the editing time requiredby empirically finding the times for operations such asmouse clicks [12]. Once the relationship between the symbol-level errors and the type of operations required to correctthem is established, this metric can be calculated automat-ically.1 The term “low-level” used in [3] has been changed to “symbol-level” for better clarity.To sum up, in all cases, symbol-level error analysis isrequired.3. SYMBOL-LEVEL ANALYSISThe goal of our symbol-level analysis system is to evalu-ate the performance of our OMR system, GAMUT (Gamera-based Adaptive Music Understanding Tool)2 on very smallscore features, before the symbolic representation of thescore has been assembled. Some of these are:• assignment of accidentals to notes• combination of clefs and accidentals into key signa-tures• construction of time signatures from individual dig-its• number of beams in each beam complex• exact location of noteheads relative to the staff andledger lines• identification and disambiguation of dots (augmen-tation,staccato, bass clef,fermata, etc.)• interpreting repeat,Da capoand other “flow-control”markings• recognition of tuplets (triplets, quintuplets etc.)It is clear from each of these cases that each example canbe evaluated cleanly, either strictly correct or incorrect,rather than having multiple correct representations or agraded scale of correctness.The system works by generating result tables, one foreach of the evaluation categories, with three columns:1. the subject of the test, highlighted with some sur-rounding context2. GAMUT’s interpretation of the excerpt3. the groundtruthIf the groundtruth is incorrect, it may be edited and sub-mitted back to the system for re-evaluation.Figures 1 and 2 are examples of these evaluation tables.This approach, while clearly very primitive, has alreadybeen very useful for improving the accuracy of GAMUT.For instance, when the test system was first implemented,the number of correctly pitched noteheads was around 79%.The test system allowed us to improve the algorithm to thepoint where it now operates at 96%. It is also very easy toadd new test categories as the need arises.2 In previous incarnations, this system was known as AOMR (Adap-tive Optical Music Recognition), which has since become a generic term.Figure 1. Excerpt of the table showing the performance ofaugmentation dots. The first three examples do not matchthe groundtruth, since they are not genuine augmentationdots.Figure 2. Excerpt of the table showing beam interpreta-tion. The first four elements do not match the groundtruth,which is a list list of numbers representing the number ofbeams at each point along the set of beams.4. CONCLUSIONSAs research domains mature, evaluation metrics becomemore important. As the MIR community grows, effortsshould be made to create metrics and groundtruth datasets.We presented here a simple but fundamental model forevaluating OMR systems and different algorithms withinone OMR system. Automatically comparing the final sym-bolic outputs of OMR systems at the high-level is stilla largely unsolved problem, and one which we hope theMIR community will consider as an application of recentdevelopments in the searching and analysis of symbolicmusic data.5. REFERENCES[1] Beck, K., 2002. Test driven development.Boston: Addison-Wesley.[2] Bruno, I., P. Bellini, and P. Nesi. 2003. As-sessing optical music recognition tools. Paperpresented atSecond MUSICNETWORK OpenWorkshop.[3] Chhabra, A., and I. Phillips. 1998. A bench-mark for graphics recognition systems.Pro-ceedings of IEEE Workshop on EmpiricalEvaluation Methods in Computer Vision.[4] Hinz, S., C. Wiedemann, and A. Baumgartner.2000. A scheme for read extraction in rural ar-eas and its evaluation.Proceedings of the IEEEWorkshop on Applications of Computer Vision,134–9.[5] Hu, J., R. Kashi, D. Lopresti, G. Nagy, and G.Wilfong. 2001. Why table ground-truthing ishard. International Conference on DocumentAnalysis and Recognition (ICDAR),129–33.[6] Kanai, J., S. Rice, T. Nartker, and G. Nagy.1995. Automated evaluation of OCR zoning.IEEE Transactions on Pattern Analysis andMachine Intelligence,17: 86–90.[7] Mao, S., and T. Kanungo. 2001. Empiricalperformance evaluation methodology and itsapplication to page segmentation algorithms.IEEE transactions on Pattern Analysis andMachine Intelligence,23(3): 242–56.[8] Ng, K., and A. Jones. 2003. A quick-testfor optical music recognition systems. Paperpresented atSecond MUSICNETWORK OpenWorkshop.[9] Okamoto, M., H. Imai, and K. Takagi. 2001.Performance evaluation of a robust methodfor mathematical expression recognition.Pro-ceedings of the International Conference onDocument Analysis and Recognition,121–8.[10] Randriamasy, S. and L. Vincent. 1994. Bench-marking and page segmentation algorithms.Proceedings of the IEEE Conference on Com-puter Vision and Pattern Recognition,411–6.[11] Wenyin, L., and D. Dori. 1997. A protocol forperformance evaluation of line detection algo-rithms. Machine Vision and Applications,9:240–50.[12] Wenyin, L., Z. Liang, T. Long, and D.Dori. 1999. Cost evaluation of interactivelycorrecting recognized engineering drawings.Proceedings of IAPR Workshop on GraphicsRecognition,335–40.

Symbol Level Groundtruthing Environment in OMR

https://jscholarship.library.jhu.edu/bitstream/1774.2/44383/1/2004.%20Symbol%20Level%20Groundtruthing%20Environment%20in%20OMR.pdf

Symbol Level Groundtruthing Environment in OMR

Abstract

Similar works

Full text

Available Versions

JScholarship