Since only simple symbol-based manipulations are needed, n-gram indexing is used for natural languages where syntactic or semantic analyses are often difficult. Music, whose automatic analysis for patterns such as motifs and phrases are difficult, inaccurate or computationally expensive, is thus similar to natural languages. The use of n-gram in music retrieval systems is thus a natural choice. In this paper, we study a number of issues regarding n-gram indexing of musical features using simulated queries. They are: whether combinatorial explosion is a problem in n-gram indexing of musical features, the relative discrimination power of six different musical features, the value of n needed for them, and the average amount of false positives returned when n-grams are used to index music.published_or_final_versio

Kao, B

Yip Chi Lap

HKU Scholars Hub

Title A study on n-gram indexing of musical featuresAuthor(s) Yip Chi Lap; Kao, BCitation Ieee International Conference On Multi-Media And Expo, 2000II/TUESDAY, p. 869-872Issued Date 2000URL http://hdl.handle.net/10722/45624Rights Creative Commons: Attribution 3.0 Hong Kong LicenseA STUDY ON N-GRAM INDEXING OF MUSICAL FEATURES YIP Chi Lap Ben KAO clyip@csis.hku.hk kao@csis.hku.hk Department of Computer Science and Information Systems, The University of Hong Kong ABSTRACT Since only simple symbol-based manipulations are needed, n-gram indexing is used for natural languages where syntactic or semantic analyses are often difficult. Music, whose automatic analysis for patterns such as motifs and phrases are difficult, inaccurate or computationally expen- sive, is thus similar to natural languages. The use of n-gram in music retrieval systems is thus a natural choice. In this paper, we study a number of issues regarding n- gram indexing of musical features using simulated queries. They are: whether combinatorial explosion is a problem in n-gram indexing of musical features, the relative discrimi- nation power of six different musical features, the value of n needed for them, and the average amount of false positives returned when n-grams are used to index music. 1. INTRODUCTION Because of its simplicity, n-gram indexing has been used in text retrieval systems for years. It is done by associating every n-character fragment, called an n-gram, of a docu- ment with the document identifier in an inverted index. It is particularly suitable for indexing text in languages without easy-to-parse word boundaries, such as Chinese, Japanese, or Korean. Only simple character-based manipulation is needed to build n-gram indices; elaborate syntactic or se- mantic analysis of text, which are particularly difficult for these languages, are not required. Music, seen as a temporal sequence of notes and rests, is similar to those natural languages in one aspect: although it is often easy to obtain note sequences from computer rep- resentations of music, further analysis for musical patterns such as motifs, phrases and movements are often difficult, inaccurate, or computationally expensive. This is because music, like natural languages, do not have rigid grammati- cal rules, and is multidimensional in the sense that the same piece can be interpreted in multiple ways [4]. Simple index- ing techniques that do not require extensive musical anal- ysis thus have their appeal for music indexing in retrieval systems, and n-gram indexing is thus a natural choice. In indexing natural languages, bigrams and trigrams, that is, n-grams with n = 2 or 3, are most often used. In- 0-7803-6536-4/00/$10.00 (c) 2000 IEEE deed, the use of bigrams and trigrams fits the characteristics of natural languages. For example, since more than two- third of Chinese words consist of two characters [5], bigram indexing is often used for Chinese. For the same reason, n is often 2 or 3 for Korean [3]. However, although n-gram indexing seems to be suitable for music, few studies can be found in the literatures on what the n should be for musical n-grams. Since there are potentially lCln n-gram combi- nations for a musical feature sequence with IC1 alphabets, combinatorial explosion would be a problem if a large n is needed. Besides the problem of potential combinatorial explo- sion, other issues arise when n-grams are used for indexing. Unlike indexing methods that use data structures for string matching such as the suffix tree 163, positional information are normally not stored in n-gram indices. Since a query is considered to match a document when all its n-grams can be found in it, false positives may occur. For example, if bigram indexing (n = 2) is used, the query abcd is con- sidered to match the text abacdbc because all the query bigrams ab, bc and cd can be found in the text, though abcd cannot be found in abacdbc. False drops, on the other hand, are not possible. To our knowledge, few articles directly addressed the issue of indexing musical features using n-grams. Two of them are [2] and [71. In [2], 4- to 6-grams of musical inter- vals quantized in four ways were studied in an informetric framework. In [7], the average number of pieces musical n- grams can match was studied. In both works, statistics was based on the assumption that all n-grams were selected with equal probability. Since in real applications this assump- tion may not always hold, in this paper, we study the issues of an n-gram-based music retrieval system using simulated queries. In particular, we seek to answer the following ques- tions. First, whether combinatorial explosion is a problem in musical n-gram indexing. Second, the value of n needed for six different musical features when the statistical distri- bution of n-gram is taken into account. Third, the relative discrimination power of different features, and fourth, the average number of false positives returned when n-grams are used to index music. 869 2. THE MUSIC COLLECTION Since the Internet is one of the major sources of digital music, our collection consists of 1003 MIDI files of songs and music popular in Hong Kong, Taiwan, or Japan, all ob- tained from the Internet. All the pieces were assumed to be in equal temperament tuning system. The MIDI tracks containing melodies were manually extracted for the exper- iments. Because some of the MIDI files contain performed rather than notated music, automatic onset quantization was done on all the pieces to reduce the effect of performance timing variabilities. Essentially, all note onset and offset times were quantized to fit into a set of time values de- termined by parameters such as the time signature of the piece. The sequence of notes and rests obtained were then analyzed for musical features. 3. MUSICAL FEATURES 0 5 10 15 20 25 30 35 40 Length of query in feature points Figure 1 : Average fraction of database containing matching pieces using string matching method 5 ,  1 5 09 4  WbmDurauonRauo * 5 8 l  U 0 0 5 10 15 20 25 30 35 40 Length of query In feature points Three traditionally used musical features, namely Inter- val Sequence, Profile and Note Duration Ratio Sequence, and three “Coarse Interval” features introduced in [7], namely C13, CI5 and C17, were used in the experiments. An Interval Sequence is a sequence of numbers each denoting the musi- cal interval size, in number of semitones, between two tem- porally consecutive notes. Coarse Interval sequences CIz are uniformly quantized Interval Sequences where z inter- vals are grouped together with the unison interval at the cen- ter of a group. Mathematically, an interval of s is mapped to sgn(s) for the feature CI(2n + 1). The feature Profile shows the shape of the melody by the ups and downs of its note pitches; it is nonuniformly quantized Interval Se- quence using the three-valued sign function. Those features derived from Interval Sequence (i.e., CIz and Profile) often have a smaller alphabet size than Interval Sequence, which can help reducing index sizes. A Note Duration Ratio Se- quence is a sequence of ratios of sounding durations be- tween temporally consecutive notes. Some examples of the features introduced here can be found in [7]. 4. THE EXPERIMENTS In our experiments, every piece of music in our collec- tion is first analyzed for the described features. Then, for every feature, multiple n-gram indices are built with dif- ferent values of n. To search a particular n-gram index, a given query of length q 2 n feature points is first broken down into (q  - n + 1) n-grams. By looking up the n-gram index, we obtain, for each piece in the database, the number of query n-grams that matches it. A query is considered to completely match a piece when all the q - n + 1 query n- grams can be found in it. In our experiments, the values of n from 1 to 8 are used. Results where less than q - n + 1 0-7803-6536-4/00/$10.00 (c) 2000 IEEE 870 Figure 2: Coefficient of variation on number of matching pieces using string matching method query n-grams are matched cannot be reported here due to space limitations. Queries in our experiments are generated by sampling methods. First, a query length of q feature points is fixed. Then, segments of musical features of length q is selected from the musical features of the pieces so that the probabil- ity of any segment q feature points in length being selected is the same. This way, for any given n-gram, its probability of being in the length-q segment will be proportional to its occurrence frequency in the collection. In our experiments, 1000 queries are generated for each value of q, and the value of q varies from 4 to 32 in our experiments. To compare the number of matching pieces when n- gram indexing is used under different values of n, we match every set of length-q query samples with all the n-gram in- dices for all n < q. The queries are also matched against the database using an exact string-matching method. This gives us match results in which no false drops nor false positives can occur. 5. RESULT AND ANALYSIS To study the relative discrimination power of the fea- tures, Figure 1 shows a graph on the average fraction of the database that matches the samples against the length of query in feature points using a string matching method. Also, curves of coefficient of variation, that is, standard de- viation normalized by the average value, for different fea- tures are also plot in Figure 2. As expected, as the query length q increases, the average fraction of songs in the data- base that match the queries decreases, since the queries be- come more specific. However, comparing the six curves, it is found that a rather large n is needed for C15 and C17 curves to be sufficiently low; on average, more than 10% of the database gives a match even when the queries consist of 16 CI7 feature points. In contrast, a smaller n is needed for the Profile curve. Since Profile is Interval Sequence nonuni- formly quantized to three values, with the fact that musical intervals tend to be small-sized in the musical scale [ 13, this observation indicates that uniformly quantized Interval Se- quence with large quantization step size are not discrimi- nating features. Indeed, the unison interval has been found to be an important discriminant of music [7]. For features based on Interval Sequence to be more discriminating, finer quantization step size could be applied (e.g., use C13 rather than CI7), or nonuniformly quantization can be done. Other possibilities that can be investigated in future research in- clude use of Interval Sequence nonuniformly quantized to more than three levels. The coefficient of variation shows the dispersion of data relative to the average. Note that the coefficient of varia- tion for a feature is not very meaningful when n is so small that a large proportion of the database is matched, since the standard deviation cannot become too large from the aver- age value, This is the case for most features when q = 4 and the coarse intervals when q = 6. From the graph, we found that the coefficients of variation for the features In- terval Sequence and C13 reach a value of below one at a moderate value of n around 15. That is, the relative disper- sion of the fraction of matching pieces is small for such ann value. Since practical query lengths for Interval Sequences and C13 are also at that range, these features are good in the sense that the number of matching pieces using two queries of the same length would not differ by too much; they “par- tition” the database more or less evenly. The feature Profile also shows a similar property although a higher value of n is needed. Coarse Interval features and Note Duration Ra- tio Sequences, on the other hand, give relatively high value of coefficient of variation in our considered range of query lengths. This means some of their feature sequences match very few pieces in the database, while some match a lot. In other words, some of these feature sequences occur in many pieces while some very few. These features can still be good features to use provided that feature sequences that match a lot of pieces can be identified and removed. This is sim- ilar to stopword removal in natural language text retrieval systems. False positives can affect retrieval efficiency because piece-by-piece filtering is needed to remove them. To study the amount of false positives n-gram indexing would give, for every feature we found the average proportion of songs in the database that are false positives. The values found for 0-7803-6536-4/00/$10.00 (c) 2000 IEEE every n considered are plot against the length of queries and are shown in Figure 3. Each graph in the figure corresponds to one feature, and each curve in the graphs corresponds to one value of n. The topmost curves are for n = 1, second topmost for n = 2, and so on. For example, the false positive proportion for C15 can be more than 10% unless n is larger than 7, and that for C17 is always larger than 7.5% for query lengths more than 12 feature points even when n = 8. This means that relatively large values of n are required for them, which can offset the gain obtained by their smaller alphabet size compared to In- terval Sequence. Further investigation is needed on whether there are improvements when, as previously described, fea- ture stopwords are identified and removed. From these graphs, we also found that the value of n is different for different features. For example, with a moder- ate query length of 12 feature points, to achieve a false pos- itive proportion of less than 2%, the values of n required are at least 3 , 5 , 7  and 5 for Interval Sequence, CI3, Profile and Note Duration Ratio Sequence respectively. An n of more than 8 is required for features C15 and C17. Since these val- ues of n for the first four features are not very large and in general features requiring a larger n have smaller alphabet sizes [7], combinatorial explosion would not be a serious problem in n-gram indexing of most musical features stud- ied here. In short, n-gram indexing is a feasible scheme to use in music retrieval systems. Note that the value of n observed above should be in- terpreted differently from those recommended in [7]. It is because in that paper, all n-grams were considered to be se- lected equally probably, while here, the sampling procedure automatically takes n-gram distribution into account. A number of observations can be made from these graphs. 6. SUMMARY AND FUTURE WORK The results of our experiments show that n-gram index- ing is a viable option for building music retrieval systems. Indexing using n-gram has the advantage of being efficient during both search and index construction processes. So- phisticated music analysis is not required, and combinato- rial explosion problem is found to be relatively small. Our simulated queries on n-gram indices of six musical features show that the choice of n depends on the musical feature, but n is often smaller than 8. We also found that if no musical feature “stopwords” are identified and removed, nonuniformly quantized Inter- val Sequences such as Profile are more discriminating than uniformly quantized ones such as C15 and C17. Hence, to design new features based on quantization of Interval Se- quence, nonuniformly quantization to more than three levels can be a good choice. Our study on the relative dispersion on the fraction of 87 1 .% 6 E 0 5 IO 15 20 25 30 35 40 Length ol query in leacure p i n t s  (a) Interval Sequence 08 e 0 6  B P 0 4  2 & 0 2  E E 0 P 0 5 10 15 20 25 30 35 40 Length 01 query in lealure points (b) Profile 0 5 10 I5 20 25 30 35 40 Length of query in feature points 0 5 I0 I5 20 25 30 35 40 Length of query in feature points (d) CI5 0 5 10 15 20 25 30 35 40 Length of query in feature points 4 "  (e) C17 P 0 4  2 & 0 2  E > n  " 0 5 10 I5 20 25 30 35 40 Length of query in feature p i n t s  (0 Note Duration Ratio Sequence Figure 3: Fraction of false positives in the datab different query lengths 0-7803-6536-4/00/$10.00 (c) 2000 IEEE the database that matches the samples shows that Interval Sequence and C13 are good in the sense that they "parti- tion" the database more or less evenly at a reasonable query lengths. The feature Profile also shows a similar property although a longer query lengths are needed. Larger rela- tive dispersions are found for Coarse Interval features and Note Duration Ratio Sequences; some of them match very few pieces in the database, while some match a lot. Musical "stopword" identification and removal are needed to make them more discriminating. Besides reporting results on approximate n-gram match- ing of music features as described in Section 4, The authors are currently working on a number of issues in music re- trieval. Some of them include the design of features that are suitable for handling polyphonic music, the study on the effect of combining different features in content-based music search, and the effect of different musical genres on music retrieval systems. Moreover, the design of indexing schemes besides n-gram are also being carried out. 7. ACKNOWLEDGMENTS Special acknowledgments are given to CHENG Man Yee, HO Wai Shing, TANG Fung Michael, and YUEN Kin Wai for their help on the extraction of melodic lines from the music collection which made the experiments possible. 8. REFERENCES [I] W. Jay Dowling. Scale and contour: Two components of a theory of memory for melodies. Psychological Re- view, 85(4):341-354, July 1978. [2] J. Stephen Downie. Informetrics and music informa- tion retrieval: An informetric examination of a folksong database. In Proc. 26th Annual Con$ of the Canadian Assoc. for Info. Sc. (CAIS 1998), 1998. [3] Joo Ho Lee and Jeong So0 Ahn. Using n-grams for Korean text retrieval. In Proc. SZGlR '96, pages 216- 224,1996. Beyond MIDI: The [4] Eleanor Selfridge-Field, editor. Handbook of Musical Codes. The MIT Press, 1997. [5] Ching Y. Suen. Computational studies of the most fre- quent Chinese words. World Scientific, 1986. [6] Esko Ukkonen. Constructing suffix trees on-line in lin- ear time. In J. van Leeuwen, editor, Algorithms, Soft- ware, Architecture: Information Processing 92, vol- ume 1,  pages 484492.  Elsevier Science B.V., 1992. [7] Chi Lap Yip and Ben Kao. A study on musical features for melody databases. In Proc. DEXA 1999, number 1677 in LNCS, pages 724-733. Springer-Verlag, 1999. lase under 872 

A study on n-gram indexing of musical features

http://hub.hku.hk/bitstream/10722/45624/1/57316.pdf

A study on n-gram indexing of musical features

Abstract

Similar works

Full text

Available Versions

HKU Scholars Hub