We discuss the problem of music content analysis within the formal framework of experimental design

Grossmann, H

International Conference on Machine Learning

Maruri-Aguilar, H

Parker, B

STURM, BLT

English

Queen Mary Research Online

The scientific evaluation of music content analysis systems: Validempirical foundations for future real-world impactSTURM, BLT; Maruri-Aguilar, H; Parker, B; Grossmann, H; International Conference onMachine Learning     CC-BY  For additional information about this publication click this link.http://qmro.qmul.ac.uk/xmlui/handle/123456789/11294   Information about this research object was correct at the time of download; we occasionallymake corrections to records, please therefore check the published record when citing. Formore information contact scholarlycommunications@qmul.ac.ukThe scientific evaluation of music content analysis systems:Valid empirical foundations for future real-world impactBob L. Sturm B.STURM@QMUL.AC.UKSchool of Electronic Engineering and Computer Science, Queen Mary University of London, U.K.Hugo Maruri-Aguilar, Ben Parker {H.MARURI-AGUILAR, B.PARKER}@QMUL.AC.UKSchool of Mathematical Sciences, Queen Mary University of London, U.K.Heiko Grossmann HEIKO.GROSSMANN@OVGU.DEInstitute for Mathematical Stochastics, Otto-von-Guericke University Magdeburg, GermanyAbstractWe discuss the problem of music content analysiswithin the formal framework of experimental design.1. PreliminariesBy the formalism developed in Sturm et al. (2014), define amusic universe Ω, a music recording universeRΩ, descrip-tor vocabularies F (features) and V (tokens), and Booleansemantic rules A′ : f → {T, F} and A : s → {T, F},where f and s are finite sequences of elements in F and V ,respectively. Define the semantic universeSV,A := {s ∈ Vn|n ∈ N ∧A(s) = T}. (1)The semantic feature universe SF,A′ is built similarly, usingF and A′. A recorded music description system S is a mapS : RΩ → SV,A (2)which is a composition of two maps: E : RΩ → SF,A′and C : SF,A′ → SV,A. The map E is commonly knownas a “feature extractor,” and C as a “classifier.” A recordedmusic dataset is an indexed sampleD = ((ri, si) : i ∈ I) ⊂ RΩ × SV,A (3)where I indexes the samples. The sequence (si)i∈I iscalled the ground truth of D. Finally, music content analy-sis research encompasses all aspects above in order to con-nect “users” (people, organisations, etc.) with music andinformation about music.Sturm, Maruri-Aguilar, Parker, Grossmann. Licensed under a Cre-ative Commons Attribution 4.0 International License (CC BY 4.0).Attribution: Sturm, Maruri-Aguilar, Parker, Grossmann. “The sci-entific evaluation of music content analysis systems: Valid empir-ical foundations for future real-world impact,” Machine Learningfor Music Discovery Workshop at the 32nd International Confer-ence on Machine Learning, Lille, France, 2015.2. Experimental design in generalExperimental design (Bailey, 2008) is necessary to plan,implement, analyse, and report valid tests of hypotheseswhile meeting constraints that are physical, economical,ethical, and so on. Fundamental concerns of experimen-tal design are the specification of the treatments, the exper-imental and observational units (plots), the identificationof structures in treatments and plots, the mapping of unitsto treatments, the relevance of the measurement, the mod-elling and analysis of measurements, and securing that anexperiment can validly address given hypotheses.An experimental design maps plots {ω1, . . . , ωN} to treat-ments T = {1, . . . , t}, as represented by a design matrixX := [u1, · · · , ut]N×t, of N × 1 indicator vectors ui,where the jth row of ui is 1 if plot ωj receives treatmenti, and 0 if not. Of interest are two subspaces: the treat-ment subspace VT := C(X), which is the column space ofX, and its orthogonal complement V ⊥T , which contains allN × 1 vectors orthogonal to X. Structure in the treatmentsand/or plots may prompt other decompositions of the spaceV = RN in order to test specific hypotheses.An experiment results in measurements on treated units,producingN responses y ∈ V . The effect of the treatmentson y is often described by a linear modelY = τ + Z (4)where τ =∑τiui ∈ VT and τ1, . . . , τt are the treatmentparameters (unknown constants). Moreover, Z is a randomvector which can be modelled further to reflect structurein the plots and random error. Testing hypotheses about{τi} relies upon assumptions about Z. The simple textbookmodel for unstructured plots and a completely randomiseddesign assumes that Z ∼ N (0, σ2IN ). Structure in theplots necessitates the use of a different model for Z. Hy-pothesis testing by the method of analysis of variance de-The scientific evaluation of music content analysis systemscomposes y over orthogonal subspaces of V , such as VTand V ⊥T , in accordance with the hypotheses and the plotstructure, and then assesses their contributions.3. Experimental design for MCA systemsOf principal interest is a comparison of music content anal-ysis (MCA) systems {S1, S2, . . .}, as in (2), which are thetreatments T . A common experiment toward this proceeds:1. Build or choose a D, indexed by I2. Partition I into non-overlapping sets Itrain, Itest3. Build {S1, S2, . . .} from tuples of D indexed by Itrain4. Treat recordings of D indexed by Itest and compare resultsto ground truth, i.e., record successes and failures.This is often done using K-fold cross validation (KfCV),or repeated random partitioning, in which case steps 2-4are repeated. To summarise the results of this experiment,a figure of merit (FoM) is computed, e.g., accuracy, whichis then used to make comparisons between treatments.This process is typified by the experiment of Tzanetakis& Cook (2002), which is one of the most cited articles inMCA. They use a D with 1000 recordings and a groundtruth with 100 occurrences of each of the 10 elements(genre) in SV,A. They perform the experiment above withseveral systems using the same instance of E but differentclassifiers built from different methods. For each kind ofsystem Sk, they perform 100 repetitions of 10fCV in D.Each repetition uses a fresh random partition of the 100recordings per genre in D into 10 mutually exclusive sub-sets A(j)1 , . . . , A(j)10 of size 10, where j = 1, . . . , 10 iden-tifies the genre. For i = 1, . . . , 10 the ith fold is then de-fined as the set-theoretic union of A(1)i , . . . , A(10)i . Each10fCV uses for each i ∈ {1, . . . , 10} the ith fold as Itestand the union of the remaining nine folds as Itrain. Foreach (ri, si) ∈ D with i ∈ Itest the response is 1 if the sys-tem output matches the ground truth, that is if Sk(ri) = si,and 0 otherwise. This gives 1000 responses per repetitionand a corresponding proportion of matches. Tzanetakis andCook compute the mean rate of success (accuracy) and thestandard deviation of those proportions over the 100 repe-titions. The table below shows some of their results.Classifier FoM (accuracy)Kind (mean± std. dev.)GS 0.59± 0.04GMM(3) 0.61± 0.04KNN(3) 0.60± 0.04Three conclusions they make from their experiment are:A. Since D is a “representative” sample fromRΩ × SV,A, theFoM (based on D) of GS, GMM(3) and KNN(3) in Table 3are “indicative” of the FoM that would be obtained if thesekinds of systems were applied to the entire universeRΩ.B. Since the FoM of GS, GMM(3) and KNN(3) using the pro-posed E are better than randomly mapping RΩ to SV,A (inwhich case the FoM is expected to be 0.10), the set SF,A′ isinformative for SV,A in the whole universeRΩ.C. Since the FoM in Table 3 are better than randomly mappingRΩ to SV,A, the systems are recognising SV,A inRΩ.The validity of each of these conclusions relies on twostrong assumptions. Conclusion A assumes that D is a ran-dom sample from RΩ × SV,A, a set that is never explic-itly defined. Since there is clear evidence that D is not arandom sample (Sturm, 2014), the experiment may or maynot result in FoM that reflect the performance of a systemapplied to RΩ. The other two conclusions relate to the be-haviour expected of a system operating randomly, and as-sume that an MCA system’s ability to reproduce the groundtruth of D is either due to chance or caused by “musicalcontent” relating RΩ to SV,A. This assumption is not true:another way to reproduce the ground truth of D is by ex-ploiting “non-musical content” (characteristics of subsetsof RΩ unrelated to SV,A), e.g., a confounding introducedby the construction of D. The existence of such confound-ing in the dataset used in Tzanetakis & Cook (2002), aswell as others, has been clearly demonstrated (Pampalket al., 2005; Sturm, 2014). Because of its lack of controlover such a possibility, the experiment above has no valid-ity for conclusions B and C.A possible measurement model for the experiment in linewith Bailey (2008) would recognise the individual crossvalidations as the observational units and “repetitions” asan additional plot factor. A system of each kind k is ap-plied as a treatment to 100 whole repetitions (each consist-ing of 10 units). The response for each unit u is the propor-tion of correct classifications of the 100 pairs (ri, si) withi ∈ Itest(u, v), where Itest(u, v) denotes the test set for thevalidation u in repetition v. Denote by pkuv the probabilitythat a system of type k gives a correct classification for uand v. A model for this situation ispkuv = β0 + βk + γv + uv, (5)where, using an appropriate parameterisation, β0 is theprobability of success of a random system and βk is the ad-ditional contribution of the system to type k. Further, γv isa contribution due to the random partitioning in repetitionv, and uv models all other “errors” independent of the sys-tem type due to cross validation instances u within v. Forthis interpretation of β0 and βk to be valid, the experimentwould also need to include a random system.To provide evidence for the conclusions B and C of Tzane-takis & Cook (2002) one would need to test the hypothesisβk = 0 in (5). This test would then be performed in thestratum for repetitions. For three kinds of systems, as in thetable above, and an additional random system (each with100 repetitions and 10 validations) and assuming normal-ity, the corresponding F test has 396 denominator degreesof freedom. Assuming all individual cross validations areindependent can result in an incorrect test overstating thesignificance of the results. The applicability of this is sus-pect, however, since γv is known to depend on the systemtype (Pampalk et al., 2005; Sturm, 2014).The scientific evaluation of music content analysis systemsBailey, R. A. Design of comparative experiments. CambridgeUniversity Press, 2008.Pampalk, E., Flexer, A., and Widmer, G. Improvements of audio-based music similarity and genre classification. In Proc. IS-MIR, pp. 628–233, Sep. 2005.Sturm, B. L. The state of the art ten years after a state of the art:Future research in music information retrieval. J. New MusicResearch, 43(2):147–172, 2014.Sturm, B. L., Bardeli, R., Langlois, T., and Emiya, V. Formalizingthe problem of music description. In ISMIR, pp. 89–94, 2014.Tzanetakis, G. and Cook, P. Musical genre classification of audiosignals. IEEE Trans. Speech Audio Process., 10(5):293–302,July 2002.

Design of comparative experiments.

Formalizing the problem of music description.

Improvements of audiobased music similarity and genre classification.

Musical genre classification of audio signals.

The state of the art ten years after a state of the art: Future research in music information retrieval.

The scientific evaluation of music content analysis systems: Valid empirical foundations for future real-world impact

The scientific evaluation of music content analysis systems: Valid empirical foundations for future real-world impact

Abstract

Similar works

Full text

Available Versions

Queen Mary Research Online