Describes work by the Laboratory for Recognition and Organization of Speech and Audio, Department of Electrical Engineering, Columbia University, on attempting to define a single matrix of similarities between 400 different pop music artists, including survey website (musicseer.com) which attracted over 20,000 similarity judgments

Berenzweig, Adam

Ellis, Daniel P. W.

Lawrence, Steve

Whitman, Brian

English

It would be interesting and valuable to devise an automatic measure of the similarity between two musicians based only on an analysis of their recordings. To develop such a mea-sure, however, presupposes some ‘ground truth ’ training data describing the actual similarity between certain pairs of artists that constitute the desired output of the measure. Since artist similarity is wholly subjective, such data is not eas-ily obtained. In this paper, we describe several attempts to construct a full matrix of similarity measures between a set of some 400 popular artists by regularizing limited subjec-tive judgment data. We also detail our attempts to evaluate these measures by comparison with direct subjective simi-larity judgments collected via a web-based survey in April 2002. Overall, we find that subjective artist similarities are quite variable between users—casting doubt on the concept of a single ‘ground truth’. Our best measure, however, gives reasonable agreement with the subjective data, and forms a useable stand-in. In addition, our evaluation methodol-ogy may be useful for comparing other measures of artist similarity. 1

Daniel P. W. Ellis

Brian Whitman

Adam Berenzweig

Steve Lawrence

CiteSeerX

The Quest for Ground Truth in Musical Artist Similarity

NEUROSURGERY ENTHUSIASTIC WOMEN SOCIETY

The Quest for Ground Truth in Musical Artist Similarity.

Columbia University Academic Commons

2002-10-16Ellis, Whitman,Berenzweig, Lawrence1The Quest for Ground Truth inMusical Artist SimilarityDan Ellis, Brian Whitman, Adam Berenzweig, Steve LawrenceColumbia, MIT, NECOutline1. Similarity and Artists2. Sources of Information3. Evaluation4. Applications and Conclusions2002-10-16Ellis, Whitman,Berenzweig, Lawrence21. Similarity and music Want similarity for recommendation etc. Multiple possible information sources Do they agree?  Which is best? Exercise: define single ‘best-fit’ pairwiseartist distance matrix: Training data for audio-based metrics?04.11.41.2a-ha4.104.44.6aerosmith1.44.402.5ace of base1.24.62.50abbaa-haaerosmithace of baseabba2002-10-16Ellis, Whitman,Berenzweig, Lawrence3Problems Similarity judgment ... is individual ... is deep ... depends on the context/criteria ... can be asymmetric Many aspects of similarity But .. try it and see what happens2002-10-16Ellis, Whitman,Berenzweig, Lawrence4Distances Similarity (1..0) or distance (0..∞)? How to interpret the numbers:Sim = Pr(artist A called artist B) ? Relate the two: Symmetry? “A is like B” ⇔ “B is like A” Triangle inequality: Distances → Geometric embeddingdist = − log sim[ ]( )kdist A,C( ) ≤ dist A,B( ) + dist B,C( )ACB2002-10-16Ellis, Whitman,Berenzweig, Lawrence52. Data sources Many potential sources of subjective‘ground-truth’ data: Explicit descriptions (“A is similar to B, C..”) Behavioral i.e. used in similar contexts Described in similar terms Issues: Converting to numerical values Extending to cover every pair of artists2002-10-16Ellis, Whitman,Berenzweig, Lawrence6Explicit similarity statements All Music Guide similar artists lists Binary similarity (in/not in list)2002-10-16Ellis, Whitman,Berenzweig, Lawrence7Erdös Distance Distance from A to B is minimum # hops also ‘Resistive’ Erdös0422a-ha4045aerosmith2401ace of base2510abbaa-haaerosmithace of baseabbaabbaa-haace of basepet shop boysroxette2002-10-16Ellis, Whitman,Berenzweig, Lawrence8Collection co-occurrence Assumption:User U possess music by A and B⇒ A and B are similar problematic for individuals, averages out OpenNap data:Use Napster protocol to record thecollections of ~3000 users 400,000 identified tracks P(A|B) × popularity mismatch discount2002-10-16Ellis, Whitman,Berenzweig, Lawrence9Similarity of associated text Find web pages mentioning particularartists (“google abba music”) tf•idf weighted terms for each artist Similarity of artists= weightings of common terms Klepmit data2002-10-16Ellis, Whitman,Berenzweig, Lawrence10Multidimensional Scaling Map distances to spatial arrangementminimizing MSE of Euclidean distances Regularizes distance matrices: Issues: # dimensions, distance warpingbackstreet_boyswhitney_housnew_orderaaron_carterabbaace_of_basea_haall_saintsannie_lennoxaquabelinda_carlislespearsceline_dionchristina_aguileracudueiffel_65erasurefaith hilljamiroquaijanet_jacksonjessica_simpson lara_fabianlauryn_hillmadonnamariah_careymatthew sweetnelly_furtadopaula_abdulpet_shop_boysprinceroxettesadea_mumbasavage_gardsealia_twainsoft_cellspice_girlstoni_braxtonwestlife2002-10-16Ellis, Whitman,Berenzweig, Lawrence113. Evaluation How to compare distance metrics? Need independent test data→ Collect new evaluation data set Sparse sampling of artist judgmentsto validate the complete set2002-10-16Ellis, Whitman,Berenzweig, Lawrence12Information elicitation “What is similarity of A and B?” no consistency, arbitrary scale Target + Choices paradigm: Chosen is more like target    than (known) others2002-10-16Ellis, Whitman,Berenzweig, Lawrence13musicseer.com Web site to collect artist-similarityjudgments from anonymous volunteers Two procedures: poperdos game:Get from artist A to B in fewest # steps Direct survey:Choose most similar artist from a list2002-10-16Ellis, Whitman,Berenzweig, Lawrence14poperdos game Competitive motivation of informants? Choices from AllMusic lists + OpenNap→ roughly equidistant choices2002-10-16Ellis, Whitman,Berenzweig, Lawrence15Survey Sequence of single judgments More uniform sampling of targets Infer which bands are known to informant Dummy entries ‘catch’ random guessing2002-10-16Ellis, Whitman,Berenzweig, Lawrence16“10,000 random band names” Generative model of band names trained on real band name ‘grammars’+ vocabulary of music-related web pages Most popular decoy choices: Neither Palindrome Taker Archbishop Riot Skylight Cuisine Blond and Bipolar2002-10-16Ellis, Whitman,Berenzweig, Lawrence17musicseer data 1,782 users best user: 748 judgments, ten above 200 22,470 judgments ~140k triplets <target chosen unchosen> Spike after “10,000 names” blogging http://musicseer.com/results/05001000150020002500Musicseer Trials/Day2002-04 2002-05 2002-06 2002-07 2002-08 2002-09 2002-092002-10-16Ellis, Whitman,Berenzweig, Lawrence18Scoring the similarity metrics Avg. ranking of user choice by metric Agreement percentages for... judgments tripletsTarget: PoliceChoices: U2(ranked by Pretendersmetric) StingQueenINXSPhil CollinsInformant choiceranked #2 by metric⇒ no agreementfor this judgment... but 4 of 5 triplets do agree2002-10-16Ellis, Whitman,Berenzweig, Lawrence19Results Judgment-level agreement %:i.e. user choice ranks #1 under metric “Ceiling” reflects inconsistency limitTop rank agreement01020304050607080cei cmb erd e3d opn kn2 rnd ANK%SrvKnw 4789x3.58SrvAll 6178x8.93GamKnw 7410x3.96GamAll 7421x8.922002-10-16Ellis, Whitman,Berenzweig, Lawrence204. Applications Original motivation: Comparing ground-truth data sources Targets for training music similarityclassifier What happens when you try? Actual outcome musicseer data Evaluation metrics2002-10-16Ellis, Whitman,Berenzweig, Lawrence21Using evaluation data Significant information in musicseerevaluation data basis for a metric in itself? Use these evaluation procedures foracoustic-based similarity measures ‘Inconsistency’ may point to aspects...2002-10-16Ellis, Whitman,Berenzweig, Lawrence22Playola Acoustic similarity browser2002-10-16Ellis, Whitman,Berenzweig, Lawrence23Conclusions Musical artist similarity - a tricky concept but how tricky? best metric agreed with 50% of judgments Evaluation procedure + data hasbroader applications Most valuable outcome? Web-based user data collection isviable

It would be interesting and valuable to devise an automatic measure of the similarity between two musicians based only on an analysis of their recordings. To develop such a measure, however, presupposes some 'ground truth' training data describing the actual similarity between certain pairs of artists that constitute the desired output of the measure. Since artist similarity is wholly subjective, such data is not easily obtained. In this paper, we describe several attempts to construct a full matrix of similarity measures between a set of some 400 popular artists by regularizing limited subjective judgment data. We also detail our attempts to evaluate these measures by comparison with direct subjective similarity judgments collected via a web-based survey in April 2002. Overall, we find that subjective artist similarities are quite variable between users—casting doubt on the concept of a single 'ground truth'. Our best measure, however, gives reasonable agreement with the subjective data, and forms a useable stand-in. In addition, our evaluation methodology may be useful for comparing other measures of artist similarity

http://www.ee.columbia.edu/~madadam/papers/ellis02truth.pdf

The Quest for Ground Truth in Musical Artist Similarity

Abstract

Similar works

Full text

Available Versions

CiteSeerX

NEUROSURGERY ENTHUSIASTIC WOMEN SOCIETY

Columbia University Academic Commons

Columbia University Academic Commons