An overview of several of the music-related projects at the Laboratory for Recognition and Organization of Speech and Audio, Department of Electrical Engineering, Columbia University

Ellis, Daniel P. W.

English

Columbia University Academic Commons

Learning from Music - Ellis 2004-12-18 p.     /2411.  Learning Music2.  Music Similarity3.  Melody, Drums, Event extraction4.  ConclusionsWhat can we Learn from Large Music Databases?Dan EllisLaboratory for Recognition and Organization of Speech and Audio Dept. Electrical Engineering, Columbia University, NY USAdpwe@ee.columbia.eduLearning from Music - Ellis 2004-12-18 p.     /242Learning from Music• A lot of music data availablee.g. 60G of MP3 ≈ 1000 hr of audio/15k tracks• What can we do with it?implicit definition of ‘music’• Quality vs. quantitySpeech recognition lesson:10x data, 1/10th annotation, twice as useful• Motivating Applicationsmusic similarity / classificationcomputer (assisted) music generationinsight into musicLearning from Music - Ellis 2004-12-18 p.     /243Ground Truth Data• A lot of unlabeled music data availablemanual annotation is much rarer• Unsupervised structure discovery possible.. but labels help to indicate what you want• Weak annotation sourcesartist-level descriptionssymbol sequences without timing (MIDI)errorful transcripts• Evaluation requires ground truthlimiting factor in Music IR evaluations?File: /Users/dpwe/projects/aclass/aimee.wavf 9   Printed: Tue Mar 11 13:04:285001000150020002500300035004000450050005500600065007000Hz0:02 0:04 0:06 0:08 0:10 0:12 0:14 0:16 0:18 0:20 0:22 0:24 0:26 0:28tmus mus musvoxvoxLearning from Music - Ellis 2004-12-18 p.     /244Talk RoadmapMusicaudioDrumsextractionEigen- rhythmsEventextractionMelodyextractionFragmentclusteringAnchormodelsSimilarity/recommend'nSynthesis/generation?Semanticbases12 365 4Learning from Music - Ellis 2004-12-18 p.     /2451. Music Similarity Browsing• Musical information overloadrecord companies filter/categorize musican automatic system would be less odious• Connecting audio and preferencemap to a ‘semantic space’?n-dimensionalvector in "AnchorSpace"AnchorAnchorAnchorAudioInput(Class j)p(a1|x)p(a2|x)p(an|x)GMMModelingConversion to Anchorspacen-dimensionalvector in "AnchorSpace"AnchorAnchorAnchorAudioInput(Class i)p(a1|x)p(a2|x)p(an|x)GMMModelingConversion to AnchorspaceSimilarityComputationKL-d, EMD, etc.with Adam BerenzweigLearning from Music - Ellis 2004-12-18 p.     /246Anchor Space• Frame-by-frame high-level categorizationscompare toraw features?properties in distributions? dynamics?third cepstral coeffifth cepstral coefmadonnabowie Cepstral Features 1  0.5 0 0.5 0.8 0.6 0.4 0.200.20.40.6CountryElectronicamadonnabowie  10 5Anchor Space Features 15  10  5 150Learning from Music - Ellis 2004-12-18 p.     /247‘Playola’ Similarity BrowserLearning from Music - Ellis 2004-12-18 p.     /248Semantic Bases• What should the ‘anchor’ dimensions be?hand-chosen genres?  Xsomehow choose automatically• “Community metadata”:Use Web to get words/phrases.... that are informative about artists.. and that can be predicted from audio• Refine classifiers to below artist level e.g. by EM?Brian Whitmana indicates the number of frames in which a term classi-fier positively agrees with the truth value (both classifierand truth say a frame is ‘funky,’ for example). b indicatesthe number of frames in which the term classifier indi-cates a negative term association but the truth value indi-cates a positive association (the classifier says a frame isnot ‘funky,’ but truth says it is). The value c is the amountof frames the term classifier predicts a positive associationbut the truth is negative, and the value of d is the amount offrames the term classifier and truth agree to be a negativeassociation. We wish to maximize a and d as correct clas-sifications; by contrast, random guessing by the classifierwould give the same ratio of classifier labels regardless ofground truth i.e. a/b ≈ c/d. WithN = a+ b+ c+ d, theK-L distance between the observed distribution and suchrandom guessing is:KL =aNlog(N a(a + b) (a + c))+bNlog(N b(a + b) (b + d))+cNlog(N c(a + c) (c + d))+dNlog(N d(b + d) (c + d))(3)This measures the distance of the classifier away from adegenerate distribution; we note that it is also the mu-tual information (in bits, if the logs are taken in base 2)between the classifier outputs and the ground truth labelsthey attempt to predict.Table 2 gives a selected list of well-performing termmodels. Given the difficulty of the task we are encour-aged by the results. Not only do the results give us termmodels for audio, they also give us insight into whichterms and description work better for music understand-ing. These terms give us high semantic leverage withoutexperimenter bias: the terms and performance were cho-sen automatically instead of from a list of genres.7.3. Automatic review generationThe multiplication of the term model c against the testinggram matrix returns a single value indicating that term’srelevance to each time frame. This can be used in re-view generation as a confidence metric, perhaps setting athreshold to only allow high confidence terms. The vectorof term and confidence values for a piece of audio can alsobe fed into other similarity and learning tasks, or even anatural language generation system: one unexplored pos-sibility for review generation is to borrow fully-formedsentences from actual reviews that use some amount ofterms predicted by the term models and form coherentparagraphs of reviews from this generic source data. Workin language generation and summarization is outside thescope of this article but the results for the term predictionadj Term K-L bits np Term K-L bitsaggressive 0.0034 reverb 0.0064softer 0.0030 the noise 0.0051synthetic 0.0029 new wave 0.0039punk 0.0024 elvis costello 0.0036sleepy 0.0022 the mud 0.0032funky 0.0020 his guitar 0.0029noisy 0.0020 guitar bass and drums 0.0027angular 0.0016 instrumentals 0.0021acoustic 0.0015 melancholy 0.0020romantic 0.0014 three chords 0.0019Table 2. Selected top-performing models of adjective andnoun phrase terms used to predict new reviews of musicwith their corresponding bits of information from the K-Ldistance measure.task and the below review trimming task are promising forthese future directions.One major caveat of our review learning model is itstime insensitivity. Although the feature space embeds timeat different levels, there is no model of intra-song changesof term description (a loud song getting soft, for example)and each frame in an album is labeled the same duringtraining. We are currently working on better models oftime representation in the learning task. Unfortunately,the ground truth in the task is only at the album level andwe are also considering techniques to learn finer-grainedmodels from a large set of broad ones.7.4. Review RegularizationMany problems of non-musical text and opinion or per-sonal terms get in the way of full review understanding. Asimilarity measure trained on the frequencies of terms in auser-submitted review would likely be tripped up by obvi-ously biased statements like “This record is awful” or “Mymother loves this album.” We look to the success of ourgrounded term models for insights into the musicality ofdescription and develop a ‘review trimming’ system thatsummarizes reviews and retains only the most descriptivecontent. The trimmed reviews can then be fed into fur-ther textual understanding systems or read directly by thelistener.To trim a review we create a grounding sum term oper-ated on a sentence s of word length n,g(s) =∑ni=0 P (ai)n(4)where a perfectly grounded sentence (in which the predic-tive qualities of each term on new music has 100% preci-sion) is 100%. This upper bound is virtually impossiblein a grammatically correct sentence, and we usually seeg(s) of {0.1% .. 10%}. The user sets a threshold andthe system simply removes sentences under the threshold.See Table 3 for example sentences and their g(s). We seethat the rate of sentence retrieval (how much of the reviewis kept) varies widely between the two review sources;AMG’s reviews have naturally more musical content. SeeFigure 4 for recall rates at different thresholds of g(s).Learning from Music - Ellis 2004-12-18 p.     /2492. Transcription as Classification• Signal models typically used for transcriptionharmonic spectrum, superposition• But ... trade domain knowledge for datatranscription as pure classification problem:single N-way discrimination for “melody”per-note classifiers for polyphonic transcriptionTrainedclassifierAudiop("C0"|Audio)p("C#0"|Audio)p("D0"|Audio)p("D#0"|Audio)p("E0"|Audio)p("F0"|Audio)with Graham PolinerLearning from Music - Ellis 2004-12-18 p.     /2410Classifier Transcription Results• Trained on MIDI syntheses (32 songs)SMO SVM (Weka)• Tested on ISMIR MIREX 2003 setforeground/background separationsystem “jazz3” overallfg+bg 71.5% 44.3%just fg 56.1% 45.4%Frame-level pitch concordanceLearning from Music - Ellis 2004-12-18 p.     /2411Forced-Alignment of MIDI• MIDI is a handy description of musicnotes, instruments, tracks.. to drive synthesis• Align MIDI ‘replicas’ to get GTruth for audioestimate time-warp relation"Don't you want me" (Human League), verse117 18 19 20 21 22 23 24 25 2602419 20 21 22 23 24 25 26 27 28406080MIDI notesMIDI ReplicaOriginalfreq / kHzMIDI #024freq / kHztime / secwith Rob TuretskyLearning from Music - Ellis 2004-12-18 p.     /24123. Melody Clustering• Goal: Find ‘fragments’ that recur in melodies.. across large music database.. trade data for model sophistication• Data sourcespitch tracker, or MIDI training data• Melody fragment representationDCT(1:20) - removes average, smoothes detailTrainingdataMelodyextraction5 secondfragmentsTopclustersVQ clusteringwith Graham PolinerLearning from Music - Ellis 2004-12-18 p.     /2413Melody clustering results• Clusters match underlying contour:• Finds some similarities:e.g. Pink + NsyncLearning from Music - Ellis 2004-12-18 p.     /24144. Eigenrhythms: Drum Pattern Space• Pop songs built on repeating “drum loop”variations on a few bass, snare, hi-hat patterns• Eigen-analysis (or ...) to capture variations?by analyzing lots of (MIDI) data, or from audio• Applicationsmusic categorization“beat box” synthesisinsightwith John ArroyoLearning from Music - Ellis 2004-12-18 p.     /2415Aligning the Data• Need to align patterns prior to modeling...tempo (stretch): by inferring BPM & normalizingdownbeat (shift): correlate against ‘mean’ templateLearning from Music - Ellis 2004-12-18 p.     /2416Eigenrhythms (PCA)• Need 20+ Eigenvectors for good coverage of 100 training patterns (1200 dims)• Eigenrhythms both add and subtractLearning from Music - Ellis 2004-12-18 p.     /2417Posirhythms (NMF)• Nonnegative: only adds beat-weight• Capturing some structurePosirhythm 1BDSNHHBDSNHHBDSNHHBDSNHHBDSNHHBDSNHHPosirhythm 2 Posirhythm 3 Posirhythm 4 Posirhythm 5 500 0 samples (@ 200 Hz)beats (@ 120 BPM)100 150 200 250 300 350 4001 2 3 4 1 2 3 4Posirhythm 6 50 100 150 200 250 300 350-0.100.1Learning from Music - Ellis 2004-12-18 p.     /2418Eigenrhythms for Classification• Projections in Eigenspace / LDA space• 10-way Genre classification (nearest nbr):PCA3: 20% correctLDA4: 36% correct-20 -10 0 10-10-50510PCA(1,2) projection (16% corr)-8 -6 -4 -2 0 2-4-20246LDA(1,2) projection (33% corr)bluescountrydiscohiphophousenewwaverockpoppunkrnbLearning from Music - Ellis 2004-12-18 p.     /2419Eigenrhythm BeatBox• Resynthesize rhythms from eigen-spaceLearning from Music - Ellis 2004-12-18 p.     /24205. Event Extraction• Music often contains many repeated eventsnotes, drum soundsbut: usually overlapped...• Vector Quantization finds common patterns:representation...aligning/matching...how much coverage required?TrainingdataFindalignmentsCombine &re-estimateEventdictionaryLearning from Music - Ellis 2004-12-18 p.     /2421Drum Track Extraction• Initialize dictionary with Bass Drum, Snare• Match only on a few spectral peaksnarrowband energy most likely to avoid overlap• Median filter to re-estimate template.. after normalizing amplitudescan pick up partials from common notes with Ron Weiss, after Yoshii et al. ’04Learning from Music - Ellis 2004-12-18 p.     /2422Generalized Event Detection• Based on ‘Shazam’ audio fingerprints (Wang’03)relative timing of F1-F2-ΔT triples discriminates piecesnarrowband features to avoid collision (again)• Fingerprint events, not recordings:choose top triples, look for repeatsrank reduction of triples x time matrix2 Basic principle of operation The recognition algorithm is described. 2.1 Combinatorial Hashing Each audio file is “fingerprinted,” a process in which reproducible hash tokens are extracted.  A time-frequency analysis is performed, marking the coordinates of local maxima of a spectrogram (Figures 1A and 1B), thus reducing an audio file down to a relatively sparse set of time-frequency pairs.  This reduces the search problem to one similar to astronavigation, in which a small patch of time-frequency constellation points must be quickly located within a large universe of points in a strip-chart universe with dimensions of bandlimited frequency versus nearly a billion seconds in the database.  The peaks are chosen using a criterion to ensure that the density of chosen local peaks is within certain desired bounds so that the time-frequency strip for the audio file has reasonably uniform coverage.  The peaks in each time-frequency locality are also chosen according amplitude, with the justification that the highest amplitude peaks are most likely to survive the distortions listed above. Hashes are formed from the constellation map, in which pairs of time-frequency points are combinatorially associated.  Anchor points are chosen, each anchor point having a target zone associated with it.  Each anchor point is sequentially paired with points within its target zone, each pair yielding two frequency components plus the time difference !"#$%&'$()*+',-.$/0$*(1%&23,456 457 458 455 459 4:6 4:7 4:8 4:5 4:9 496646667666;6668666<666!"#$%&'$()*+',-.$/0$*(1%&23,456 457 458 455 459 4:6 4:7 4:8 4:5 4:9 496646667666;6668666<666with Michael MandelLearning from Music - Ellis 2004-12-18 p.     /2423Event detection results• Procedurefind hash triplescluster thempatterns in hash co-occurrence = events?0 50 100 150 200 250 300050100150Learning from Music - Ellis 2004-12-18 p.     /2424Conclusions• Lots of data + noisy transcription + weak clustering⇒ musical insights?MusicaudioDrumsextractionEigen- rhythmsEventextractionMelodyextractionFragmentclusteringAnchormodelsSimilarity/recommend'nSynthesis/generation?SemanticbasesLearning from Music - Ellis 2004-12-18 p.     /2425• Note transcription, then note→chord ruleslike labeling chords in MIDI transcripts• Spectrum→chord rulesi.e. find harmonic peaks, use knowledge of likely notes in each chord• Trained classifierdon’t use any “expert knowledge”instead, learn patterns from labeled examples• Train ASR HMMs with chords ≈ wordsApproaches to Chord Transcriptionwith Alex ShehLearning from Music - Ellis 2004-12-18 p.     /2426• All we need are the chord sequences for our training examplesHal Leonard “Paperback Song Series”- manually retyped for 20 songs:“Beatles for Sale”, “Help”, “Hard Day’s Night”- hand-align chords for 2 test examplesChord Sequence Data Sources# The Beatles - A Hard Day's Night #G Cadd9 G F6 G Cadd9 G F6 G C D G C9 G G Cadd9 G F6 G Cadd9 G F6 G C D G C9 G Bm Em Bm G Em C D G Cadd9 G F6 G Cadd9 G F6 G C D G C9 G D G C7 G F6 G C7 G F6 G C D G C9 G Bm Em Bm G Em C D G Cadd9 G F6 G Cadd9 G F6 G C D G C9 G C9 G Cadd9 Fadd9time / secfreq / kHz0 5 10 15 20 25012345The Beatles - Hard Day's NightLearning from Music - Ellis 2004-12-18 p.     /2427• Recognition weak, but forced-alignment OKChord ResultsMFCCs are poor(can overtrain) PCPs better(ROT helps generalization)(random ~3%)Feature Recognition AlignmentMFCC 8.7% 22.0%PCP_ROT 21.7% 76.0%Frame-level AccuracytrueE G D Bm GalignE G DBm GrecogE G Bm Am Em7 Bm Em716.27 24.84time / secintensityA#BC#D#EF#G#pitch classBeatles - Beatles For Sale - Eight Days a Week (4096pt)200406080100120Learning from Music - Ellis 2004-12-18 p.     /2428• Chord model centers (means) indicate chord ‘templates’:What did the models learn?0 5 10 15 20 2500.050.10.150.20.250.30.350.4PCP_ROT family model means (train18)DIMDOM7MAJMINMIN7C D E F G A B C (for C-root chords)

What Can We Learn from Large Music Databases?

https://academiccommons.columbia.edu/doi/10.7916/D83R1216/download

What Can We Learn from Large Music Databases?

Abstract

Similar works

Full text

Available Versions

Columbia University Academic Commons