Prof. Ellis presents a summary of LabROSA's new work, with a focus on recognizing environmental sounds, particularly for video classification by soundtrack

Ellis, Daniel P. W.

Columbia University Academic Commons

Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /3511. Machine Listening2. Background Classification3. Foreground Event Recognition4. Open IssuesRecognizing and ClassifyingEnvironmental SoundsDan EllisLaboratory for Recognition and Organization of Speech and AudioDept. Electrical Eng., Columbia Univ., NY USAdpwe@ee.columbia.edu         http://labrosa.ee.columbia.edu/Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /351. Machine Listening2• Extracting useful information from sound... like animals doDectectClassifyDescribeEnvironmentalSoundSpeech MusicTaskDomainASR MusicTranscriptionMusicRecommendationVAD Speech/MusicEmotion“Sound   Intelligence”EnvironmentAwarenessAutomaticNarrationRecognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Environmental Sound Recognition• Goal:  Describe soundtracks with a vocabulary of user-relevant acoustic events/sources• Challenges:Defining acoustic event vocabularyOverlapping soundsGround-truth training dataClassifier accuracy3time/sec            Spectrogram : YT_beach_001.mpeg, 0 1 2 3 4 5 6 7 8 9time/sec0 1 2 3 4 5 6 7 8 902k4kfreq / Hz1010water wavecheersp1sp2sp3 cheercheer voicevoicevoice voice voicesp4Bgd1k3kRecognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Environmental Sound Applications• Audio Lifelog Diarization• Consumer Video Classification & Search• Live hearing prosthesis app• Robot environment sensitivity409:0009:3010:0010:3011:0011:3012:0012:3013:0013:3014:0014:3015:0015:3016:0016:3017:0017:3018:00preschoolcafepreschoolcafelectureofficeofficeoutdoorgrouplabcafemeeting2officeofficeoutdoorlabcafeRonManuelArroyo?Lesser2004-09-13L2cafeofficeoutdoorlectureoutdoormeeting2outdoorofficecafeofficeoutdoorofficepostlecofficeDSP03compmtgMikeSambarta?2004-09-14Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Prior Work• Environment Classificationspeech/music/silent/machine5Zhang & Kuo ’01• Nonspeech Sound RecognitionMeeting room Audio Event Classificationsports events - cheers, bat/ball sounds, ...Temko & Nadeu ’06Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Consumer Video Dataset• 25 “concepts” from Kodak user studyboat, crowd, cheer, dance, ...68GMM+Bha (9/15)pLSA500+lognorm (12/15)1G+KL2 (10/15)0.10.20.30.40.50.60.70.80.91Overlapped Conceptsgroup of 3+musiccrowdcheersingingone personnightgroup of twoshowdancingbeachparkbabyplaygroundparadeboatsportsgraduationsunsetbirthdayanimalweddingpicnicmuseumski03 m c c s o n g s d b p b p p b s g s s b a w p mLabeled Concepts• Grab top 200 videos from YouTube searchthen filter for quality, unedited = 1873 videosmanually relabel with conceptsRecognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Obtaining Labeled Data• Amazon Mechanical Turk10s clips9,641 videos in 4 weeksColumbia Consumer Video (CCV) set7!Y-G Jiang et al. 2011Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /352. Background Classification• Baseline for soundtrack classificationdivide sound into short frames (e.g. 30 ms)calculate features (e.g. MFCC) for each framedescribe clip by statistics of frames (mean, covariance)= “bag of features”• Classify by e.g. Mahalanobis distance + SVM8VTS_04_0001 - Spectrogramfreq / kHz  1 2 3 4 5 6 7 8 9012345678-20-100102030time / sectime / seclevel / dBvalueMFCC bin  1 2 3 4 5 6 7 8 92468101214161820-20-15-10-505101520MFCC dimensionMFCC dimensionMFCC covariance5 10 15 202468101214161820-50050Video SoundtrackMFCCfeaturesMFCCCovarianceMatrixRecognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Codebook Histograms• Convert high-dim. distributions to multinomial• Classify by distance on histogramsKL, Chi-squared+ SVM9-20 -10 0 10 20-10-8-6-4-202468MFCC(0)MFCC(1)1234567891011121314151 2 3 4 5 6 7 8 9 10 11 12 13 14 15050100150GMM mixturecountMFCCfeaturesGlobal GaussianMixture ModelPer-CategoryMixture ComponentHistogramRecognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Latent Semantic Analysis (LSA)• Probabilistic LSA (pLSA) models each histogram as a mixture of several ‘topics’.. each clip may have several things going on• Topic sets optimized through EM p(ftr | clip) = ∑topics p(ftr | topic) p(topic | clip)use (normalized?) p(topic | clip) as per-clip features10GMM histogram ftrs GMM histogram ftrs“Topic”“Topic”AV ClipAV Clipp(ftr | clip)p(topic | clip) p(ftr | topic)=*Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Background Classification Results• Wide range in performanceaudio (music, ski) vs. non-audio (group, night)large AP uncertainty on infrequent classes11K Lee & Ellis ’10Concept classgroup of 3+musiccrowdcheersingingone personnightgroup of twoshowdancingbeachparkbabyplaygroundparadeboatsportsgraduationskisunsetbirthdayanimalweddingpicnicmuseumMEAN00.10.20.30.40.50.60.70.80.91Average PrecisionGuessingMFCC + GMM ClassifierSingle−Gaussian + KL28−GMM + Bha1024−GMM Histogram + 500−log(P(z|c)/Pz))Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Sound Texture Features• Characterize sounds by perceptually-sufficient statistics.. verified by matched resynthesis• Subbanddistributions & env x-corrsMahalanobis distance ...12McDermott  Simoncelli ’09Ellis, Zheng, McDermott ’11SoundAutomaticgaincontrolEnvelopecorrelationCross-bandcorrelations(318 samples)Modulationenergy(18 x 6)mean, var,skew, kurt(18 x 4)melfilterbank(18 chans) \x\\x\\x\\x\\x\\x\FFT Octave bins0.5,1,2,4,8,16 HzHistogram61712732404  51015   0.5 2 8 32  0.5 2 8 32   5 10 15  5 10 150level1freq / Hzmel band1159_10 urban cheer clapTexture features1062_60 quiet dubbed speech music0 2 4 6 8 10time / smoments mod frq / Hz mel band moments mod frq / Hz mel band0 2 4 6 8 10K S V MK S V MRecognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Sound Texture Features• Test on MED 2010 development data10 specially-collected manual labels13MED 2010 Txtr Feature (normd) means  M V S K ms cbcruralurbanquietnoisyotherDubbedSpeechMusicCheercLap−0.500.5MED 2010 Txtr Feature (normd) stddevs  M V S K ms cbcruralurbanquietnoisyotherDubbedSpeechMusicCheercLap00.51MED 2010 Txtr Feature (normd) means/stddevs  M V S K ms cbcruralurbanquietnoisyotherDubbedSpeechMusicCheercLap−0.500.5Rural Urban Quiet Noisy Dubbed Speech Music Cheer Clap Avg0.50.60.70.80.91  MMVSKmscbcMVSK+msMVSK+cbcMVSK+ms+cbc• Contrasts in feature setscorrelation of labels...• Perform ~ same as MFCCscombine wellRecognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Auditory Model Features• Based on Lyon/Patterson Auditory Image ModelSimplified version is 10x faster (RTx5 → RT/2)• Captures fine time structure in multiple bands.. The information that is lost in MFCC features14CochleafilterbankSoundshort-timeautocorrelationdelay linefrequency channelsCorrelogramslicefreqlagtimeSubband VQSubband VQSubband VQSubband VQHistogramFeature VectorRecognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Auditory Model Feature Results• Results vary with class, but fusion helps 15%Results for CCV (9317 videos)150 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Basketball Baseball Soccer IceSkating Skiing Swimming Biking Cat Dog Bird Graduation Birthday WeddRecep WeddCerem WeddDance MusPerf NonMusPerf Parade Beach Playground mAP MFCC SBPCA Fusion Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Real-World Dictionary• BBC Sound Effects as reference library16BEH I T A A MEC RB I BRAS EE W S S A A C B H AAEGA L L F HHSST A C CE I B CSF EE I I F FURHTBBC Sound EffectsExterior AtmosphereHouseholdInterior AtmosphereTransportationAnimals and BirdsAudiences Children Crowds FootsMiscEuropean SoundscapesComputers Printers PhonesRivers Streams WaterBirdsIndustryBig Ben Taxi Bus AtmospheresRural SoundscapesAutoSport LeisureExplosions Guns AlarmsElectronically Generated SoundsWeather 1Ships And Boats 1Ships And Boats 2AmericaAircraftChinaBabiesHospitalsAfrica The Human WorldAfrica The Natural WorldEquestrian EventsGreeceAdventure SportsLivestock 1Livestock 2Farm MachineryHorsesHorses & DogsSchools & CrowdsSpainTrainsAge Of SteamConstructionCatsEmergencyIstanbulBirdsCrowdsSuburbiaFranceEnglandExterior Atmospheres-Rural BackgroundIndia & Nepal-City LifeIndia Pakistan Nepal-CountrysidFootsteps 1Footsteps 2Urban South AmericaRural South AmericaHungaryThe Czech Republic - Slovakia1000+ examples ... comprehensive?similarity via normalized textures (over 10s chunks)Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35BBC Audio Semantic Classes• Use BBC Sound Effects Library2238 sound files Short keyword descriptions• Train classifiers for 100 most common keywordsSelect 45 best-performing• Added as “semantic units”.. Useful for description?17 SFX001–04–01    Wood Fire Inside Stove  5:07    ! SFX001–05–01    City Skyline City Skyline    9:46    ! SFX001–06–01    High Street With Traffic, Footsteps! SFX001–07–01    Car Wash Automatic, Wash Phase Inside R    ! SFX001–08–01    Motor Cycle Yamaha Rd 350: Motor Cycle    ! SFX001–09–01    Motor Cycle Yamaha Rd 350, Rider Runs U! 290 footsteps       59 men! 267 on              59 general! 240 animals         55 switch! 197 ambience        53 starts! 193 interior        53 crowds! ...                 ...!BBC2238 top 25 mutual AP  walkingstairsstonesirenpavementcountryascendingwarfarewoodfootstepsrunningfuturisticwomanambiencewomenmenboathorseslitreemergencydescendingbabieswoodenanimalsbackgrounds0.20.40.60.81Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Audio Semantic Results• Semantic-level feature fusion helps ~15% relativeResults on TREC MED 2011 DEVT1 (6314 videos)180 0.1 0.2 0.3 0.4 0.5 0.6 E001-Attempting_a_board_trick E002-Feeding_an_animal E003-Landing_a_fish E004-Wedding_ceremony E005-Working_on_a_woodworking_project E006-Birthday_party E007-Changing_a_vehicle_tire E008-Flash_mob_gathering E009-Getting_a_vehicle_unstuck E010-Grooming_an_animal E011-Making_a_sandwich E012-Parade E013-Parkour E014-Repairing_an_appliance E015-Working_on_a_sewing_project mAP mfccs-230 sema100mfcc sbpcas-4000 sema100sbpca sema100sum Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Audio Semantic Results• Browsing results is surprisingly good (sometimes)19Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /353. Foreground: Transient Features• Transients = foreground events?• Onset detector finds energy burstsbest SNR• PCA basis to represent each300 ms x auditory freq• “bag of transients”20Cotton, Ellis, Loui ’11Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Nonnegative Matrix Factorization• Decompose spectrograms into templates + activationfast & forgiving gradient descent algorithm2D patchessparsity controlcomputation time...21Smaragdis  Brown ’03Abdallah  Plumbley ’04Virtanen ’07X =W ·HOriginal mixtureBasis 1(L2)Basis 2(L1)Basis 3(L1)freq / Hz44288313982203347454780 1 2 3 4 5 6 7 8 9 10time / sRecognizing Environmental Sounds - Dan Ellis 2012-10-24       /35NMF Transient Features• Learn 20 patches from Meeting Room Acoustic Event data• Compare to MFCC-HMM detector22Cotton, Ellis ’11freq (Hz)442139834744421398347444213983474442139834740.0 0.5 0.0 0.5time (s)0.0 0.5 0.0 0.5 0.0 0.5Patch 1 Patch 2 Patch 3 Patch 4 Patch 5Patch 6 Patch 7 Patch 8 Patch 9 Patch 10Patch 11 Patch 12 Patch 13 Patch 14 Patch 15Patch 16 Patch 17 Patch 18 Patch 19 Patch 2010 15 20 25 30050100150200250300SNR (dB)Acoustic Event Error Rate (AEER%)Error Rate in Noise  MFCCNMFcombined• NMF more noise-robustcombines well ...Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Foreground Event Localization• Global vs. local class modelstell-tale acoustics may be ‘washed out’ in statisticstry iterative realignment of HMMs:“background” model shared by all clips23Old Way:All frames contributeNew Way:Limited temporal extentstime / sfreq / kHzYT baby 002:  voice  baby  laugh5 10 1501234voice babylaughtime / sfreq / kHz5 10 1501234voice laughlaughbgbgvoice baby babyK Lee, Ellis, Loui ’10Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Foreground Event HMMs• Training labels only at clip-level• Refine models by EM realignment• Use for classifying entire video...or seeking to relevant part24−500            Spectrogram : YT_beach_001.mpeg, Manual Clip-level Labels : “beach”, “group of 3+”, “cheer”0 1 2 3 4 5 6 7 8 902k4kViterbi Path by 27−state Markov Modelfreq / Hz10animalbabybeachbirthdayboatcrowdgraduationgroup of 3+group of 2museumnightone personparkpicnicplaygroundshowsportssunsetweddingdancingparadesingingskicheermusicglobal BG1global BG225 concepts + 2 global BGstime/secManual Frame-level Annotationwater wavecheerunseen1speaker2speaker3 cheervoicevoicevoice voice voicespeaker4Background1k3kdrum windetcvoicevoiceLee & Ellis ’10Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Refining Ground Truth• Audio Ground Truth at coarse time resolutionbetter-focused labels give better classifiers?but little information in very short time frames• Train classifiers on shorter (2 sec) segments?Initial labels applyto whole clipRelabel based onmost likely segmentsin clipRetrain classifier25TagsVideo 0EwtL2yr0y8ParadeTime LabelsFeaturesTime LabelsFeaturesPosteriors03FF1Sz53YU-ParadeParade PSVM TrainSVM ClassifySVM ClassifySVM TrainIteration 0Iteration 1Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Refining Ground Truth• Refining labels is “Multiple Instance Learning”“Positive” clips have at least one +ve frame“Negative” clips are all –ve• Refine based on previous classifier’s scores26TextThreshold from CDFs of +ve and –ve framesRecognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Refining Ground Truth• Score by aggregating frame labels to whole clip• Systems improve for 3-5 epochs, ~10% total270 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Basketball Baseball Soccer IceSkating Skiing Swimming Biking Cat Dog Bird Graduation Birthday WeddRecep WeddCerem WeddDance MusPerf NonMusPerf Parade Beach Playground mAP mfcc-230-2s-E1 mfcc-230-2s-E2 mfcc-230-2s-E3 Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /354. Open Issues• Better object/event separationparametric modelsspatial information?computational auditory scene analysis...• Large-scale analysis• Integration with video28Frequency Proximity HarmonicityCommon OnsetBarker et al. ’05Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Audio Annotation• Co-ordinating program-wide effort to share labelsDefining label setCreating labeled dataCMU 43 label set as basis• Efficiency of labeling is keyAudio-only vs. audio+video?Coarse-level human labels  + automatic refinement?29   CoLordina7ng$programLwide$eﬀort$to$share$labels$-  Deﬁning*label*set*-  Crea9ng*labeled*data*-  CMU*43*label*set*as*basis*  Eﬃciency$of$labeling$is$key$-  AudioDonly*vs.*audio+video?*-  CoarseDlevel*human*labels***+*automa9c*reﬁnement?*Audio*Annota9on*(c)*IBM*Corpora9on*and*Columbia*University*2012D10D22*animal singing clatter anim_bird music_sing rustle anim_cat music scratch anim_ghoat knock hammer anim_horse thud washboard human_noise clap applause laugh click whistle scream bang squeak child beep tone mumble engine_quiet sirene speech engine_light water speech_ne power_tool micro_blow radio engine_heavy wind white_noise cheer other_creak crowd Burger & Metze, CMU, 2012Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Audio-Visual Atoms• Object-related features from both audio (transients) & video (patches)30Joint Audio-Visual Atom discoverytime. . .visual atom. . .visual atom. . .visual atom. . .timeaudio spectrumfrequencytimefrequencyaudio atomaudio atomaudio atomJiang et al. ’09Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Audio-Visual Atoms• Multi-instance learning of A-V co-occurrences 31audio atomaudio atom…visual atomtimeN audio atoms…M Visual Atoms… …visual atomtimecodebook learning Joint A-V CodebookM×N A-V Atoms…Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Audio-Visual Atoms32Audio onlyVisual onlyVisual + Audioblack suit+ romanticmusicWeddingmarchingpeople+ paradesoundParadesand+ beachsoundsBeachRecognizing Environmental Sounds - Dan Ellis 2012-10-24       /35Summary• Machine Listening:Getting useful information from sound• Background sound classification... from whole-clip statistics?• Foreground event recognition... by focusing on peak energy patches• Speech content is very important... separate with pitch, models, ...33Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35References• Jon Barker, Martin Cooke, & Dan Ellis, “Decoding Speech in the Presence of Other Sources,” Speech Communication 45(1): 5-25, 2005.• Courtenay Cotton, Dan Ellis, & Alex Loui, “Soundtrack classification by transient events,” IEEE ICASSP, Prague, May 2011.• Courtenay Cotton & Dan Ellis, “Spectral vs. Spectro-Temporal Features for Acoustic Event Classification,” submitted to IEEE WASPAA, 2011.• Dan Ellis, Xiaohong Zheng, Josh McDermott, “Classifying soundtracks with audio texture features,” IEEE ICASSP, Prague, May 2011.• Wei Jiang, Courtenay Cotton, Shih-Fu Chang, Dan Ellis, & Alex Loui, “Short-Term Audio-Visual Atoms for Generic Video Concept Classification,” ACM MultiMedia, 5-14, Beijing, Oct 2009.• Keansub Lee & Dan Ellis, “Audio-Based Semantic Concept Classification for Consumer Video,” IEEE Tr. Audio, Speech and Lang. Proc. 18(6): 1406-1416, Aug. 2010.• Keansub Lee, Dan Ellis, Alex Loui, “Detecting local semantic concepts in environmental sounds using Markov model based clustering,” IEEE ICASSP, 2278-2281, Dallas, Apr 2010.• Byung-Suk Lee & Dan Ellis, “Noise-robust pitch tracking by trained channel selection,” submitted to IEEE WASPAA, 2011.• Andriy Temko & Climent Nadeu, “Classification of acoustic events using SVM-based clustering schemes,” Pattern Recognition 39(4): 682-694, 2006• Ron Weiss & Dan Ellis, “Speech separation using speaker-adapted Eigenvoice speech models,” Computer Speech & Lang. 24(1): 16-29, 2010.• Ron Weiss, Michael Mandel, & Dan Ellis, “Combining localization cues and source model constraints for binaural source separation,” Speech Communication 53(5): 606-621, May 2011.• Tong Zhang & C.-C. Jay Kuo, “Audio content analysis for on-line audiovisual data segmentation,” IEEE TSAP 9(4): 441-457, May 200134Recognizing Environmental Sounds - Dan Ellis 2012-10-24       /35AcknowledgmentSupported by the Intelligence Advanced Research Projects Activity (IARPA) via Department of Interior National Business Center contract number D11PC20070. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright annotation thereon. Disclaimer:  The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DoI/NBC, or the U.S. Government.35

Recognizing and Classifying Environmental Sounds

https://academiccommons.columbia.edu/doi/10.7916/D89C75P9/download

Recognizing and Classifying Environmental Sounds

Abstract

Similar works

Full text

Available Versions

Columbia University Academic Commons