Discusses speech recognition, auditory scene analysis, and content-based indexing and retrieval

Ellis, Daniel P. W.

English

Columbia University Academic Commons

 08 - 1 Sound content analysisfor indexing and understanding A Sound content analysis  - Dan Ellis 2000-02-Dan EllisInternational Computer Science Institute, Berkeley C<dpwe@icsi.berkeley.edu>OutlineSound content analysisSpeech recognitionAuditory scene analysisAudio content indexingConclusions12345 08 - 2 Sound content analysis d     112 time/ssns/eventsSound content analysis  - Dan Ellis 2000-02-• Overall goal:  ‘Useful’ data from soun- which depends on the goal• Involving:- continuous fi  discrete- source separation- extract ‘semantic’ content0 2 4 6 8 10frq/Hz020004000wordactio 08 - 3 The space of sound analysis researcherceptionionalceneistorylingSound content analysis  - Dan Ellis 2000-02-abstractconcreteengineering pSpeechrecognitionComputatauditory sanalysAudimodeCoding/compressionContent-basedretrieval 08 - 5 Outline   Sound content analysis  - Dan Ellis 2000-02-Sound content analysis Speech recognition- Classic speech recognition- The connectionist-HMM hybrid- Strength through combinationsAuditory scene analysisAudio content indexingConclusions12345 08 - 6 Speech recognition: Dictation              ={ S 1 .. S N }   ne wordsy  probs.2n prob.rwordsanguage model("sat"|"the","cat")("saw"|"the","cat")Sound content analysis  - Dan Ellis 2000-02-• Observations X = {X1..XN} fi  States S• State sequence {Si} (e.g. phones) defi• Training (on large datasets) is the ke- EM iteration for acoustic & transition S * P S X( )Sargmax =P Xi Si( ) P Si Si 1–( )×iÕSargmax =P S X,( )P X( )------------------Sargmax =acoustic prob. transitioMarkov assumptionFeaturecalculationsoundAcousticclassifierfeaturevectorsHMMdecodephonelikelihoodsWord models Lpps ah t 08 - 7 The connectionist-HMM hybrid (Morgan & Bourlard, 1995)            n mixtures            n acoustics Language modelMMcoderWordsSound content analysis  - Dan Ellis 2000-02-• P(Xi|Si) is acoustic likelihood model- model distribution with, e.g., Gaussia• Replace with posterior, P(Si|Xi):- neural network estimates phone give- discriminative• Simpler structure for researchC0C1C2Cktntn+wh#pclbcltcldclFeature calculationsoundNeural network acoustic classifierPosterior probabilitiesfor context-independentphone classesWordmodelsHde 08 - 8 Visualizing speech recognitionSound content analysis  - Dan Ellis 2000-02-FeaturecalculationsoundAcousticclassifierfeature vectorsNetworkweightsHMMdecoderphone probabilitiesphone & word labelingWord modelsLanguage model 08 - 9 Combination schemes • How to use complementary features?inalothesesWordpothesesWordpotheses“^”“•”Sound content analysis  - Dan Ellis 2000-02-Feature 1calculationAcousticclassifierHMMdecoderFeature 2calculationInputsoundAcousticclassifierSpeechfeaturesHMMdecoderPhoneprobabilitiesWordhypothesesHypothesiscombinationFhypFeature 1calculationAcousticclassifierHMMdecoderFeature 2calculationInputsoundAcousticclassifierSpeechfeaturesPhoneprobabilities hyPosteriorcombinationFeature 1calculationAcousticclassifierHMMdecoderFeature 2calculationInputsoundSpeechfeaturesPhoneprobabilities hyFeaturecombination 8 - 10 Combining feature streams o models? uicklyteractions e): mpared to  ne WER ratio 97.6%89.6%01.1%85.8%76.5%74.1%63.0%Sound content analysis  - Dan Ellis 2000-02-0• How to allocate feature dimensions t- lower-dimension models train more q- higher-dimension models find more in• PLP & MSG for Aurora (digits in nois- PLP are ‘conventional’ features- MSG developed as robust alternative- Evaluate by word-error rate (WER) codefault baselineFeatures Parameters baseliplp12•dplp12 136kplp12^dplp12 124kmsg3a•msg3b 145k 1msg3a^msg3b 133kplp12•dplp12•msg3a•msg3b 281kplp12^dplp12^msg3a^msg3b 245kplp12^dplp12•msg3a^msg3b 257k 8 - 11 Tandem connectionist models (with Hermansky et al., OGI)  models? e phones) r alone! t structure  ing data o HTKcoderTandem systemoutputSound content analysis  - Dan Ellis 2000-02-0• How can we combine neural net & GM- (GMM system does not know they ar• Result: better performance than eithe- neural net has trained discriminatively- GMM HMMs learn context-dependenfi extract complementary info from trainSystem-features baseline WER ratiHTK-mfcc 100.0%Hybrid-mfcc 84.6%Tandem-mfcc 64.5%Tandem-plp+msg 47.2%Pre-nonlinearityoutputsPCAorthogn'nHTKGM modelSpeechfeaturesFeaturecalculationInputsoundNeuralnet model(Posteriordecoder)(Phoneprobabilities)(Hybrid systemoutput)SubwordlikelihoodsdeOthogonalfeatures 8 - 12 Aurora “Distributed SR” evaluation  lcommandem2dBementSound content analysis  - Dan Ellis 2000-02-0• 7 telecoms company submissions:- Tandem systems from OGI-ICSI-Qua-20.00%0.00%20.00%40.00%60.00%80.00%100.00%Baseline S1 S2 S3 S4 S5Tandem1 S6TAvg. WER -20-0Baseline improvAurora DSR Evaluation 1999 Results 8 - 13 Outstanding issues in speech recognition   s?Sound content analysis  - Dan Ellis 2000-02-0• Are we on the right path?- useful dictation products exist- evaluation results improve every year- .. but appear to be asymptoting• Is dictation enough?- a useful focus initially- .. but not speech understanding- .. and has skewed research• What should be our research prioritie- straight ASR research is hard to fund- need to look at harder domains- need to connect it to applications8 - 14OutlineSound content analysis  - Dan Ellis 2000-02-0Sound content analysis Speech recognitionAuditory scene analysis- Psychological phenomena- Computational modeling- Prediction-driven analysis- Incorporating speechAudio content indexingConclusions123458 - 15Auditory Scene Analysis (ASA)onarn39time/s−70−60−50−40dBSound content analysis  - Dan Ellis 2000-02-0“The organization of sound scenes according to their inferred sources”• Sounds rarely occur in isolation- need to ‘separate’ for useful informati• Human audition is very effective- computational models have a lot to le200400100020004000f/Hz city220 1 2 3 4 5 6 7 88 - 16Psychology of ASAoids, noise) 1990]tssing cues:monicity, tiesSourcepropertiesSound content analysis  - Dan Ellis 2000-02-0• Extensive experimental research- perception of simplified stimuli (sinus• “Auditory Scene Analysis” [Bregman- first: break mixture into small elemen- elements are grouped in to sources u• Grouping ‘rules’ (Darwin, Carlyon, ...)- common onset/offset/modulation, harspatial location, ...- relate to intrinsic (ecological) regulariFrequencyanalysisGroupingmechanismOnsetmapHarmonicitymapPositionmap(after Darwin, 1996)8 - 17Computational Auditory Scene Analysis(CASA)n 1992):esSourcegroupsSound content analysis  - Dan Ellis 2000-02-0• Literal model of Bregman... (e.g. Brow• Goals- identify and segregate different sourc- resynthesize separate outputs!inputmixturesignalfeatures(maps)discreteobjectsFront end ObjectformationGroupingrulesonsetperiodfrq.modtimefreq8 - 18Grouping model results0.8 1.0 time/sSound content analysis  - Dan Ellis 2000-02-0• Able to extract voiced speech:• Limitations- resynthesis via filter-mask- only periodic targets- robustness of discrete objects0.2 0.4 0.6 0.8 1.0 time/s1001502003004006001000150020003000frq/Hzbrn1h.aif0.2 0.4 0.61001502003004006001000150020003000frq/Hzbrn1h.fi.aif8 - 19Context, expectations & predictionsd as an 100s of msevelSound content analysis  - Dan Ellis 2000-02-0Perception is not directbut a search for plausible hypotheses• Bregman’s “old-plus-new” principle:A change in a signal will be interpreteadded source whenever possible• E.g. the ‘continuity illusion’:- tones alternates with noise bursts- noise is strong enough to mask tone... so listener discriminate presence- continuous tone perceived for gaps ~fi Inference acts at low, preconscious l100020004000f/Hz ptshort0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4time/s8 - 20Modeling top-down processing:‘Prediction-driven’ CASA (PDCASA):ergy elementsourceroupsictineSound content analysis  - Dan Ellis 2000-02-0• Data-driven...vs. Prediction-driven• PDCASA key features:- ‘complete explanation’ of all scene en- vocabulary of periodic/noise/transient- multiple hypotheses- explanation hierarchyinputmixturesignalfeaturesdiscreteobjectsFront end ObjectformationGroupingrulesSginputmixturesignalfeaturespredictionerrorshypothesespredictedfeaturesFront end Compare& reconcileHypothesismanagementPred& combPeriodiccomponentsNoisecomponents8 - 21PDCASA for the continuity illusione portions:Sound content analysis  - Dan Ellis 2000-02-0• Subjects hear the tone as continuous... if the noise is a plausible masker• Data-driven analysis gives just visibl• Prediction-driven can infer masking:100020004000f/Hz ptshort0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4i /8 - 22PDCASA analysis of a complex scene−70−60−50−40dBf/Hz CitySound content analysis  - Dan Ellis 2000-02-0200400100020004000f/Hz Noise1200400100020004000f/Hz Noise2,Click12004001000200040000 1 2 3 4 5 6 7 8 9501002004001000Horn1 (10/10)Crash (10/10)Horn2 (5/10)Truck (7/10)Horn3 (5/10)Squeal (6/10)Horn4 (8/10)Horn5 (10/10)0 1 2 3 4 5 6 7 8 9time/s200400100020004000f/Hz Wefts1−4501002004001000Weft5 Wefts6,7 Weft8 Wefts9−128 - 23CASA for speech recognitionstructureé)tognizerentesesl domainPredict& combineSound content analysis  - Dan Ellis 2000-02-0• Data-driven: CASA as preprocessor- problems with ‘holes’ (but: Okuno)- doesn’t exploit knowledge of speech • Missing data (Cooke &c, de Cheveign- CASA cues distinguish present/absen- RESPITE project: modifications to rec• Prediction-driven: speech as compon- same ‘reconciliation’ of speech hypoth- need to express ‘predictions’ in signa inputmixture Front end Compare& reconcileHypothesismanagementPeriodiccomponentsNoisecomponentsSpeechcomponents8 - 24Other signal-separation approachestlywski ’94)aximizing Sound content analysis  - Dan Ellis 2000-02-0• HMM decomposition (RK Moore ’86)- recover combined source states direc• Blind source separation (Bell & Sejno- find exact separation parameters by mstatistic e.g. signal independencemodel 1model 2observations8 - 25Outstanding issues in CASAtionSound content analysis  - Dan Ellis 2000-02-0• What is the architecture?- data-driven versus prediction-driven- representations at different levels- hypothesis search• How to combine different cues?- priority of different cues- resolving conflicting cues- bottom-up versus top-down• How to exploit training data?- .. the big lesson from speech recogni• Evaluation- .. a more subtle lesson8 - 26OutlineSound content analysis  - Dan Ellis 2000-02-0Sound content analysis Speech recognitionAuditory scene analysisAudio content indexing- Spoken document retrieval- Handling nonspeech audio- Object-based analysis and retrieval- Audio-video content organizationConclusions123458 - 27Audio content indexing:R)as indexest greatL ELECTION E LEADER REE  EPISODE OF RVATIVE PARTY O WOOD BUYS  THIS IS l (IR)4Sound content analysis  - Dan Ellis 2000-02-0Spoken document retrieval (SD• Idea: speech recognition transcripts • Best broadcast news systems are no- 15-30% WER on real broadcasts• Word errors vary in their impact:F0: THE VERY EARLY RETURNS OF THE NICARAGUAN PRESIDENTIASEEMED TO FADE BEFORE THE LOCAL MAYOR ON A LOT OF LAWF4: AT THIS STAGE OF THE ACCOUNTING FOR SEVENTY SCOTCH ONDANIEL ORTEGA IS IN SECOND PLACE THERE WERE TWENTY THPRESIDENTIAL CANDIDATES OF THE ELECTIONF5: THE LABOR MIGHT DO WELL TO REMEMBER THE LOST A MAJORTRANSATLANTIC CONNECT TO A CORPORATION IN BOTH CONSEOFFICIALS FROM BRITAIN GOING TO WASHINGTON THEY WENT TGEORGE BUSH ON HOW TO WIN A SECOND TO NONE IN LONDONSTEPHEN BEARD FOR MARKETPLACE• Good enough for information retrieva- e.g. TREC-8 average precision:reference transcript ~ 0.530% WER ~ 0.48 - 28Thematic Indexing of Spoken Language(with Sheffield, Cambridge, BBC)tedSound content analysis  - Dan Ellis 2000-02-0• SDR for BBC broadcast news archive- 1000+ hr archive, automatically upda8 - 29Speech and nonspeech(with Gethin Williams)odel: differencesic testset0.2 0.25 0.3Dynamismond segments of speecn/musicSpeech      Music       Speech+MusicSound content analysis  - Dan Ellis 2000-02-0• ASR run over entire soundtracks?- for nonspeech, result is nonsense• Watch behavior of speech acoustic m- average per-frame entropy- ‘dynamism’ - mean-squared 1st-order• 1.3% error on 2.5 second speech-mu0 0.05 0.1 0.15Dynamism vs. Entropy for 2.5 sec00.511.522.533.5Entropy020004000frq/Hz0 2 4 6 8 10 12 time/s02040SpectrogramPosteriorsspeech music speech+music8 - 30Element-based audio indexingesX libraries..gsResultsSound content analysis  - Dan Ellis 2000-02-0• Search for nonspeech audio databas- e.g. Muscle Fish ‘SoundFisher’ for SF• Segment-level features- well-performing features:spectral centroid, dynamics, tonality .• Each segment is an object- not applicable to continuous recordinSegmentfeatureanalysisSound segmentdatabaseSegmentfeatureanalysisSeach/comparisonQuery exampleFeature vectors8 - 31Object-based audio indexingtch subsetsopertiesarmonicity)Seach/comparisonResultsSound content analysis  - Dan Ellis 2000-02-0• Using ‘generic sound elements’- decompose sound into elements; ma- how to generalize?- how to use segment-style features?• Form into objects for higher-order pr- CASA-type object formation (onset, hGenericelementanalysisContinuous audioarchiveGenericelementanalysisQuery exampleElement representationsObjectformation(Objectformation)Word-to-classmappingObjects + propertiesProperties aloneSymbolic queryrushing water8 - 32Audio-video organization & retrievalogicresultssymbolicIRengineQuerySound content analysis  - Dan Ellis 2000-02-0• How it might work...audiovideoAudio-videodata analSpeechprocessingNonspeechanalysisVideofeaturesA-VmatchBoundaries &structuringIndexingtermsSummarization8 - 33AV indexing componentstitionwsthematicSound content analysis  - Dan Ellis 2000-02-0• Recovering broad temporal structure- speaker turns ; speech & music ; repe- characteristic of genres e.g. news sho- indexible attributes in themselves• Posing queries:- term-based- proximity to examples- dynamic audio-visual sketches?• How to define index/query terms?- different kinds of terms: literal versus - machine learning of event classes• Summarization- for displaying ‘hits’: impacts usability- text / image / video / sound- tricks e.g. to find most salient words8 - 34Open issues in audio indexing(not WER)entsEG4SA)sSound content analysis  - Dan Ellis 2000-02-0• Information from speech- multiple, confidence-tagged results? - prosodics; emphasis; speaking style- speaker tracking, identity, character• Information from nonspeech- how to define objects- how to match symbolic search terms• Integrating audio and video- combining information for search elem- forms of query• Related applications- ‘structured content’ encoders (e.g. MP- semantic hearing aids ; robot monitor8 - 35OutlineSound content analysis  - Dan Ellis 2000-02-0Sound content analysis Speech recognitionAuditory scene analysisAudio content indexingConclusions123458 - 36Conclusions:ysis toolssis:intstion5Sound content analysis  - Dan Ellis 2000-02-0The state of sound content anal• Speech recognition:- focussed application, practical results- powerful statistical pattern recognition- able to exploit large training sets• Computational Auditory Scene Analy- real-world sounds are mixtures- discover advanced ecological constra- results still rather preliminary• Content-based retrieval:- compelling problem; forgiving applica- leveraging audio-visual correlations- fertile ground for research

Sound content analysis for indexing and understanding

https://academiccommons.columbia.edu/doi/10.7916/D8C254K9/download

Sound content analysis for indexing and understanding

Abstract

Similar works

Full text

Available Versions

Columbia University Academic Commons