Discusses models of source behavior as the way to conquer uncertainty in mixtures

Ellis, Daniel P. W.

Weiss, Ron J.

English

Columbia University Academic Commons

Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /2211.  Source Models and Scene Analysis2.  Using Source Models3.  Adapting Source Models4.  Source Model IssuesLearning, Using, and AdaptingModels in Scene AnalysisDan Ellis &  Ron WeissLaboratory for Recognition and Organization of Speech and AudioDept. Electrical Eng., Columbia Univ., NY USA{dpwe,ronw}@ee.columbia.edu         http://labrosa.ee.columbia.edu/Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /221.  Source Models and Scene Analysis2• Sounds rarely occur in isolation.. so analyzing mixtures (“scenes”) is a problem.. for humans and machines• How to solve this problem?02_m+s-15-evil-goodvoice-fade0 2 4 6 8 10 12 time/sfrq/Hz02000100030004000Voice (evil)StabRumble StringsChoirVoice (pleasant)Analysislevel / dB-60-40-200Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22Approaches to Separation  or combinations ...3ICA•Multi-channel•Fixed filtering•Perfect separation – maybe!CASA•Single-channel•Time-var. filter•Approximate separationtarget xinterference n n^x^-mix+procx^proc maskistftsmix x+nstftModel-based•Any domain•Param. search•Syntheticoutput?mix x+n x^sourcemodelsparam. fitparamssynthesisLearning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22combination physics source modelsSeparation vs. Inference• Ideal separation is rarely possiblemany situations where overlaps cannot be removed• Overlaps → Ambiguityscene analysis = find “most reasonable” explanation• Ambiguity can be expressed probabilisticallyi.e. posteriors of sources {Si} given observations X: P({Si}| X) ∝ P(X |{Si}) ∏i P(Si|Mi)search over all source signal sets {Si} ??• Better source models → better inference4Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22A Simple Example• Source models are codebooksfrom separate subspaces5                 Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22A Slightly Less Simple Example• Sources with Markov transitions6                               Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22What is a Source Model?• Source Model describes signal behaviorencapsulates constraints on form of signal(any such constraint can be seen as a model...)• A model has parametersmodel + parameters → instance• What is not a source model?detail not provided in instancee.g. using phase from original mixtureconstraints on interaction between sourcese.g. independence, clustering attributes7Excitation source g[n]nnResonance filter H(ej!)!Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /222. Using Models: Speech Separation• Cooke & Lee’s Speech Separation Challengepairs of short, grammatically-constrained utterances:<command:4><color:4><preposition:4><letter:25><number:10><adverb:4>       e.g. "bin white by R 8 again"task: report letter + number for “white”(special session at Interspeech ’06)• Separation or Description?8vs.mixseparationwordsidentifytarget energysourceknowledgespeechmodelst-f masking+ resynthesis ASRmix wordsidentifyspeechmodelsfind bestwords modelLearning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22Codebook Models• Given models for sources, find “best” (most likely) states for spectra:can include sequential constraints...• E.g. stationary noise:9{i1(t), i2(t)}= argmaxi1,i2p(x(t)|i1, i2)p(x|i1, i2) =N (x;ci1+ ci2,) combinationmodelinference ofsource statetime / sfreq / mel binOriginal speech0 1 220406080In speech-shaped noise  (mel magsnr = 2.41 dB)0 1 220406080VQ inferred states  (mel magsnr = 3.6 dB)0 1 220406080Roweis ’01, ’03Kristjannson ’04, ’06Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22Speech Recognition Models• Speech recognizers contain speech modelsASR is just argmax P(W | X)• Recognize mixtures with Factorial HMMi.e. two state sequences, one model for each voiceexploit sequence constraints, speaker differences10Varga & Moore ’90model 1model 2 observations / timeLearning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22Speech Factorial Separation• IBM’s 2006 Iroquois speech separation systemKey features:detailed state combinationslarge speech recognizerexploits grammar constraints34 per-speaker models• “Superhuman” performance... in some conditions11Kristjansson, Hershey et al. ’066 dB 3 dB 0 dB - 3 dB - 6 dB - 9 dB050100Same Gender SpeakersSDL Recognizer No dynamics Acoustic dyn. Grammar dyn. Humanwer / %HIGH RESOLUTION SIGNAL RECONSTRUCTIONTrausti KristjanssonMachine Learning and Applied StatisticsMicrosoft Researchtraustik@microsoft.comJohn HersheyUniversity of California, San DiegoMachine Perception Labjhershey@cogsci.ucsd.eduABSTRACTWe present a framework for speech enhancement and ro-bust speech recognition that exploits the harmonic structureof speech. We achieve substantial gains in signal to noise ra-tio (SNR) of enhanced speech as well as considerable gainsin accuracy of automatic speech recognition in very noisyconditions.The method exploits the harmonic structure of speechby employing a high frequency resolution speech model inthe log-spectrum domain and reconstructs the signal fromthe estimated posteriors of the clean signal and the phasesfrom the original noisy signal.We achieve a gain in signal to noise ratio of 8.38 dB forenhancement of speech at 0 dB. We also present recognitionresults on the Aurora 2 data-set. At 0 dB SNR, we achievea reduction of relative word error rate of 43.75% over thebaseline, and 15.90% over the equivalent low-resolution al-gorithm.1. INTRODUCTIONA long standing goal in speech enhancement and robustspeech recognition has been to exploit the harmonic struc-ture of speech to improve intelligibility and increase recog-nition accuracy.The source-filter model of speech assum s that speechis produced by an excitation source (the vocal cords) whichhas strong regular harmonic structure during voiced phonemes.The overall shape of the spectrum is then formed by a fil-ter (the vocal tract). In non-tonal languages the filter shapealone determines which phone component of a word is pro-duced (see Figure 2). The source on the other hand intro-duces fine structure in the frequency spectrum that in manycases varies strongly among different utterances of the samephone.This fact has traditionally inspired the use of smoothrepresentations of the speech spectrum, such as the Mel-frequency cepstral coefficients, in an attempt to accuratelyestimate the filter component of speech in a way that is in-variant to the non-phonetic effects of the excitation[1].There are two observations that motivate the consider-ation of high frequency resolution modelling of speech fornoise robust speech recognition and enhancement. First isthe observation that most noise sources do not have har-monic structure similar to that of voiced speech. Hence,voiced speech sounds should be more easily distinguish-able from environmental noise in a high dimensional signalspace1.0 200 400 600 800 10001015202530354045Noisy vector, clean vector and estimate of clean vectorFrequency [Hz]Energy [dB]x estimatenoisy vectorclean vectorFig. 1. The noisy input vector (dot-dash line), the corre-sponding clean vector (solid line) and the estimate of theclean speech (dotted line), with shaded area indicating theuncertainty of the estimate (one standard deviation). Noticethat the uncertainty on the estimate is considerably larger inthe valleys between the harmonic peaks. This reflects thelower SNR in these regions. The vector shown is frame 100from Figure 2A second observation is that in voiced speech, the signalpower is concentrated in areas near the harmonics of thefundamental frequency, which show up as parallel ridges in1Even if the interfering signal is another speaker, the harmonic structureof the two signals may differ at different times, and the long term pitchcontour of the speakers may be exploited to separate the two sources [2].0-7803-7980-2/03/$17.00 © 2003 IEEE 291 ASRU 2003Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /223. Adapting Source Models• Power of model-based separation depends on detail of model• Speech separation relies on prior knowledge of every speaker?• Can this be practical?1232 64 128 250012345In Train 32 Set  Diff GenderSNR (dB)  Oracle SD SA SA (SD init)Num training speakers32 64 128 250012345Not In Train Set  Diff GenderSNR (dB)Num training speakersLearning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22  Speaker models Speaker subspace basis vectors Other modelsEigenvoices• Idea: Identify manifold in model parameter spacegeneralize without losing detail?13Kuhn et al. ’98, ’00Weiss & Ellis ’07, ’08, ’09Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22Eigenvoice Bases• Mean model280 states x 320 bins= 89,600 dimensions• Eigencomponentsshift formants/colorationadditional components forchannel14Frequency (kHz)Mean Voice  b d g p t k jh ch s z f th v dh m n l r w y iy ih eh ey ae aaaw ay ah aoowuw ax2468Frequency (kHz)Eigenvoice dimension 1  b d g p t k jh ch s z f th v dh m n l r w y iy ih eh ey ae aaaw ay ah aoowuw ax2468Frequency (kHz)Eigenvoice dimension 2  b d g p t k jh ch s z f th v dh m n l r w y iy ih eh ey ae aaaw ay ah aoowuw ax2468Frequency (kHz)Eigenvoice dimension 3  b d g p t k jh ch s z f th v dh m n l r w y iy ih eh ey ae aaaw ay ah aoowuw ax24685040302010024680246802468Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22Speaker-Adapted Separation• Factorial HMM analysiswith tuning of source model parameters = eigenvoice speaker adaptation15Weiss & Ellis ’08Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22Speaker-Adapted Separation• Eigenvoices for Speech Separation taskspeaker adapted (SA) performs midway between speaker-dependent (SD) & speaker-indep (SI)16SISASDLearning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22Combining Spatial + Speech Model• Interaural parameters give    ILDi(ω),  ITDi,  Pr(X(t, ω) = Si(t, ω))• Speech source model can give     Pr(Si(t, ω) is speech signal)• Can combine into one big EM framework...17E-stepM-stepu is: Pr(cell from source i)      phoneme sequenceΘ is: interaural params      speaker paramsWeiss, Mandel & Ellis ’08Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22Combining Spatial + Speech Model• Source models function as priors• Interaural parameter spatial separationEM estimation of TF masks, spatial originsource model prior improves spatial estimate1800.20.40.60.81Ground truth (12.04 dB)Frequency (kHz)0 0.5 1 1.502468DUET (3.84 dB)0 0.5 1 1.5024682D FD BSS (5.41 dB)0 0.5 1 1.502468MESSL (5.66 dB)Frequency (kHz)Time (sec)0 0.5 1 1.502468MESSL SP (10.01 dB)Time (sec)0 0.5 1 1.502468MESSL EV (10.37 dB)Time (sec)  0 0.5 1 1.502468Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /224. Source Model Issues• Model Domainparsimonious expression of constraintsnice combination physics• Tractabilitysize of search spacetricks to speed search/inference• Acquisition *hand-designed vs. learnedstatic vs. short-term• Factorizationindependent aspectshierarchy & specificity *19Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22Learning Source Models• Speech models learned from labeled datasingle, known speaker + transcriptsdata fully aligned to models• Otherwise ...wait for “clear shot”? reinforce based on best-guess separation?ML model updates?[Ozerov et al. 2005]20Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22How Many Models?• More specific models ➝ better separationneed individual dictionaries for “everything”??• Model adaptation and hierarchyspeaker adapted models : base + parametersextrapolation beyond normal• Time scales of model acquisitioninnate/evolutionary (hair-cell tuning)developmental (mother tongue phones) dynamic - the “Bolero” effect21CNBH,  Physiology Department,  Cambridge UniversityRecognition of Scaled Vowels Smith, Patterson, Turner, Kawahara and Irino JASA (2005)Domain of normal speakers/a//u// //e//o/meanpitch / HzVT length ratioSmith, Pattersonet al. ’05Learning, Using, Adapting Models - Dan Ellis 2009-04-23 -    /22Summary & Conclusions• Source models provide the constraints to make scene analysis possible• Eigenvoices (model subspace) can be used to provide detailed models that generalize• It is not clear how to extend this to all possible sounds, present and future• Relevance to perception?22

Learning, Using, and Adapting Models in Scene Analysis

https://academiccommons.columbia.edu/doi/10.7916/D8PV6THS/download

Learning, Using, and Adapting Models in Scene Analysis

Abstract

Similar works

Full text

Available Versions

Columbia University Academic Commons