Artificial neural networks (ANN) have proven to be well suited to the task of articulatory feature (AF) recognition. Previous studies have taken a cascaded approach where separate ANNs are trained for each feature group, making the assumption that features are statistically independent. We address this by using ANNs to provide virtual evidence to a dynamic Bayesian network (DBN). This gives a hybrid ANN/DBN model and allows modelling of inter-feature dependencies. We demonstrate significant increases in AF recognition accuracy from modelling dependencies between features, and present the results of embedded training experiments in which a set of asynchronous feature changes are learned. Furthermore, we report on the application of a Viterbi training scheme in which we alternate between realigning the AF training labels and retraining the ANNs

Frankel, Joe

King, Simon

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      A Hybrid ANN/DBN Approach to Articulatory Feature RecognitionCitation for published version:Frankel, J & King, S 2005, A Hybrid ANN/DBN Approach to Articulatory Feature Recognition. in Interspeech2005 - Eurospeech: 9th European Conference on Speech Communication and Technology . InternationalSpeech Communication Association, pp. 3045-3048.Link:Link to publication record in Edinburgh Research ExplorerPublished In:Interspeech 2005 - EurospeechGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 05. Apr. 2019RTSCA Hybrid ANN/DBN Approachto Articulatory FeatureRecognition THEU NI V E R SI TYOFED I N B URGHJoe Frankel and Simon KingCentre for Speech Technology Research, University of Edinburgh{joe,simonk}@cstr.ed.ac.uk•Previously proposed dynamic Bayesian networks(DBN) for articulatory feature (AF) recognition anddeveloped embedded training scheme to overcomelimitations of phone-derived feature labels.•Performance of DBNs comparable to that of ANNs–DBNs model feature dependencies at the statelevel, ensuring robust GMM observation process.–ANNs discriminative classifiers.•Now combine the strengths of ANNs and DBNs togive a hybrid ANN/DBN system which:–models inter-feature dependencies.– incorporates ANN posteriors as virtual evidence.•Yields significant AF recognition accuracy increase.AbstractWider goals:•ASR system in which articulatory features mediatebetween words and acoustic observations.•Automatic learning of pronunciation variation interms of articulatory features.In this work:• Improve articulatory feature recognition through hy-brid ANN/DBN approach.ObjectiveWhy an articulatory feature state representation?•Much of the variation in natural speech is systematicand due to the asynchronous, overlapping nature ofspeech production–describing this variation in terms of phones re-quires context-dependent models.•A multi-stream AF representation allows expressionof coarticulation effects and pronunciation variationas feature insertions, deletions and substitutions–giving improved parsimony and efficient use ofdata.•Basis for language-independent ASR–designed around set of universal constraints.Why use DBNs?•Unified framework in which to integrate variouscomponents of a full recognition system.MotivationDynamic Bayesian Networks•Bayesian network (BN) provides a means of encod-ing dependencies between a set of random variables(RVs).•Graphical specification language:–RVs represented by nodes, dependencies by edges–square/round and unshaded/shaded denote dis-crete/continuous and hidden/observed•Dynamic Bayesian network (DBN): BN repeatedover time, with links across time. HMM special case:time•Training, decoding implemented using GMTK.Virtual evidence (VE)•When incorporating virtual evidence–make direct observation of probability distributionover discrete RV values.time•Similar in spirit to hybrid ANN/HMMASR.DBNs and virtual evidenceArtificial Neural Networks•Training: input-output pairs consist of frames ofacoustic parameters and articulatory feature values.•VE generation: network activations interpreted asposterior pk(fk |yt) for each level of feature Fk, andused to compute a scaled likelihood:pk(yt | fk) ∝ pk(fk |yt)pk(fk)12 MFCCs+energyapproximantvowel  silencefricative  nasal  stopvoicedvoicelesssilenceroundedunrounded  nilsilencehigh  mid  low silencelabiodentaldental  alveolar  velarfrontcentral  backnil  silenceHidden LayerInputs OutputsArtificial Neural Network Feature GroupsAcoustic Featuresdelta−deltasdeltasdynamicstaticsilenceVE from ANNs•Subset of OGI Numbers corpus:–naturally spoken, limited vocabulary (30 words).–detailed transcripts which include diacritics.–parameterization: 12 MFCCs and energy, (10msshift, 25ms windows), first and second derivatives.•Articulatory features:– 6 multi-levelled feature groups.– framewise canonical labels derived from phones.–diacritics incorporated in phone-feature mappings.Data•Factorial state with inter-feature dependencies.•Virtual evidence observation process.2 time-slices shown here in graphical model notation:voicingfrontbackmannerplacestaticroundingfrontbackvoicingmannerplacestaticroundingDBN - topology•Sparse structure of conditional probability tables(CPT) determines allowable feature combinations.•CPTs estimated on canonical labels are very sparse.– increases speed of 6-factorial model inference.Aim of embedded training scheme:•Reduce number of zero-valued cells in CPTs to in-crease number of possible feature combinations–model asynchronous feature changes.•Retain sufficient sparsity that inference is practical.Outline of embedded training scheme:• Initialize by training on canonical labels.•Zero CPT cells incremented to smallminimumvalue.•Embedded training: AF label sequence is known,though label timings are not.– feature combinations with strong acoustic likeli-hood accumulate probability mass giving non-zeroCPT cells. All other CPT cells shrink back to zero.DBN - embedded trainingEvaluation of AF recognition problematic as requirescomparison with phone-derived AF labels. We report:•%AF recognition accuracy averaged over all features–allows asynchrony, though penalizes inser-tions/deletions which speaker may have made.Baseline given by hybrid ANN/HMM – equivalentmodel with feature streams assumed independent•Results show that modelling dependencies throughthe state leads to increased accuracy.% accuracyfeature ANN/HMM ANN/DBNmanner 84.6% 88.9%place 81.6% 87.2%voicing 84.2% 87.2%rounding 84.7% 88.5%front-back 84.1% 88.2%static 81.6% 86.6%overall 83.5% 87.8%•Results of embedded training– increase in number of feature combinationsin output suggests model learning degree ofasynchrony.model accuracy combinationsANN/HMM 83.5% 2498ANN/DBN 87.8% 54ANN/DBN - asynch 87.8% 97•Results after Viterbi training of ANNs–asynchronous ANN/DBN used to realign data, andANNs retrained on relabelled data.– result important as previous attempts at Viterbitraining of ANNs have let to model degenerating.model accuracyANN/DBN 87.8%ANN (Viterbi) /DBN 87.9%AF recognition - resultsThis work on AF recognition has:• Incorporated ANN virtual evidence into DBNs giv-ing a hybrid ANN/DBN feature recognizer.•Shown superior performance over baseline system inwhich features are assumed independent.•Learned asynchronous changes through embeddedtraining of the feature CPTs.For the future:• Integrate into AF-based recognizer, and implement:– embedded training with feature inser-tions/deletions.– automatic learning of pronunciation variation interms of articulatory features.Conclusions

Edinburgh Research Archive

A hybrid ANN/DBN approach to articulatory feature recognition

https://www.pure.ed.ac.uk/ws/files/19879404/frankel_king_interspeech2005_poster.pdf

A hybrid ANN/DBN approach to articulatory feature recognition

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer

Edinburgh Research Archive