This paper demonstrates how unsupervised cross-lingual adaptation of HMM-based speech synthesis models may be performed without explicit knowledge of the adaptation data
language. A two-pass decision tree construction technique is deployed for this purpose. Using parallel translated datasets, cross-lingual and intralingual adaptation are compared in a controlled manner. Listener evaluations reveal that the
proposed method delivers performance approaching that of unsupervised intralingual adaptation

Byrne, WJ

Gibson, M

Hirsimaki, T

Karhila, R

Kurimo, M

English

This paper demonstrates how unsupervised cross-lingual adaptation of HMM-based speech synthesis models may be performed without explicit knowledge of the adaptation data language. A two-pass decision tree construction technique is deployed for this purpose. Using parallel translated datasets, cross-lingual and intralingual adaptation are compared in a controlled manner. Listener evaluations reveal that the proposed method delivers performance approaching that of unsupervised intralingual adaptation

Apollo (Cambridge)

UNSUPERVISED CROSS-LINGUAL SPEAKER ADAPTATION FOR HMM-BASED SPEECHSYNTHESIS USING TWO-PASS DECISION TREE CONSTRUCTIONMatthew Gibson1, Teemu Hirsim̈aki2, Reima Karhila2, Mikko Kurimo2, William Byrne11Cambridge University Engineering Department, Trumpington Street, Cambridge CB2 1PZ, U.K.2Helsinki University of Technology, Adaptive Informatics Research Centre, P.O.Box 5400, FinlandABSTRACTThis paper demonstrates howunsupervisedcross-lingualadaptation of HMM-based speech synthesis models may beperformed without explicit knowledge of the adaptation datalanguage. A two-pass decision tree construction technique isdeployed for this purpose. Using parallel translated datasets,cross-lingual and intralingual adaptation are compared ina controlled manner. Listener evaluations reveal that theproposed method delivers performance approaching that ofunsupervisedintralingual adaptation.Index Terms— HMM-based speech synthesis, unsuper-vised speaker adaptation, cross-lingual.1. INTRODUCTIONCross-lingual (or interlingual) speaker adaptation is definedas the adaptation of acoustic models associated with one lan-guage, thetarget language, using adaptation data uttered ina different language, thesource language. Recent work [1,2] has addressed the task of supervised cross-lingual adap-tation for HMM-based speech synthesis (or text-to-speech,TTS). This work used TTS models of both source and tar-get languages, and defined a phoneme or state-level map-ping between the source and target language acoustic mod-els. This mapping was deployed during cross-lingual adapta-tion to translate the source transcription to a target languagephoneme or state sequence.Techniques similar to those described above rely upon theavailability of both source and target language TTS models,and the mapping mechanism between these models must beestablished prior to adaptation. The work described in this pa-per quantifies the value of such prior knowledge by adoptingan alternative approach which requires no knowledge of thesource language acoustic model (or source language) and itsrelationship to the target language acoustic model. Further,this alternative technique is applied to the task ofunsuper-visedcross-lingual adaptation.Using parallel translated adaptation datasets recorded bythe same speaker,intralingual (within-language) and cross-lingual adaptation are compared in a controlled manner. Lis-tener evaluations reveal that the proposed unsupervised cross-lingual adaptation delivers performance approaching that ofunsupervised intralingual adaptation.The paper is structured as follows. Sections 2 and 3 re-spectively discuss the idea and implementation of the cross-lingual adaptation technique. Section 4 describes the experi-mental procedure used to evaluate the technique and Section5 analyses the results of the evaluation. Section 6 summarisesthe contributions of this work.2. UNSUPERVISED CROSS-LINGUAL ADAPTATIONOF SPEECH SYNTHESIS MODELSThe cross-lingual adaptation technique used in this worktreats the source language adaptation data as if it were utteredin the target language. Target language acoustic models anda phoneme-loop grammar are used to recognise the adap-tation data, thus mapping it onto a phoneme sequencep̂ inthe target language. Subsequently, the estimated phonemesequencêp is used as the reference sequence by the speakeradaptation algorithm. This process is similar to standardapproaches to unsupervised intralingual adaptation. Notethat to avoid language-specific constraints, no dictionary orlanguage model is used during recognition.Several advantages are associated with this technique.Firstly, a pre-trained source language acoustic model is notrequired. Secondly, no previously learned mapping betweensource and target language acoustic models is necessary.Thirdly, the technique may be applied even when the sourcelanguage is unknown. However, it may be argued that auto-matic mapping of source language speech to a target languagephoneme sequence is suboptimal. The impact of this auto-matic mapping is measured in Section 4.To perform unsupervised adaptation in this way, it isnecessary to perform automatic speech recognition (ASR)with the target TTS models to obtain transcriptionp̂. Thisis problematic because the acoustic models typically usedin HMM-based speech synthesis are not easily integratedinto the ASR search procedure. This, in turn, is becausethe context-dependent acoustic models used in HMM-basedspeech synthesis represent suprasegmental information (e.g.syllabic stress). However, these models, henceforth referredto asfull contextmodels, may be constructed such that theymay be mapped to triphone acoustic models suitable for ASR.This technique, introduced in [3], is summarised in Section 3.3. TWO-PASS DECISION TREE CONSTRUCTIONFull context models are clustered using a decision tree toenable robust estimation of their parameters. By impos-ing constraints upon the decision tree structure, multiple-component triphone mixture models may be derived fromsingle-component full context models. This constrained de-cision tree construction process is illustrated in Figure 1.The first stage, indicated as Pass 1 in Figure 1, uses onlyquestions relating to left, right and central phonemes to con-struct a phonetic decision tree. This decision tree is used togenerate a set of tied triphone contexts, which are easily inte-grated into the ASR search. Pass 2 extends the decision treeconstructed in Pass 1 by introducing additional questions re-lating to suprasegmental information. The output of Pass 2 isan extended decision tree which defines a set of tied full con-texts. After this two-pass decision tree construction, singlecomponent Gaussian state output distributions are estimatedto model the tied full contexts associated with each leaf nodeof the extended decision tree. These models are then used forspeech synthesis.A mapping from the single-component full context mod-els to multiple-component triphone models is defined as fol-lows. Each leaf node of the extended decision tree has unique‘triphone ancestor’ node; its ancestor leaf node of the Pass 1decision tree. Each set of Gaussian components associatedwith the same ‘triphone ancestor’ are grouped as componentsof a multiple component mixture distribution to model thecontext defined by the ‘triphone ancestor’. The derived tri-phone models are illustrated at the bottom of Figure 1. Themixture weight of each mixture component is calculated fromthe occupancies associated with components of the Pass 2 leafnode contexts. The inverse mapping from triphone models tofull context models is obtained by associating each Gaussiancomponent with its original full context. Given this mappingbetween full context and triphone models, unsupervised adap-tation of full context acoustic models may be achieved viaadaptation of triphone models, as described below.As illustrated in Figure 2, triphone models derived fromfull context models are used to estimate a triphone-level tran-scription of source language adaptation data. This estimatedtranscription is then used to adapt the triphone models. Theadapted triphone models are subsequently mapped back tofull context models using the inverse mapping.4. EXPERIMENTSFull context English average voice models are estimatedusing speaker adaptive training (SAT, [4]) and the WallStreet Journal (WSJ) SI84 dataset. Acoustic features usedC-Nasal?L-Vowel?L-Vowel?C-Vowel?…C-Nasal?L-Vowel?C-Vowel?…C-Nasal?R-stressed?2 syllables in utt?…Pass 1Pass 2R-stressed?2 3C-Nasal?L-Vowel?2 syllables in utt?14 5C-Nasal?L-Vowel?54321Full context models (single-component)Triphone models(multi-component)Mapping InversemappingModelmappingFig. 1. Two-pass decision tree construction. Mapping func-tions permit sharing of full context models for TTS and tri-phone models for ASR.Adaptation data(source language)AdaptationEstimated triphonetranscription (target language)RecogniseInversemappingAdapted triphone target language acousticmodels (ASR)Adapted fullcontext target language acoustic models (TTS)Mapping Triphone target language acoustic models (ASR)Full context target language acoustic models (TTS)Fig. 2. Unsupervised cross-lingual adaptation of full contexttarget language acoustic models.are STRAIGHT-analysed Mel-cepstral coefficients [5], fun-damental frequency, band aperiodicity measurements, andthe first and second order temporal derivatives of all features.The acoustic models use explicit duration models [6] andmulti-space probability distributions [7]. Decision trees (oneper state and stream combination) are constructed using thetwo-pass technique of Section 3. Adapted TTS systems arederived from the average voice models using the adaptationmethod described in Section 3 and constrained maximumlikelihood linear regression. Speech utterances are gener-ated from models via feature sequence generation [8] andresynthesis of a waveform from the feature sequence [5].4.1. Adaptation and evaluation datasetsThe adaptation datasets comprise94 utterances from a cor-pus of parallel text of European parliament proceedings [9].English and Finnish versions of this dataset are recorded inidentical acoustic environments by a native Finnish speakeralso competent in English. Statistics relating to these datasetsare provided in Table 1. The evaluation dataset comprises En-Language # utterances # minutes # wordsEnglish 94 12.3 1546Finnish 94 10.9 1066Table 1. Europarl adaptation datasets.glish utterances (distinct from the adaptation utterances) fromthe same Europarl corpus.4.2. EvaluationThe following systems are evaluated.• System A: average voice.• System B: unsupervised cross-lingual adapted.• System C: unsupervised intralingual adapted.• System D: supervised intralingual adapted.• System E: vocoded natural speech.System B is the result of applying unsupervised cross-lingualadaptation to the average voice models using the Finnishadaptation dataset. System C results from unsupervisedadaptation using the English adaptation dataset. System D isidentical to System C with the exception that the correct tran-scription is used during adaptation. System E analyses andresynthesises the evaluation utterances using STRAIGHT[5].All systems were evaluated by listening to synthesised ut-terances via a web browser interface, as used in the BlizzardChallenge 2007. The evaluation comprised four sections. Inthe first pair of sections, listeners judged the naturalness ofan initial set of synthesised utterances. In the second pair ofsections, listeners judged the similarity of a second set of syn-thesised utterances to the target speaker’s speech. Four of thetarget speaker’s natural English utterances were available forcomparison. Each synthetic utterance was judged using a fivepoint psychometric response scale, where ‘5’ and ‘1’ are re-spectively the most and least favourable responses.Twenty-four native English and sixteen native Finnishspeakers conducted the evaluation. Different Latin squareswere used for each section to define the order in which sys-tems were judged. Each listener was assigned a row of eachLatin square, and judged five different utterances per section,each synthesised by a different system.5. RESULTSFigure 3 summarises listener judgements of ‘similarity to tar-get speaker’ and ‘naturalness’ using boxplots [10] while Ta-ble 2 displays the average mean opinion scores (MOS) ofthese judgements for each system in the columns labelled‘av’. Analysis of these judgements by listener native languageis provided in the columns labelled ‘En’ and ‘Fi’, respectivelydenoting English and Finnish.SysSourceSup?MOS MOSlang. similarity naturalnessEn Fi av En Fi avA - - 1.2 1.1 1.1 2.3 2.4 2.3B Fi N 2.3 1.5 2.0 2.4 2.4 2.4C En N 2.6 1.7 2.2 2.6 2.7 2.7D En Y 2.7 2.0 2.4 2.5 2.8 2.6E - - 4.6 4.6 4.6 3.7 4.1 3.8Table 2. Mean opinion scores of evaluated systems.A difference between two systems is deemed significantif the Bonferoni-corrected pairwise Wilcoxon signed rank test[10] discovers significance at the95% confidence level.5.1. DiscussionThe average similarity to the target speaker given by adaptedsystems (B, C and D) are all significantly greater than that ob-served for the unadapted System A. A significant difference isalso observed between the average similarity of adapted sys-tems B (2.0) and D (2.4). The similarity score for system Dis a reasonable upper limit on the performance ofsupervisedcross-lingual adaptation since system D uses the correct tran-scription of the acoustic data and deploys no potentially sub-optimal mapping between source and target language acousticmodels.The similarity score for system C is a reasonable upperlimit on the performance ofunsupervisedcross-lingual adap-tation for the given imperfect adaptation data transcription.System B yields an average similarity (2.0) approaching thatof System C (2.2). This difference may be due to the addi-tional knowledge used by System C i.e. prior knowledge ofsource language acoustics and a perfect mapping from sourceto target language acoustic models. In this case the small per-formance degradation measures the value of such knowledge.A second possible explanation should be kept in mind, how-ever. The target speaker’s characteristics may change whenspeaking his non-native English. When adapting using na-tive Finnish speech, such alterations are not observed, and sopossibly not captured.A B C D E12345Similarity to target speakerSystemScoreA B C D E12345NaturalnessSystemScoreFig. 3. Listener opinion scores for similarity to target speaker and naturalness.Small increases in naturalness are noted between theadapted systems (B, C and D) and the unadapted systemA. This demonstrates that the improvements in similaritydiscussed above do not compromise the naturalness of thesynthetic speech.Lastly, note that, with respect to speaker similarity,Finnish and English judges display similar patterns. However,on average, Finnish listeners ascribe lower scores. Furtheranalysis is required to explain this observation. Influentialfactors may include familiarity with the target speaker andcultural differences.6. CONCLUSIONThis paper has measured the value of source language knowl-edge upon the task of unsupervised cross-lingual adaptationof HMM-based synthesis models. A cross-lingual adapta-tion technique which uses no such knowledge delivers perfor-mance approaching that of unsupervised intralingual adapta-tion. Future work should measure how well this techniquegeneralises across different speakers and language pairs. Fu-ture work may also determine if significant differences existbetween unsupervised intralingual and cross-lingual adapta-tion, and explain any such differences.7. ACKNOWLEDGEMENTSWe are very grateful to the organizers of the Blizzard Challenge forproviding experimental evaluation scripts. This research was par-tially funded by the European Communitys Seventh Framework Pro-gramme (FP7/2007-2013), grant agreement 213845 (EMIME).8. REFERENCES[1] Y. Wu, S. King, and K. Tokuda, “Cross-lingual speaker adap-tation for HMM-based speech synthesis,” inProceedings ISC-SLP, 2008.[2] Y. Wu, Y. Nankaku, and K. Tokuda, “State mapping basedmethod for cross-lingual speaker adaptation in HMM-basedspeech synthesis,” inProceedings Interspeech, 2009.[3] M. Gibson, “Two-pass decision tree construction for unsuper-vised adaptation of HMM-based synthesis models,” inPro-ceedings Interspeech, 2009.[4] J. Yamagishi, T. Kobayashi, Y. Nakano, K. Ogata, and J. Iso-gai, “Analysis of speaker adaptation algorihms for HMM-based speech synthesis and a constrained SMAPLR adaptationalgorithm,” IEEE Audio, Speech & Language Processing, vol.17(1), pp. 66–83, 2009.[5] H. Kawahara, I. Masuda-Katsuse, and A. Cheveigne, “Re-structuring speech representations using a pitch-adaptive time-frequency smoothing and an instantaneous-frequency-basedF0 extraction: possible role of a repetitive structure in sounds,”Speech Communication, vol. 27, pp. 187–207, 1999.[6] H. Zen, K. Tokuda, T. Masuko, T. Kobayashi, and T. Kitamura,“Hidden semi-Markov model based speech synthesis,” inPro-ceedings Interspeech, 2004.[7] K. Tokuda, T. Masuko, N. Miyazaki, and T. Kobayashi, “Multi-space probability distribution HMM,” IEICE Trans. Inf. &Syst., vol. E85-D(3), pp. 455–464, 2002.[8] K. Tokuda, T. Yoshimura, T. Masuko, T. Kobayashi, and T. Ki-tamura, “Speech parameter generation algorithms for HMM-based speech synthesis,” inProceedings ICASSP, 2000.[9] P. Koehn, “Europarl: A Parallel Corpus for Statistical MachineTranslation,” inMT Summit, 2005.[10] R. Clark, M. Podsiadlo, M. Fraser, C. Mayo, and S. King, “Sta-tistical analysis of the Blizzard Challenge 2007 listening testresults,” inProceedings Blizzard, 2007.

Analysis of speaker adaptation algorihms for HMMbased speech synthesis and a constrained SMAPLR adaptation algorithm,”

Cross-lingual speaker adaptation for HMM-based speech synthesis,”

Europarl: A Parallel Corpus for Statistical Machine Translation,”

Hidden semi-Markov model based speech synthesis,”

N.Miyazaki, andT.Kobayashi, “Multispace probability distribution HMM,”

Restructuring speech representations using a pitch-adaptive timefrequency smoothing and an instantaneous-frequency-based F0 extraction: possible role of a repetitive structure in sounds,”

Speech parameter generation algorithms for HMMbased speech synthesis,”

State mapping based method for cross-lingual speaker adaptation in HMM-based speech synthesis,”

Statistical analysis of the Blizzard Challenge

Two-pass decision tree construction for unsupervised adaptation of HMM-based synthesis models,”

Unsupervised cross-lingual speaker adaptation for HMM-based speech synthesis using two-pass decision tree construction

Matthew Gibson

Teemu Hirsimaki

Reima Karhila

Mikko Kurimo

William Byrne

Crossref

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.295.9072

Unsupervised cross-lingual speaker adaptation for HMM-based speech synthesis using two-pass decision tree construction

Abstract

Similar works

Full text

Available Versions

Apollo (Cambridge)

Crossref