Barra-Chicote, R

Ferreiros, J

Lopez-Ludena, V

Lorenzo, J

Martinez, B

Montero, J M

Yamagishi, J

English

Current text-to-speech systems are developed using studio-recorded speech in a neutral style or based on acted emotions. However, the proliferation of media sharing sites would allow developing a new generation of speech-based systems which could cope with spontaneous and styled speech. This paper proposes an architecture to deal with realistic recordings and carries out some experiments on unsupervised speaker diarization. In order to maximize the speaker purity of the clusters while keeping a high speaker coverage, the paper evaluates the F-measure of a diarization module, achieving high scores (&gt;85%) especially when the clusters are longer than 30 seconds, even for he more spontaneous and expressive styles (such as talk shows or sports)

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Towards an Unsupervised Speaking Style Voice BuildingFrameworkCitation for published version:Lorenzo, J, Martinez, B, Barra-Chicote, R, Lopez-Ludena, V, Ferreiros, J, Yamagishi, J & Montero, JM2012, 'Towards an Unsupervised Speaking Style Voice Building Framework: Multi-Style SpeakerDiarization'. in Proc. Interspeech 2012: 13th Annual Conference of the International Speech CommunicationAssociation, Portland, Oregon, USA, September 9-13, 2012. ISCA.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Publisher final version (usually the publisher pdf)Published In:Proc. Interspeech 2012General rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 20. Feb. 2015Towards an Unsupervised Speaking Style Voice Building Framework:multi–style speaker diarizationJ. Lorenzo-Trueba1, B. Martinez-Gonzalez1, V. Lopez–Luden˜a1,R. Barra-Chicote1, J. Ferreiros1, J. Yamagishi2, J.M. Montero11Speech Technology Group, ETSIT, Universidad Politecnica de Madrid, Spain2Centre for Speech Technology Research, University of Edinburgh, United Kingdomjaime.lorenzo@die.upm.es, barra@die.upm.esAbstractCurrent text–to–speech systems are developed usingstudio-recorded speech in a neutral style or based onacted emotions. However, the proliferation of mediasharing sites would allow developing a new generationof speech–based systems which could cope with sponta-neous and styled speech. This paper proposes an architec-ture to deal with realistic recordings and carries out someexperiments on unsupervised speaker diarization. In or-der to maximize the speaker purity of the clusters whilekeeping a high speaker coverage, the paper evaluates theF–measure of a diarization module, achieving high scores(>85%) especially when the clusters are longer than 30seconds, even for the more spontaneous and expressivestyles (such as talk shows or sports).Index Terms: expressive speech synthesis, speaker di-arization, speaking styles, voice cloning.1. IntroductionThere has been a sharp increase in the use of computersin every aspect of daily life, from tablets to smart–phones, leading to a proliferation of media sharingsites which provide new data for developing speech–enabled applications. Under this prospects, and with thepurpose of increasing accessibility and ease of use, itbecomes very interesting to provide the machines withincreasingly versatile human–machine interfaces, notonly on speech recognition but also on speech synthesis.Unfortunately, current speech synthesis technologiesshow severe deficiencies when dealing with spontaneoushuman–like speech, a great obstacle when implementingwidespread applications.One of the goals in The Simple4all Project [1] is theautomatic modification of the speaking style of a neu-tral or expressive voice without needing to record a newspeaker under the target expressivity conditions, main-taining the quality of the original models but achieving ahigh style (or emotion) identification rate, and being ableto control the intensity of the expressivity in a continuousrange. The first step for being able to successfully con-trol the speaking style of the synthetic speech is to obtainenough data from speakers with different speaking styleswith which we could build speaking style average mod-els.Another goal of Simple4all is to create the mostportable speech synthesis system possible: one that couldbe automatically (or with limited manual supervision) ap-plied to many domains and tasks, which implies deal-ing with a wide variety of expressive situations and do-mains. In order to use speech collected from the me-dia or from media sharing sites, speech synthesis sys-tems must be robust to the variation of the acoustic andenvironmental conditions. The system mush be able torobustly cope with noisy ASR–processed corpora andwith challenging data such as interviews, debates, homerecordings, political speeches, etc. The use of diariza-tion techniques for speaker–turn segmentation will allowthe system creating homogeneous voices from heteroge-neous recordings, because the number of speakers wouldbe automatically estimated in a fully unsupervised way,and language–independent diarization techniques auto-matically could provide the temporal labels of the turnsof a certain speaker [2, 3].In this work we present the architecture of style–cloning system and an evaluation of the performanceof a speaker diarization system for unsupervisedly–generating clusters of speech from several styled record-ings. Those clusters will be used for building averagestyle models in order to incorporate the expressiveness ofthose recordings to the synthetic voice of new speakers.2. Unsupervised Multi-Speaker ExpressiveVoice Building FrameworkConsidering the amount of multimedia data currentlyavailable in the Internet and in the media (television, ra-dio, pod-casts, audio–books, etc.) we define an unsuper-vised voice–building framework that will allow develop-ing multiple voices with different speaking styles. Thearchitecture shown in Figure 1 will minimise the manualFigure 1: Unsupervised Multi-Speaker Expressive VoiceBuilding Frameworkprocessing of the multimedia data and the requirement ofexpert knowledge.This framework imposes the use of many speechprocessing technologies. The implementation of thisframework is reasonable due to the availability of open–source technologies: Voice Activity Detection (VAD)[4], Speaker Diarization [5], Speech/Music Segmenta-tion [6], Automatic Speech Recognition [7, 8], automaticSpeech-Text Time Alignment [9, 10], HMM-based Syn-thesis [11, 12] and Speaker Adaptation [13].In this paper we analyse the performance of the un-supervised cluster generation module whose output willbe used to train average voices of the recorded speak-ing styles. These styled average voices will be used asa background to add expressivity to other neutral voicesusing well–known speaker adaptive training algorithms[11] commonly used in the flexible framework of HMM-based synthesis [14].3. C-ORAL-ROM databaseThe evaluation presented in this paper is carried out us-ing the C–ORAL–ROM [15] database. This corpus is amulti–language and multi–style database covering a widespectrum of formal and informal speaking styles, in pub-lic and private situations.All the languages included are Romance (French,Italian, Portuguese and Spanish), with styles rangingfrom formal to informal, extracted either from the mediaor from private spontaneous natural speaking.In this paper, the Spanish formal media styles havebeen analysed: news broadcasts, sports, meteorologicalreports, reportage, talk–shows, scientific press and inter-views. These data have been extracted from media broad-casts of different stations, and they present a great dealof variability in the recording environments and a highnumber of speakers (124). This results in some speakersuttering only a few short sentences, making them almostirrelevant from a statistical parametrical point of view.Long recordings in this C–ORAL–ROM corpus havebeen splitted into medium–length sessions . The numberof speakers in each session is variable (between 1 and 9speakers). The maximum length for a specific speakerin one session is 5 minutes. Table 1 summarises averageTable 1: Features of the speaking style sessions in theC–ORAL–ROM database (ses. stands for session).Style # ses. SNR #spk/ses. time/ses.interviews 5 25 4 8 minmeteorology 3 26 1 3 minnews 9 29 6 5 minreportage 15 29 7 5 minscientific press 4 27 5 9 minsports 5 33 4 11 mintalk shows 12 29 5 8 mincharacteristics of the considered sessions for each speak-ing style.4. Speaker Diarization SystemUnsupervisedly–generated clusters (or pseudo–speakers)will be used for building styled average voices, using theUPM speaker diarization system described in [3].The system has been adapted to this task in order touse only one feature stream modelling 19 Mel FrequencyCepstrum Coefficients (MFCC). It carries out speakersegmentation and agglomerative clustering of segments,previously filtered by a Voice Activity Detector.4.1. Speaker Diarization ResultsFirst, we have evaluated the performance by measuringthe Diarization Error Rate (DER). Table 2 show that thesystem achieves a very low DER (lower than 5%) for thestyles with speakers that have prepared their discourse, orhave some kind of “prompt” (such as in interviews, me-teorology and news). More spontaneous speaking styles(such as reportage, sports, scientific press and talk shows)obtain a moderate DER (between 10% and 25%).However, the purpose of this task is to unsupervis-edly generate clusters with high precision (or speaker pu-rity) and enough speech to appropriately train a speaking-style average voice or be used as a specific target voice.In this task, miss errors (MISS) are not as relevant as inother tasks (i.e. close captioning) because it will not de-grade the models or the output; high percentage of MISSwill just force to collect and process more data. Simi-larly, False Alarms (FA) are also less relevant than theSpeaker Error Rate (SER), since the speech–text timealignment module will recover from those FA errors andalso because assigning silent frames to a specific cluster(pseudo–speaker) will just provide an improved silencemodel (assuming a good performance of the automatictime–alignment module).Table 2: Speaker diarization results (%) for each speak-ing style.Style Miss FA SpNsp SER DERinterviews 0.00 0.30 0.30 6.60 6.93meteorology 0.00 0.40 0.40 0.70 1.14news 0.00 0.30 0.30 4.40 4.66reportages 0.00 1.40 1.40 22.40 23.78scientific press 0.00 0.30 0.30 15.70 16.01sports 1.30 0.20 1.50 11.80 13.27talk shows 2.30 0.30 2.60 13.20 15.784.2. Unsupervised Pseudo–Speakers for SpeakingStyle Average VoicesIn this task, the longer and the purer (lower number ofactual speakers) a cluster is, the better it would be thatcluster for training a natural identifiable styled voice.This is why the SER commonly used in speaker di-arization tasks [2] is not good enough to evaluate the per-formance of our cluster segmentation system and othermetrics like the recall, precision and F-score for each gen-erated cluster are required to evaluate the available dataand purity of each generated pseudo–speaker.Figure 2 shows the recall as a function of the length ofthe agglomerated clusters. The recall of a cluster has beenestimated as the ratio between the length of the speech as-sociated to the speaker that contributed with more speechframes in that cluster and the the length of the speechin that session for that speaker. We obtain a high recall(higher than 70%) considering that it is an unsupervisedtask over realistic data recorded from the media, not usingany phonetic transcription. The recall clearly decreasesfor clusters smaller than thirty seconds (t <= 30), sug-gesting that thirty is a minimum threshold for this task.Figure 2: Recall (%) as a function of the size of the gen-erated clusters (in seconds) for each speaking styleFigure 3 shows the precision as a function of thesize of the clusters. High precision scores are obtainedFigure 3: Precision (in %) as a function of the size of theclusters (in seconds) for each speaking stylefor all the speaking styles (greater than 85% for mostlengths). Meteorology precision could be considered astrivial since there is only one speaker in each session;however, the diarization system has no prior informationabout the real number of speakers in each session andnevertheless it has been able to guess that there is onlyone speaker using purely–unsupervised techniques. In-terviews and news obtain more robust precision scoresthan the other speaking styles. Background music whencertain speakers are talking in reportage and scientificpress sessions explains the lower precision for the longestclusters (especially in case of scientific press. This factconfirms the need of a speech/music segmentation mod-ule [6] (as proposed in the g framework) which, com-bined with the VAD module, will filter speech data beforediarization.F–measure results shown in Figure 4 combine bothrecall and precision in one performance metric. Whena cluster is longer than thirty seconds, recall is higherthan 75% for every speaking style. This result confirmsthe thirty–seconds threshold for selecting a cluster. Inaddition to this, it has been verified that those clusterswith high scores correspond to professional speakers (in-terviewers, journalists, etc) or speakers that are used tospeak in the media or in public (such as politicians). Onthe contrary, sports is the most spontaneous style andmost of the speakers (excluding the leading journalist) arenot used to talk in the media, making them more difficultto diarize.From these results we have shown that the system per-formance is significantly higher when the speakers haveprepared their discourses than when the spontaneity ishigher.We have carried out an experiment to measure thecorrelation between the performance of the system andthe SNR or the number of speakers in each session: theperformance is weakly affected by these two features(Pearson correlation coefficients lower than 0.1%).Figure 4: F-SCORE (in %) as a function of the lengthof the generated clusters (in seconds) for each speakingstyleIt is expected that the precision (or speaker purity) ofthe unsupervisedly–generated pseudo–speakers would begood enough to build accurate styled average voices witha high similarity with the original speaker and style. Infurther research it will be necessary to perceptually evalu-ate the similarity of the synthetic voices with the originalspeakers, depending of the precision scores of the clus-ters used in the building process of every speaking styleaverage voice.5. ConclusionIn this paper we have defined the architecture of the unsu-pervised multi–speaker voice building framework in TheSimple4All project. We have shown that the high per-formance achieved by our speaker diarization system interms of DER (average of 12%).As DER is not the best way to evaluate the qualityof the pseudo–speakers generated by the system, we haveanalysed quality in terms of recall, precision and F-score.We have shown that speaking styles with speakers usedto talking in public (interviewers, politicians, ect.) favourgenerating good pseudo–speaker clusters with high pre-cision and recall (higher than 90%). Similarly, speakingstyles in which the speakers typically use a certain promptor prepare their discourse, obtain higher F-scores ( 90%vs. 80%) than the pseudo-speakers with more sponta-neous speaking styles (sports, reportage and talk shows).Finally, it will be necessary to carry out a percep-tual evaluation of the accuracy of the speaking style av-erage voices and the similarity of generated expressivesynthetic voices of the target speakers.6. AcknowledgementThe work leading to these results has received fundingfrom the European Union under grant agreement 287678.It has also been supported by TIMPANO(TIN2011-28169-C05-03), INAPRA (MICINN, DPI2010-21247-C02-02), and MA2VICMR (Comunidad Autonoma deMadrid, S2009/TIC-1542) projects. Jaime Lorenzo hasbeen funded by Universidad Politecnica de Madrid underresearch grant SBUPM-QTKTZHB. Authors also thankthe other members of the Speech Technology Group andSimple4All project for the continuous and fruitful discus-sion on these topics.7. References[1] R. Clark and S. King. (2012, March). [Online]. Available:http://simple4all.org[2] X. Anguera, S. Bozonnet, N. W. D. Evans, C. Fredouille, O. Fried-land, and O. Vinyals, “Speaker diarization : A review of recentresearch,” IEEE Transactions On Audio, Speech, and LanguageProcessing, February 2012, Volume 20, NAˆ2, ISSN: 1558-7916,05 2011.[3] J. M. Pardo, R. Barra-Chicote, R. San-Segundo, R. de Cordoba,and B. Martinez-Gonzalez, “Speaker diarization features: Theupm contribution to the rt09 evaluation,” IEEE Trransactions onAudio, Speech and Language Processing, vol. 20, no. 2, pp. 426–435, 2012.[4] C. Chen, K. Filali, and J. Bilmes, “Frontend post-processing andbackend model enhancement on the aurora 2.0/3.0 databases,” inProceedings ICSLP 2002, Sep. 2002, pp. 241–244.[5] S. Meignier and T. Merlin, “Lium spkdiarization: an open sourcetoolkit for diarization,” in CMU SPUD Workshop, Dallas (Texas,USA), mar 2010.[6] A. Gallardo and R. San-Segundo, “Upm-uc3m system for musicand speech segmentation,” in Jornadas de Tecnologa del HablaFALA 2010, November 2010.[7] S. Young, G. Everman, D. Kershaw, G. Moore, J. Odell, D. Ol-lason, V. Valtchev, and P. Woodland, The HTK Book Version 3.1,Dec. 2001.[8] W. Walker, P. Lamere, P. Kwok, B. Raj, R. Singh, E. Gouvea,P. Wolf, and J. Woelfel, “Sphinx-4: A flexible open sourceframework for speech recognition,” Sun Microsystems TechnicalReport, no. TR-2004-139, Nov. 2004. [Online]. Available:http://research.sun.com/techrep/2004/abstract-139.html[9] A. Katsamanis, M. Black, P. Georgiou, L. Goldstein, andS. Narayanan, “Sailalign: Robust long speech-text alignment,” inProceedings of Workshop on New Tools and Methods for Very-Large Scale Phonetics Research, Jan. 2011.[10] K. Prahallad and A. Black, “Segmentation of monologues in au-dio books for building synthetic voices,” IEEE Transactions onSpeech, Audio & Language Processing, vol. 19, no. 5, pp. 1444–1449, 7 2011.[11] J. Yamagishi, “Average-voice-based speech synthesis,” Ph.D. dis-sertation, Tokyo Institute of Technology, 2006.[12] J. Yamagishi, T. Kobayashi, S. Renals, S. King, H. Zen, T. Toda,and K. Tokuda, “Improved average-voice-based speech synthesisusing gender-mixed modeling and a parameter generation algo-rithm considering GV,” in Proceedings of 6th ISCA Workshop onSpeech Synthesis, Aug. 2007.[13] J. Yamagishi, T. Kobayashi, Y. Nakano, K. Ogata, and J. Iso-gai, “Analysis of speaker adaptation algorithms for HMM-basedspeech synthesis and a constrained SMAPLR adaptation algo-rithm,” IEEE Transactions on Speech, Audio & Language Pro-cessing, vol. 17, no. 1, pp. 66–83, 1 2009.[14] H. Zen, T. Nose, J. Yamagishi, S. Sako, T. Masuko, A. Black, andK. Tokuda, “The HMM-based speech synthesis system (HTS) ver-sion 2.0,” in Proceedings 6th ISCA Workshop on Speech Synthesis(SSW-6), Aug. 2007.[15] E. Cresti, F. B. do Nascimento, A. M. Sandoval, J. Veronis, P. Mar-tin, and K. Choukri, “The c-oral-rom corpus a multilingual re-source of spontaneous speech for romance languages,” in Pro-ceedings of LREC, 2004.

Analysis of speaker adaptation algorithms for HMM-based speech synthesis and a constrained SMAPLR adaptation algorithm,”

and K.Tokuda,

Average-voice-based speech synthesis,”

Frontend post-processing and backend model enhancement on the aurora 2.0/3.0 databases,”

Improved average-voice-based speech synthesis using gender-mixed modeling and a parameter generation algorithm considering GV,”

Lium spkdiarization: an open source toolkit for diarization,”

Sailalign: Robust long speech-text alignment,”

Segmentation of monologues in audio books for building synthetic voices,”

Speaker diarization : A review of recent research,”

Speaker diarization features: The upm contribution to the rt09 evaluation,”

Sphinx-4: A ﬂexible open source framework for speech recognition,” Sun Microsystems

The c-oral-rom corpus a multilingual resource of spontaneous speech for romance languages,” in

Upm-uc3m system for music and speech segmentation,”

Towards an Unsupervised Speaking Style Voice Building Framework:Multi-Style Speaker Diarization

Towards an Unsupervised Speaking Style Voice Building Framework:Multi-Style Speaker Diarization

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer