This paper presents some preliminary methods to apply the Time- Frequency Interpolation technique - TFI [3] to concatenative text-to-speech synthesis. The TFI technique described here is a pitch-synchronous time-frequency approach of the well known Prototype-Waveform Interpolation technique - PWI [2]. The basic concepts of representing the speech signal in the Time-Frequency Domain as well as techniques to perform Time-Scale and Pitch- Scale modifications are described. Using the flexibility of TFI technique to perform spectral smothing, a method was developed to minimize the spectral mismatch at the boundaries of the Synthesis-Units - SUs. The proposed system was evaluated using SUs (Diphones) and prosodic modifications generated by the Festival system [1]. An informal subjective test was performed, between the proposed TFI system and the standard TD-PSOLA system, highligthing the superior quality of the proposed system in comparasion with TD-PSOLA

Morais, Edmilson

Taylor, Paul

Violaro, Fabio

Edinburgh Research Archive

CONCATENATIVE TEXT-TO-SPEECH SYNTHESIS BASED ONPROTOTYPE WAVEFORM INTERPOLATION(A TIME FREQUENCY APPROACH)Edmilson S. Morais Paul TaylorCSTR / University of Edinburgh2 Buccleuch Place Edinburgh EH8 9LW, UKemorais@cstr.ed.ac.ukFábio ViolaroDepartment of Eletrical EngineeringUniversidade Estadual de CampinasCampinas, SP, BrazilABSTRACTThis paper presents some preliminary methods to apply the Time-Frequency Interpolation technique - TFI [3] to concatenativetext-to-speech synthesis. The TFI technique described here is apitch-synchronous time-frequency approach of the well knownPrototype-Waveform Interpolation technique - PWI [2]. The basicconcepts of representing the speech signal in the Time-FrequencyDomain as well as techniques to perform Time-Scale and Pitch-Scale modifications are described. Using the flexibility of TFItechnique to perform spectral smothing, a method was devel-oped to minimize the spectral mismatch at the boundaries of theSynthesis-Units - SUs. The proposed system was evaluated us-ing SUs (Diphones) and prosodic modifications generated by theFestival system [1]. An informal subjective test was performed,between the proposed TFI system and the standard TD-PSOLAsystem, highligthing the superior quality of the proposed systemin comparasion with TD-PSOLA.1. INTRODUCTIONSome of the basic operations in a Concatenative Text-To-Speech -TTS - synthesis system are the concatenation and prosodic modi-fications of the Synthesis-Units - SUs. However, in order to guar-antee good concatenations and prosodic modifications the systemhas to be able to perform the following three operations : (1)Spectral smoothing at SUs boundaries. (2) Time Scale Modi-fications -TSM, independently from Pitch Scales Modification -PSM, and vice-versa. (3) Continuous amplitude modifications ofSUs through time.The TD-PSOLA is the technique most used in commercialConcatenative TTS synthesis systems. However TD-PSOLApresents some drawbacks, mainly under large prosodic variations: (1) PSM introduces simultaneous TSM which needs to be ap-propriately compensated. (2) TSM can only be implementedin a quantized manner, with a resolution of one pitch period(...,1/2,3/4,...,4/3,3/2,2,...). (3) Duration lengthening for unvoicedsegments introduces a periodic component that is reponsable for ametalic-like sounding of the synthesized speech. (4) The Overlapand Add procedure at the boundaries of the SUs does not guaran-tee a good spectral smoothing.In order to overcome some drawbacks of the TD-PSOLA, this pa-per presents a method based on Time Frequency Interpolation -TFI [3]. The TFI method introduced here is a Pitch-SynchronousTime-Frequency approach of the well known Prototype Wave-form Interpolation technique - PWI [2]. The goal of this paperis to show that the TFI technique presents some important advan-tages to concatenative TTS synthesis. It allows PSM indepen-dently of TSM, in a quite straightforward manner and with high-quality. TSM and PSM can be done in a continuous way, withoutany limitation of pitch period resolution. Moreover, the TFI tech-nique allows simple, flexible and efficient procedures to smoothdiphone (or any other kind of unit) boundaries.Section 2 presents the basic concepts of signal representationon Time-Frequency domain. Section 3 describes the Analysisand Re-Synthesis TFI procedures. The prosodic modificationsbased on TFI synthesis are discussed in Section 4. Section 5 de-scribes one possible technique to perform Spectral Smothing atthe boundaries of the SUs. Finally, Sections 6 and 7 present theresults of an informal subjective test and some final considera-tions.2. TFI FRAMEWORKIn this paper, the representation of a discrete signal  		in the time-frequency domain isbased on the concept of short-time  "!#%$ &' discrete spec-trum sequence [3]. Each time ( on a discrete-time axis isassociated with an )  ( * ,+.-	(/ discrete spectrum 021 3 1 	43 1 3 1  )  ( 56 . This representation de-fines a sequence of spectra 7   028  0:9 ; 02<>=9  . The(  /? spectrum is calculated projecting the sub-sequence  1@BADC14EF=91   (  (HG  (IGJ)  ( KJ overthe base functions of a specific operator L NM 0 1  L    14@OA*C1EF=O91  (1)In this paper, the spectrum 021 is calculated using the DiscreteFourier Transform ( PRQ>L ) operator :021  14@OA*C1E	=9STVU 1  W$XMYZ\[ =%]^ _C1EF^ `.^ C T =,1E(2)where a  (  cb Md)  (  (3)ProsodicRe-SynthesisModifications^AnalysisFeU/UVnEE onOriginal waveformLSFsV/UVoLSFseSynthsized waveformFPitch MarksStorage EnviromentSynthesisTFI ResidueSamplingPrototypeInverse FilterA(z)LP Analysis APitch MarksPitch Marks1A(z)A(z)Synthesis FilterFigure 1: Diagram of the proposed system : Analysis and Re-Synthesis.We will denote by L =O9 NM  the inverse operator that enables totransform the sequence of spectra 7 into the original signal   .3. ANALYSIS AND RE-SYNTHESISThis section will described the procedures of Synchronous Anal-ysis and Re-Synthesis using TFI. Figure 1 presents the block dia-gram of these two stages.3.1. AnalysisPitch-Marks The first step in the speech analysis is to provide asequence of pitch-marks and a Voiced/Unvoiced classification foreach segment between two consecutives pitch-marks. In this workthis task was solved in a very precise way using the correspond-ing laryngograph and differencial laryngograph signals. Figure2 shows the pitch-marks in a voiced segment. The distance be-tween (  and ( ] pitch-marks will be called    ] . In the originalsignal   ] is equal to the fundamental period  . However, afterprosodic modifications   ] may be different from  as will beseen in Section 4. The fundamental frequency associated to the(   pitch-mark is then    	   .Linear Prediction Model and Inverse Filter The proposed sys-tem performs a Linear Prediction -  - analysis around eachpitch-mark using an asymmetrical Hamming window starting atthe previous pitch-mark and finishing at the next one, as shownin Figure 2. The LP coefficients are transformed into Line Spec-tral Frequency Coefficients - LSF and for each two consecutivespitch-marks, (   and (\] , the corresponding pair of LSFs are up-sampled to  M   by linear interpolation (one LSF for each    ]  samples; this set of 8 LSF will be denoted by  1  ). Usingthese interpolated LSFs (  1  ) new LP coefficients are derivednhn g n i n j n k n l nm1Fo= o iTiFigure 2: Indication of the pitch-marks and the postions of the LP win-dow analysisEnf Eng Enh Eni Enj Enk Enl Enmnneninjnn nnnn i j(c)n f g h k l m n(a)(b)Figure 3: (a) - Speech signal. (b) Extration of the prediction error sub-sequences  1  @  =91  . (c) - Prototypes  1and used to filter the speech signal   , generating the Linear Predi-cion Error (residue)  .Sampling of Prototypes As shown in Figure 3, at each pitch-mark (   a sub-sequence of the linear prediction error must betaken  1  @   =O91      (      (   G ;   (   G   " andits spectrum  1   L   11   must be calculated. This proceduredefines a sequence of spectra   N  1    1    1    , where 1   1 @  =9STVU 1  W$ MY;Z [ =N]^ _C1  EF^ `.^ C T =,1  E(4)with a  (     b Md  (5)Defining    8   9   <   as the Time-Frequencyrepresentation of the whole linear prediction error    	4    	  , thus the sequence of spectra    1!   1    1   can be understood as a Pitch-Synchronous time-domain decimation of the sequence ofspectra(where only the spectra at the pitch-mark posi-tions were mantained). Henceforth, each spectrum  1#"N  1    1    1 ;  will be called $&%	')(*')(+,$- . Briefly, forthe (  /? pitch-mark, the following parameters must be stored fora ,+ ! / -	+. - decoding : (1) Prototype -  1 , (2) Voiced and Un-voiced decision - . / . , (3) Fundamental frequency -    9and (4) LSF coefficients - 10OQ .3.2. Re-SynthesisThe L Q  synthesis assumes a slow evolution of the spectra andreconstructs the full time-frequency representationby calcu-lating a linear interpolation of the sequence of prototypes  N  1!   1    1  ;  . However, in order to compute the inter-polation of prototypes between  1  and  1 , it is necessary thatthese two spectra have the same number of samples. This prob-lem can be solved appending zeroes at the end of the prototypewith fewer samples. This increasing of zeroes is associated with acompression in frequency domain and consequently an expansion(interpolation) in time domain. By appending zeroes, the dura-tion of two consecutive prototypes  1  and K1 becames equalto )   ]  ) #%B        .After adjusting the duration of the prototypes, each spectrum 1 "  1 1     1   K1  @9  K1=9  , can be obtained by lin-ear interpolation of the prototypes  1  and K1 . 1  1    1   1    1     ( M  1  G  (  M  1  (6)In order for Equation 6 to be considered as a linear interpolation,the real and imaginary parts of  1  and K1 are interpolated sep-arately. However, in a linear interpolation   (  and   (  do notneed to be linear functions. In this work linear interpolation func-tions were used :  (   >  (  (,     ] # (	  (   V   (  (7)With the spectraK1 calculated using linear interpolation, an esti-mation of the vector  1 1      (      (   G ;   (\] H canbe obtained using the following inverse transformation :  (    C 8E'@  C   EF^ C'=9 E A   G C   =O9 E` U 9 ` )   ] even  C 8EA   G C   E` U 9  ` )   ] odd(8)where `  b)   ] M  1 ! MY;Z [ ]^ "\C14EF^ `  (9)In this case #  (  is defined as :#  (  %$    (  M b Md  G&  ( M b Md   G('1 *) M ( (10)with   (  and   (  given by Equation 7 and the phase function' given by :'    ',+KG b Md2M   ]   (11)where ',+ is the phase associated to (-+ pitch-mark (previouspitch-mark).Equation 10 assures a linear transition, sample-by-sample, fromthe fundamental frequency of prototype  1  to the fundamen-tal frequency of prototype  1 . The phase function expressed inEquation 11 is necessary to assure the continuity of the speechsignal.Because the LSF coefficients were upsampled during the analysisprocedure, then they also have to be upsampled before filteringthe residue. Moreover, in order to be consistent with the interpo-lation of the prototypes  1  and  1 , the corresponding sets ofLine-Spectral Frequencies  1  and  1 (8 for each pitch-mark) have also to be linearly interpolated using the same linearfunctions   (  and   (  . The 8 new interpolated LSFs for eachframe are transformed into LP coefficients and then used to syn-thesize the speech signal.4. PROSODIC MODIFICATIONS4.1. TSM - Time Scale ModificationsTime-Scale Modifications (TSM) can be performed simply bychanging the original /- $   ,+ ! -W/-	+ ( of the prototypes. Nomodification has to be done through Frequency-Domain. For ex-ample, the original distance between the prototypes  1  and  1was defined as   ]    . However, after a TSM with a factor . ,the new distance   ] will be equal to . M   . Therefore, in orderto preserve the continuity of the phase function, the new value of   ] has to be used in the Equation 11.This procedure can be performed for both voiced and unvoicedsegments. However, a TSM bigger than 2 can produce somemetalic-like sounding in the unvoiced segments. A good solu-tion to minimize this problem is the REW (Rapidly-EvolvingWaveform) and SEW (Slowly-Evolving Waveform) decomposi-tion proposed in [2]. Using this approach the REW componentcan be upsampled through time-domain before interpolation. Thisnew REW presents a slower evolution through time and can bebetter interpolated using the TFI Technique.4.2. PSM - Pitch Scale ModificationsDuring Pitch-Scale Modifications (PSM) the original /- $  ,+ ! -	/ -	+ ( ! of the prototypes must be preserved. However, theprototypes must be changed by resampling their envelopes at thepositions of the new Pitch-Frequencies. For example, the originalPitch-Frequency of the prototype  1  is    9  . However,after a PSM with a factor of 9/ , the new Pitch-Frequency will beequal to 0  /  9/ ^   . Therefore, the new prototype must havesamples taking place at multiples of this new fundamental fre-quency. Moreover, after a PSM with a factor of 9/ , the Equation11 must be changed to :'1   ' + G b Md2M   ]1 M   (12)The PSM can only be applied for voiced segments. In thiswork the resampling of the prototype envelopes has been done bymeans of a simple linear interpolation of the real and imaginarycomponents of the spectra.0 10 20 30 40 50 60 7002468−500500 10 20 30 40 50 60 7002468−50050(a)(b)Figure 4: Frequency-response of the synthesis filter at the Boundary oftwo SUs. (a) Before smothing. (b) After smothing (   	 ).5. SPECTRAL SMOOTHINGUsing the flexibility of the TFI technique to perform spectralsmothing, several methods can be developed to minimize thespectral mismatch at the boundaries of the SUs. The method de-scribed here is applied exactly in the same way to both prototypesand LSF coefficients. Defining  as the  last prototypes in theprevious SU and  as the  first prototypes in the current SU,we will describe here the proposed procedure for a particular casewhere      b .Defining     1   1  as the last two prototypes of theprevious SU and     1    1   as the first two prototypesof the current SU, then at first the prototypes in  and   arenormalized to have exactly the same size (putting zeroes at theend of the smaller ones) and after that they are submitted to thefollowing linear combinations :  1     8 M K1  G(8 M K1   , 1    9 M K1IG  9 M K1   ,  1     M  1  G   M  1 and  1     M  1!VG  M  1   .The goal of this linear combination is to perform a sharing of in-formation, across the boundary. The coefficients    have beendefined as a normalized Blackman window with maximum am-plitude equal to 0.5, and    =      . This linear combination hasbeen performed only for 5+ -;	 segments.Now using these combined prototypes, a double interpolationthrough time-domain is performed. The first one is a Spline in-terpolation at the following time-instants : ( , 1'@ 1 , 1 @B1  ,1  @ 1  and (  . The second one is a linear interpolation that re-turns the Splined prototypes to the original positions. The aim ofthese interpolations is to minimize spectral discontinuities closeto the boundary.The Figure (4.a) shows the Frequency-response of the synthe-sis filter at the boundary of two SUs. Figure (4.b) shows theFrequency-response of the same synthesis filter after applying theproposed procedure of smoothing, where     .6. EVALUATIONAn informal subjective test was performed between the proposedsystem and the standard TD-PSOLA system. The sequenceof SUs (Diphones) and the Prosodic-Modifications were gener-ated using the Festival System [1]. The test was composed of14 sentences. These sentences were listened to by 12 speech-experts who had to express their preference. The results high-lighted a superior quality of the proposed system in compara-sion with the standard TD-PSOLA. 71.4% of the listeners saidthat the TFI was better than the TD-PSOLA and only 5.71% saidthat the TD-PSOLA was better than the TFI (22.89% of the lis-teners did not express a preference). These sentences can beheard on the following HomePage : http://www.cstr.ed.ac.uk/ emorais/pwitfi.html.7. FINAL CONSIDERATIONSIn this paper some preliminary methods to apply Time-FrequencyInterplolation [3] to Concatenative TTS Systems were presented.It was shown that the TFI technique allows PSM and TSM aswell as Spectral Smothing at the boundaries of the SUs in a quitestraightforward manner and with high quality.Comparing TD-PSOLA and TFI regarding computational cost, itis clear that TFI has much higher complexity than TD-PSOLA.However, TFI presents the advantage of allowing very easy andefficient low-bit rate coding. Moreover, expecting the computerpower to increase in the future, TFI complextiy will not be at alla problem.In order to improve the quality of TFI techinque (in terms ofProsodic Modifications and the Spectral Smothing at the bound-aries of the SUs) the authors suggest investigations on the REWand SEW decomposition [2] as well as on the adaptation of thetechnique to operate directly with the speech signal, instead ofusing the Linear Prediction model.8. REFERENCES1. A. Black, P. Taylor, R. Caley. The Festival Speech Synthesis.Avaliable at htt://www.cstr.ed.ac.uk/projects/festival.html, 4(5), Sept.1996.2. B. Kleijn, K. Paliwal, eds. Speech Coding and Synthesis. Elsevier,Amsterdam, 1998.3. Y. Shoham. High-quality Speech Coding at 2.4 to 4.0 kbps Basedon Time-Frequency Intepolation. IEEE Proc. ICASSP ‘93, II.167–II.170, April,1993.4. F. Violaro, O. Boeffard. A Hybrid Model for Text-to-Speech Synthe-sis. IEEE Trans. on Speech and Audio Processing, 6(5), Sept. 1998.5. E.S. Morais, F. Violaro, P. Barbosa. Prosodic Speech ModificationUsing Pitch-Synchronous Time-Frequency Interpolation. Proc. of theInternational Telecommunication Simposium, Sao Paulo, SP, Brazil,June, 1998.

Concatenative Text-to-Speech Synthesis Based on Prototype Waveform Interpolation (A Time Frequency Approach)

https://era.ed.ac.uk/bitstream/1842/983/1/Morais_2000_a.pdf

Concatenative Text-to-Speech Synthesis Based on Prototype Waveform Interpolation (A Time Frequency Approach)

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Archive