Corkey, Niamh

King, Simon

O'Mahony, Johannah

English

We present a novel, user-friendly approach for controlling patterns of intonation (a fundamental aspect of prosody) within a neural TTS system. This involves concisely representing F0 contours with the coefficients of their Legendre polynomial series expansion, and implementing a model (based on FastPitch) which is conditioned on these sets of coefficients during training. At inference time the model will explicitly predict a coefficient set, or a user (eg. human-in-the-loop) can provide a target coefficient set which audibly alters the intonation of the output speech, based on just a few values. This is particularly effective for intonation transfer: where these coefficient targets are extracted from a ground truth recording, making the synthesised utterance more closely reflect the intonation of the real speaker

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Intonation control for neural text-to-speech synthesis withpolynomial models of F0Citation for published version:Corkey, N, O'Mahony, J & King, S 2023, Intonation control for neural text-to-speech synthesis withpolynomial models of F0. in N Harte, J Carson-Berndsen & G Jones (eds), Proceedings of the AnnualConference of the International Speech Communication Association: Interspeech 2023. Interspeech -Annual Conference of the International Speech Communication Association, ISCA, Dublin, pp. 2014-2015,Interspeech 2023, Dublin, Ireland, 20/08/23. <https://www.isca-speech.org/archive/interspeech_2023/corkey23_interspeech.html>Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Peer reviewed versionPublished In:Proceedings of the Annual Conference of the International Speech Communication AssociationGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 13. Nov. 2023Intonation Control for Neural Text-to-Speech Synthesiswith Polynomial Models of F0Niamh Corkey, Johannah O’Mahony, Simon KingThe Centre for Speech Technology Research, University of Edinburgh, UKs1936986@ed.ac.uk, johannah.o’mahony@ed.ac.uk, Simon.King@ed.ac.ukAbstractWe present a novel, user-friendly approach for controlling pat-terns of intonation (a fundamental aspect of prosody) within aneural TTS system. This involves concisely representing F0contours with the coefficients of their Legendre polynomial se-ries expansion, and implementing a model (based on FastPitch)which is conditioned on these sets of coefficients during train-ing. At inference time the model will explicitly predict a coeffi-cient set, or a user (eg. human-in-the-loop) can provide a targetcoefficient set which audibly alters the intonation of the outputspeech, based on just a few values. This is particularly effectivefor intonation transfer: where these coefficient targets are ex-tracted from a ground truth recording, making the synthesisedutterance more closely reflect the intonation of the real speaker.Index Terms: text-to-speech, speech synthesis, intonationmodelling, prosody control, prosody transfer1. IntroductionThe rise of end-to-end neural text-to-speech (TTS) models hasenabled synthetic voices to sound remarkably natural and intel-ligible. However, most state-of-the-art TTS systems are unableto produce different prosodic renditions of a given text, thusfailing to capture a key feature of natural human speech.Approaches to combat this include prosody control/transfer,where target prosodic patterns can be manually specified or ex-tracted from reference recordings. However, most of these tech-niques involve learning rich latent representations of desiredprosodic features [1, 2], which is resource-intensive and canlead to source-speaker leakage.Here, we will focus on control/transfer of one major ele-ment of prosody: intonation. We have modeled patterns of into-nation simply: using the first three coefficients of the Legendrepolynomial series expansion of the F0 contour for each utter-ance. These coefficients are interpretable, representing the av-erage level, slope, and convexity of F0 respectively. We haveadapted the FastPitch model to be conditioned on a coefficientset for each training utterance, and explicitly predict coefficientsduring inference. In our demo, a user can supply their owntarget coefficient set instead (representing desired F0 contourshape), to influence the intonation in the final synthetic speech.2. Implementation2.1. The DataWe trained our model on LJ Speech [3], a public domain speechdataset with a single American female speaker. The originaldataset contained 13100 utterances with an average length of6.57 s, however we found that 3-coefficient representations ofthese long utterances’ F0 contours tend to be rather flat, andInput TextFFT StackEmbeddingDuration PredictorPitch PredictorUpsamplingFFT StackFC Layer Synthesised Mel-SpectrogramConv LayerConv LayerFC LayerConv LayerConv LayerFC LayerConv Layer (Embedding)MSE LossMSE LossMSE LossCoefficient PredictorConv LayerConv LayerFC LayerConv Layer (Embedding)Bidirectional LSTMMSE Loss123Figure 1: Architecture of the new implementation of the Fast-Pitch model, with ’Coefficient Predictor’ added, where ‘Conv’= 1D Convolutional, and ‘FC’ = Fully-Connected.thus would not expose the model to much variety in patterns ofintonation. Therefore, we rechunked this into shorter utterancesbased on occurences of commas and full stops in the transcrip-tions, to create shorter prosodic-phrase-like utterances. This re-sulted in 18418 utterances with an average length of 3.83 s.We extracted F0 contours from all utterances, before takingthe average value per phone and normalising values accordingto the speaker’s mean/standard deviation of F0 (as this is con-sistent with FastPitch’s representation of F0 contours). We thenlinearly interpolated the corrected contour and used the functionpolynomial.legendre.legfit from the numpy pack-age to return the three coefficients of the Legendre series (of de-gree 2) which fits each F0 contour with least squared error. Fi-nally, we range normalised each of the three coefficients acrossthe dataset between -1 and 1.2.2. The ModelWe adapted FastPitch: a neural text-to-speech model whichrapidly synthesises mel-spectrograms based on input text [4],and already includes explicit predictors for F0, duration andenergy, allowing control over these features. The open-sourcePython code used as a basis for this implementation came froma fork 1 of NVIDIA’s ‘DeepLearningExamples’ repository [5]on GitHub.We added an explicit Coefficient Predictor to the model’sarchitecture (as shown in Figure 1) which includes a bidirec-tional LSTM layer, where the prediction at the final timestep istaken as the utterance-level coefficient set. The coefficient pre-diction is made before the F0 predictions, with the hope that F01https://github.com/evdv/FastPitches0 10 20 30 40 50 60Frame50100150200250300350400450F0ground truthno target coefficientswith target coefficientsFigure 2: F0 contours (interpolated) of ground truth record-ing vs synthesised versions either with ground truth coefficientsprovided as targets, or with no targets provided, for the text ‘thelittle stone seal’.predictions can also be conditioned on the shape of intonationindicated by the coefficients. These predictions have their meansquared error loss recorded during training.During inference, if no coefficient targets are specified,this prediction is upsampled to the maximum input sequencelength in the batch, reshaped for consistency with other fea-ture predictions, and masked by the encoder mask to reflecteach different sequence length in the batch, before being em-bedded and summed to the current encoder output. Duringtraining, and during inference if targets are supplied, it is theground truth/target coefficients which are upsampled, reshaped,masked, embedded and summed to the encoder output.The final model was trained for 1000 epochs, with a learn-ing rate of 0.1 and a batch size of 16, on a single GPU.3. Intonation Transfer Results3.1. Objective MeasuresWe synthesised two versions of each text in the test set(which included 736 utterances): one with coefficients fromthe ground truth recording supplied as targets, and one withno targets supplied. We found that average root-mean-squareerror between ground truth and synthesised contours was re-duced when ground truth coefficient targets were used for syn-thesis (47.370 Hz) compared to when no targets were used(55.143 Hz). This suggests that using this method can increasethe similarity in intonation between synthesised speech and hu-man reference recordings, as exemplified in Figure 2.3.2. Formal Listening TestA listening test was conducted with 26 paid participants, using50 reference recordings from the test set. Listeners were playedthe reference recording, followed by two synthesised versionsof the same text (with-targets and no-targets, as above), and hadto select which synthesised version sounded more similar to thereference in terms of pitch/intonation. The with-targets versionwas chosen 63.7% of the time, which was found to be a signif-icant preference according to a binomial mixed effects modelanalysis (β=0.62 (0.65 prob), CI=(0.60,0.70), p < 0.001). Thismeans that this technique can audibly increase the similarityin intonation between synthesised speech and human referencerecordings, proving some degree of intonation transfer ability,at least from recordings of the same speaker/text.3.3. Informal ExtensionWe then tested the model’s ability to transfer patterns of intona-tion from different speakers and different text. A new speakerwas recorded producing the same text (‘she said that she’s yoursister’) as both a declarative statement (with an overall decli-nation of F0) and a declarative question (with a characteristicrise in F0 at the end). Three coefficients were then extractedfrom each of the contours, creating a ‘statement’ target set, anda ‘question’ target set. The same text was then synthesised witheach target set, and an informal listening test with 12 partici-pants found that listeners were always able to correctly distin-guish which version was intended to be a statement, and whicha question. The same targets were then used to synthesise twoversions of a different text (‘it’s raining in Glasgow’), and lis-teners were still always able to identify the statement/questioncorrectly.These results preliminarily suggest that this approach canbe used to transfer patterns of intonation between differentspeakers/texts, and that the effect is obvious enough to alter thethe meaning of a synthesised utterance.4. DicussionWe have shown that this approach offers a simple and efficientway to transfer patterns of intonation from reference recordingsto synthesised speech. We hope to provide an opportunity forusers to extract target coefficient sets from their own referencerecordings or even hand-drawn F0 contours to allow intuitivehuman-in-the-loop control over synthesised intonation.In future we hope to extend this method, for example by us-ing the coefficients to actually reconstruct an F0 contour withinthe model, which should allow FastPitch’s current Pitch Pre-dictor to be fully replaced; allowing more complete control viacoefficients. Anticipated future applications include using thistechnique to model intonation hierarchically (ie. at the word,phrase, and sentence level simultaneously), to enable more pre-cise intonation control, and clustering coefficient sets to identifyand recreate general patterns of intonation which correspond toparticular speech acts/emotions.5. AcknowledgementsThis project has received funding from the European Union’sHorizon 2020 research and innovation programme under theMarie Skłodowska-Curie grant agreement No 859588.6. References[1] Y.-J. Zhang, S. Pan, L. He, and Z.-H. Ling, “Learning latent rep-resentations for style control and transfer in end-to-end speechsynthesis,” in 2019 IEEE International Conference on Acoustics,Speech and Signal Processing. IEEE, 2019, pp. 6945–6949.[2] Y. Wang, D. Stanton, Y. Zhang, R.-S. Ryan, E. Battenberg, J. Shor,Y. Xiao, Y. Jia, F. Ren, and R. A. Saurous, “Style tokens: Unsu-pervised style modeling, control and transfer in end-to-end speechsynthesis,” in 2018 International Conference on Machine Learn-ing. PMLR, 2018, pp. 5180–5189.[3] K. Ito and L. Johnson, “The LJ Speech dataset,” https://keithito.com/LJ-Speech-Dataset/, 2017.[4] A. Łańcucki, “Fastpitch: Parallel text-to-speech with pitch predic-tion,” in 2021 IEEE International Conference on Acoustics, Speechand Signal Processing. IEEE, 2021, pp. 6588–6592.[5] K. Kudrynski, “Deep learning examples: FastPitch,”https://github.com/NVIDIA/DeepLearningExamples/tree/master/\PyTorch/SpeechSynthesis/FastPitch, 2020.

Intonation control for neural text-to-speech synthesis with polynomial models of F0

https://www.research.ed.ac.uk/files/381575311/corkey23_interspeech.pdf

Intonation control for neural text-to-speech synthesis with polynomial models of F0

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer