40 research outputs found
Context-aware Coherent Speaking Style Prediction with Hierarchical Transformers for Audiobook Speech Synthesis
Recent advances in text-to-speech have significantly improved the
expressiveness of synthesized speech. However, it is still challenging to
generate speech with contextually appropriate and coherent speaking style for
multi-sentence text in audiobooks. In this paper, we propose a context-aware
coherent speaking style prediction method for audiobook speech synthesis. To
predict the style embedding of the current utterance, a hierarchical
transformer-based context-aware style predictor with a mixture attention mask
is designed, considering both text-side context information and speech-side
style information of previous speeches. Based on this, we can generate
long-form speech with coherent style and prosody sentence by sentence.
Objective and subjective evaluations on a Mandarin audiobook dataset
demonstrate that our proposed model can generate speech with more expressive
and coherent speaking style than baselines, for both single-sentence and
multi-sentence test.Comment: Accepted by ICASSP 202
Towards Spontaneous Style Modeling with Semi-supervised Pre-training for Conversational Text-to-Speech Synthesis
The spontaneous behavior that often occurs in conversations makes speech more
human-like compared to reading-style. However, synthesizing spontaneous-style
speech is challenging due to the lack of high-quality spontaneous datasets and
the high cost of labeling spontaneous behavior. In this paper, we propose a
semi-supervised pre-training method to increase the amount of spontaneous-style
speech and spontaneous behavioral labels. In the process of semi-supervised
learning, both text and speech information are considered for detecting
spontaneous behaviors labels in speech. Moreover, a linguistic-aware encoder is
used to model the relationship between each sentence in the conversation.
Experimental results indicate that our proposed method achieves superior
expressive speech synthesis performance with the ability to model spontaneous
behavior in spontaneous-style speech and predict reasonable spontaneous
behavior from text.Comment: Accepted by INTERSPEECH 202
Improving Mandarin Prosodic Structure Prediction with Multi-level Contextual Information
For text-to-speech (TTS) synthesis, prosodic structure prediction (PSP) plays
an important role in producing natural and intelligible speech. Although
inter-utterance linguistic information can influence the speech interpretation
of the target utterance, previous works on PSP mainly focus on utilizing
intrautterance linguistic information of the current utterance only. This work
proposes to use inter-utterance linguistic information to improve the
performance of PSP. Multi-level contextual information, which includes both
inter-utterance and intrautterance linguistic information, is extracted by a
hierarchical encoder from character level, utterance level and discourse level
of the input text. Then a multi-task learning (MTL) decoder predicts prosodic
boundaries from multi-level contextual information. Objective evaluation
results on two datasets show that our method achieves better F1 scores in
predicting prosodic word (PW), prosodic phrase (PPH) and intonational phrase
(IPH). It demonstrates the effectiveness of using multi-level contextual
information for PSP. Subjective preference tests also indicate the naturalness
of synthesized speeches are improved.Comment: Accepted by Interspeech202
CB-Conformer: Contextual biasing Conformer for biased word recognition
Due to the mismatch between the source and target domains, how to better
utilize the biased word information to improve the performance of the automatic
speech recognition model in the target domain becomes a hot research topic.
Previous approaches either decode with a fixed external language model or
introduce a sizeable biasing module, which leads to poor adaptability and slow
inference. In this work, we propose CB-Conformer to improve biased word
recognition by introducing the Contextual Biasing Module and the Self-Adaptive
Language Model to vanilla Conformer. The Contextual Biasing Module combines
audio fragments and contextual information, with only 0.2% model parameters of
the original Conformer. The Self-Adaptive Language Model modifies the internal
weights of biased words based on their recall and precision, resulting in a
greater focus on biased words and more successful integration with the
automatic speech recognition model than the standard fixed language model. In
addition, we construct and release an open-source Mandarin biased-word dataset
based on WenetSpeech. Experiments indicate that our proposed method brings a
15.34% character error rate reduction, a 14.13% biased word recall increase,
and a 6.80% biased word F1-score increase compared with the base Conformer
Towards Improving the Expressiveness of Singing Voice Synthesis with BERT Derived Semantic Information
This paper presents an end-to-end high-quality singing voice synthesis (SVS)
system that uses bidirectional encoder representation from Transformers (BERT)
derived semantic embeddings to improve the expressiveness of the synthesized
singing voice. Based on the main architecture of recently proposed VISinger, we
put forward several specific designs for expressive singing voice synthesis.
First, different from the previous SVS models, we use text representation of
lyrics extracted from pre-trained BERT as additional input to the model. The
representation contains information about semantics of the lyrics, which could
help SVS system produce more expressive and natural voice. Second, we further
introduce an energy predictor to stabilize the synthesized voice and model the
wider range of energy variations that also contribute to the expressiveness of
singing voice. Last but not the least, to attenuate the off-key issues, the
pitch predictor is re-designed to predict the real to note pitch ratio. Both
objective and subjective experimental results indicate that the proposed SVS
system can produce singing voice with higher-quality outperforming VISinger
Improving Language Model-Based Zero-Shot Text-to-Speech Synthesis with Multi-Scale Acoustic Prompts
Zero-shot text-to-speech (TTS) synthesis aims to clone any unseen speaker's
voice without adaptation parameters. By quantizing speech waveform into
discrete acoustic tokens and modeling these tokens with the language model,
recent language model-based TTS models show zero-shot speaker adaptation
capabilities with only a 3-second acoustic prompt of an unseen speaker.
However, they are limited by the length of the acoustic prompt, which makes it
difficult to clone personal speaking style. In this paper, we propose a novel
zero-shot TTS model with the multi-scale acoustic prompts based on a neural
codec language model VALL-E. A speaker-aware text encoder is proposed to learn
the personal speaking style at the phoneme-level from the style prompt
consisting of multiple sentences. Following that, a VALL-E based acoustic
decoder is utilized to model the timbre from the timbre prompt at the
frame-level and generate speech. The experimental results show that our
proposed method outperforms baselines in terms of naturalness and speaker
similarity, and can achieve better performance by scaling out to a longer style
prompt.Comment: Submitted to ICASSP 202