235 research outputs found
Neural Discrete Representation Learning
Learning useful representations without supervision remains a key challenge
in machine learning. In this paper, we propose a simple yet powerful generative
model that learns such discrete representations. Our model, the Vector
Quantised-Variational AutoEncoder (VQ-VAE), differs from VAEs in two key ways:
the encoder network outputs discrete, rather than continuous, codes; and the
prior is learnt rather than static. In order to learn a discrete latent
representation, we incorporate ideas from vector quantisation (VQ). Using the
VQ method allows the model to circumvent issues of "posterior collapse" --
where the latents are ignored when they are paired with a powerful
autoregressive decoder -- typically observed in the VAE framework. Pairing
these representations with an autoregressive prior, the model can generate high
quality images, videos, and speech as well as doing high quality speaker
conversion and unsupervised learning of phonemes, providing further evidence of
the utility of the learnt representations
ControlVC: Zero-Shot Voice Conversion with Time-Varying Controls on Pitch and Speed
Recent developments in neural speech synthesis and vocoding have sparked a
renewed interest in voice conversion (VC). Beyond timbre transfer, achieving
controllability on para-linguistic parameters such as pitch and Speed is
critical in deploying VC systems in many application scenarios. Existing
studies, however, either only provide utterance-level global control or lack
interpretability on the controls. In this paper, we propose ControlVC, the
first neural voice conversion system that achieves time-varying controls on
pitch and speed. ControlVC uses pre-trained encoders to compute pitch and
linguistic embeddings from the source utterance and speaker embeddings from the
target utterance. These embeddings are then concatenated and converted to
speech using a vocoder. It achieves speed control through TD-PSOLA
pre-processing on the source utterance, and achieves pitch control by
manipulating the pitch contour before feeding it to the pitch encoder.
Systematic subjective and objective evaluations are conducted to assess the
speech quality and controllability. Results show that, on non-parallel and
zero-shot conversion tasks, ControlVC significantly outperforms two other
self-constructed baselines on speech quality, and it can successfully achieve
time-varying pitch and speed control.Comment: Audio samples: https://bit.ly/3PsrKLJ; Code:
https://github.com/MelissaChen15/control-v
- …