24 research outputs found
Neural Latent Aligner: Cross-trial Alignment for Learning Representations of Complex, Naturalistic Neural Data
Understanding the neural implementation of complex human behaviors is one of
the major goals in neuroscience. To this end, it is crucial to find a true
representation of the neural data, which is challenging due to the high
complexity of behaviors and the low signal-to-ratio (SNR) of the signals. Here,
we propose a novel unsupervised learning framework, Neural Latent Aligner
(NLA), to find well-constrained, behaviorally relevant neural representations
of complex behaviors. The key idea is to align representations across repeated
trials to learn cross-trial consistent information. Furthermore, we propose a
novel, fully differentiable time warping model (TWM) to resolve the temporal
misalignment of trials. When applied to intracranial electrocorticography
(ECoG) of natural speaking, our model learns better representations for
decoding behaviors than the baseline models, especially in lower dimensional
space. The TWM is empirically validated by measuring behavioral coherence
between aligned trials. The proposed framework learns more cross-trial
consistent representations than the baselines, and when visualized, the
manifold reveals shared neural trajectories across trials.Comment: Accepted at ICML 202
A Fast and Accurate Pitch Estimation Algorithm Based on the Pseudo Wigner-Ville Distribution
Estimation of fundamental frequency (F0) in voiced segments of speech
signals, also known as pitch tracking, plays a crucial role in pitch
synchronous speech analysis, speech synthesis, and speech manipulation. In this
paper, we capitalize on the high time and frequency resolution of the pseudo
Wigner-Ville distribution (PWVD) and propose a new PWVD-based pitch estimation
method. We devise an efficient algorithm to compute PWVD faster and use
cepstrum-based pre-filtering to avoid cross-term interference. Evaluating our
approach on a database with speech and electroglottograph (EGG) recordings
yields a state-of-the-art mean absolute error (MAE) of around 4Hz. Our approach
is also effective at voiced/unvoiced classification and handling sudden
frequency changes
Articulation GAN: Unsupervised modeling of articulatory learning
Generative deep neural networks are widely used for speech synthesis, but
most existing models directly generate waveforms or spectral outputs. Humans,
however, produce speech by controlling articulators, which results in the
production of speech sounds through physical properties of sound propagation.
We propose a new unsupervised generative model of speech production/synthesis
that includes articulatory representations and thus more closely mimics human
speech production. We introduce the Articulatory Generator to the Generative
Adversarial Network paradigm. The Articulatory Generator needs to learn to
generate articulatory representations (electromagnetic articulography or EMA)
in a fully unsupervised manner without ever accessing EMA data. A separate
pre-trained physical model (ema2wav) then transforms the generated EMA
representations to speech waveforms, which get sent to the Discriminator for
evaluation. Articulatory analysis of the generated EMA representations suggests
that the network learns to control articulators in a manner that closely
follows human articulators during speech production. Acoustic analysis of the
outputs suggest that the network learns to generate words that are part of
training data as well as novel innovative words that are absent from training
data. Our proposed architecture thus allows modeling of articulatory learning
with deep neural networks from raw audio inputs in a fully unsupervised manner.
We additionally discuss implications of articulatory representations for
cognitive models of human language and speech technology in general
CiwaGAN: Articulatory information exchange
Humans encode information into sounds by controlling articulators and decode
information from sounds using the auditory apparatus. This paper introduces
CiwaGAN, a model of human spoken language acquisition that combines
unsupervised articulatory modeling with an unsupervised model of information
exchange through the auditory modality. While prior research includes
unsupervised articulatory modeling and information exchange separately, our
model is the first to combine the two components. The paper also proposes an
improved articulatory model with more interpretable internal representations.
The proposed CiwaGAN model is the most realistic approximation of human spoken
language acquisition using deep learning. As such, it is useful for cognitively
plausible simulations of the human speech act
Evidence of Vocal Tract Articulation in Self-Supervised Learning of Speech
Recent self-supervised learning (SSL) models have proven to learn rich
representations of speech, which can readily be utilized by diverse downstream
tasks. To understand such utilities, various analyses have been done for speech
SSL models to reveal which and how information is encoded in the learned
representations. Although the scope of previous analyses is extensive in
acoustic, phonetic, and semantic perspectives, the physical grounding by speech
production has not yet received full attention. To bridge this gap, we conduct
a comprehensive analysis to link speech representations to articulatory
trajectories measured by electromagnetic articulography (EMA). Our analysis is
based on a linear probing approach where we measure articulatory score as an
average correlation of linear mapping to EMA. We analyze a set of SSL models
selected from the leaderboard of the SUPERB benchmark and perform further
layer-wise analyses on two most successful models, Wav2Vec 2.0 and HuBERT.
Surprisingly, representations from the recent speech SSL models are highly
correlated with EMA traces (best: r = 0.81), and only 5 minutes are sufficient
to train a linear model with high performance (r = 0.77). Our findings suggest
that SSL models learn to align closely with continuous articulations, and
provide a novel insight into speech SSL
Towards Streaming Speech-to-Avatar Synthesis
Streaming speech-to-avatar synthesis creates real-time animations for a
virtual character from audio data. Accurate avatar representations of speech
are important for the visualization of sound in linguistics, phonetics, and
phonology, visual feedback to assist second language acquisition, and virtual
embodiment for paralyzed patients. Previous works have highlighted the
capability of deep articulatory inversion to perform high-quality avatar
animation using electromagnetic articulography (EMA) features. However, these
models focus on offline avatar synthesis with recordings rather than real-time
audio, which is necessary for live avatar visualization or embodiment. To
address this issue, we propose a method using articulatory inversion for
streaming high quality facial and inner-mouth avatar animation from real-time
audio. Our approach achieves 130ms average streaming latency for every 0.1
seconds of audio with a 0.792 correlation with ground truth articulations.
Finally, we show generated mouth and tongue animations to demonstrate the
efficacy of our methodology.Comment: Submitted to ICASSP 202
Speaker-Independent Acoustic-to-Articulatory Speech Inversion
To build speech processing methods that can handle speech as naturally as
humans, researchers have explored multiple ways of building an invertible
mapping from speech to an interpretable space. The articulatory space is a
promising inversion target, since this space captures the mechanics of speech
production. To this end, we build an acoustic-to-articulatory inversion (AAI)
model that leverages autoregression, adversarial training, and self supervision
to generalize to unseen speakers. Our approach obtains 0.784 correlation on an
electromagnetic articulography (EMA) dataset, improving the state-of-the-art by
12.5%. Additionally, we show the interpretability of these representations
through directly comparing the behavior of estimated representations with
speech production behavior. Finally, we propose a resynthesis-based AAI
evaluation metric that does not rely on articulatory labels, demonstrating its
efficacy with an 18-speaker dataset
Speech synthesis from neural decoding of spoken sentences
Technology that translates neural activity into speech would be transformative for people who are unable to communicate as a result of neurological impairments. Decoding speech from neural activity is challenging because speaking requires very precise and rapid multi-dimensional control of vocal tract articulators. Here we designed a neural decoder that explicitly leverages kinematic and sound representations encoded in human cortical activity to synthesize audible speech. Recurrent neural networks first decoded directly recorded cortical activity into representations of articulatory movement, and then transformed these representations into speech acoustics. In closed vocabulary tests, listeners could readily identify and transcribe speech synthesized from cortical activity. Intermediate articulatory dynamics enhanced performance even with limited data. Decoded articulatory representations were highly conserved across speakers, enabling a component of the decoder to be transferrable across participants. Furthermore, the decoder could synthesize speech when a participant silently mimed sentences. These findings advance the clinical viability of using speech neuroprosthetic technology to restore spoken communication
Brain2Char: a deep architecture for decoding text from brain recordings
Objective.Decoding language representations directly from the brain can enable new brain-computer interfaces (BCIs) for high bandwidth human-human and human-machine communication. Clinically, such technologies can restore communication in people with neurological conditions affecting their ability to speak.Approach. In this study, we propose a novel deep network architecture Brain2Char, for directly decoding text (specifically character sequences) from direct brain recordings (called electrocorticography, ECoG). Brain2Char framework combines state-of-the-art deep learning modules-3D Inception layers for multiband spatiotemporal feature extraction from neural data and bidirectional recurrent layers, dilated convolution layers followed by language model weighted beam search to decode character sequences, and optimizing a connectionist temporal classification loss. Additionally, given the highly non-linear transformations that underlie the conversion of cortical function to character sequences, we perform regularizations on the network's latent representations motivated by insights into cortical encoding of speech production and artifactual aspects specific to ECoG data acquisition. To do this, we impose auxiliary losses on latent representations for articulatory movements, speech acoustics and session specific non-linearities.Main results.In three (out of four) participants reported here, Brain2Char achieves 10.6%, 8.5%, and 7.0% word error rates respectively on vocabulary sizes ranging from 1200 to 1900 words.Significance.These results establish a newend-to-end approachon decoding text frombrain signalsand demonstrate the potential of Brain2Char as a high-performance communication BCI