133 research outputs found
Towards Unified All-Neural Beamforming for Time and Frequency Domain Speech Separation
Recently, frequency domain all-neural beamforming methods have achieved
remarkable progress for multichannel speech separation. In parallel, the
integration of time domain network structure and beamforming also gains
significant attention. This study proposes a novel all-neural beamforming
method in time domain and makes an attempt to unify the all-neural beamforming
pipelines for time domain and frequency domain multichannel speech separation.
The proposed model consists of two modules: separation and beamforming. Both
modules perform temporal-spectral-spatial modeling and are trained from
end-to-end using a joint loss function. The novelty of this study lies in two
folds. Firstly, a time domain directional feature conditioned on the direction
of the target speaker is proposed, which can be jointly optimized within the
time domain architecture to enhance target signal estimation. Secondly, an
all-neural beamforming network in time domain is designed to refine the
pre-separated results. This module features with parametric time-variant
beamforming coefficient estimation, without explicitly following the derivation
of optimal filters that may lead to an upper bound. The proposed method is
evaluated on simulated reverberant overlapped speech data derived from the
AISHELL-1 corpus. Experimental results demonstrate significant performance
improvements over frequency domain state-of-the-arts, ideal magnitude masks and
existing time domain neural beamforming methods
Speech enhancement with frequency domain auto-regressive modeling
Speech applications in far-field real world settings often deal with signals
that are corrupted by reverberation. The task of dereverberation constitutes an
important step to improve the audible quality and to reduce the error rates in
applications like automatic speech recognition (ASR). We propose a unified
framework of speech dereverberation for improving the speech quality and the
ASR performance using the approach of envelope-carrier decomposition provided
by an autoregressive (AR) model. The AR model is applied in the frequency
domain of the sub-band speech signals to separate the envelope and carrier
parts. A novel neural architecture based on dual path long short term memory
(DPLSTM) model is proposed, which jointly enhances the sub-band envelope and
carrier components. The dereverberated envelope-carrier signals are modulated
and the sub-band signals are synthesized to reconstruct the audio signal back.
The DPLSTM model for dereverberation of envelope and carrier components also
allows the joint learning of the network weights for the down stream ASR task.
In the ASR tasks on the REVERB challenge dataset as well as on the VOiCES
dataset, we illustrate that the joint learning of speech dereverberation
network and the E2E ASR model yields significant performance improvements over
the baseline ASR system trained on log-mel spectrogram as well as other
benchmarks for dereverberation (average relative improvements of 10-24% over
the baseline system). The speech quality improvements, evaluated using
subjective listening tests, further highlight the improved quality of the
reconstructed audio.Comment: 10 page
IANS: Intelligibility-aware Null-steering Beamforming for Dual-Microphone Arrays
Beamforming techniques are popular in speech-related applications due to
their effective spatial filtering capabilities. Nonetheless, conventional
beamforming techniques generally depend heavily on either the target's
direction-of-arrival (DOA), relative transfer function (RTF) or covariance
matrix. This paper presents a new approach, the intelligibility-aware
null-steering (IANS) beamforming framework, which uses the STOI-Net
intelligibility prediction model to improve speech intelligibility without
prior knowledge of the speech signal parameters mentioned earlier. The IANS
framework combines a null-steering beamformer (NSBF) to generate a set of
beamformed outputs, and STOI-Net, to determine the optimal result. Experimental
results indicate that IANS can produce intelligibility-enhanced signals using a
small dual-microphone array. The results are comparable to those obtained by
null-steering beamformers with given knowledge of DOAs.Comment: Preprint submitted to IEEE MLSP 202
Machine Learning for Beamforming in Audio, Ultrasound, and Radar
Multi-sensor signal processing plays a crucial role in the working of several everyday technologies, from correctly understanding speech on smart home devices to ensuring aircraft fly safely. A specific type of multi-sensor signal processing called beamforming forms a central part of this thesis. Beamforming works by combining the information from several spatially distributed sensors to directionally filter information, boosting the signal from a certain direction but suppressing others. The idea of beamforming is key to the domains of audio, ultrasound, and radar.
Machine learning is the other central part of this thesis. Machine learning, and especially its sub-field of deep learning, has enabled breakneck progress in tackling several problems that were previously thought intractable. Today, machine learning powers many of the cutting edge systems we see on the internet for image classification, speech recognition, language translation, and more.
In this dissertation, we look at beamforming pipelines in audio, ultrasound, and radar from a machine learning lens and endeavor to improve different parts of the pipelines using ideas from machine learning. We start off in the audio domain and derive a machine learning inspired beamformer to tackle the problem of ensuring the audio captured by a camera matches its visual content, a problem we term audiovisual zooming. Staying in the audio domain, we then demonstrate how deep learning can be used to improve the perceptual qualities of speech by denoising speech clipping, codec distortions, and gaps in speech.
Transitioning to the ultrasound domain, we improve the performance of short-lag spatial coherence ultrasound imaging by exploiting the differences in tissue texture at each short lag value by applying robust principal component analysis. Next, we use deep learning as an alternative to beamforming in ultrasound and improve the information extraction pipeline by simultaneously generating both a segmentation map and B-mode image of high quality directly from raw received ultrasound data.
Finally, we move to the radar domain and study how deep learning can be used to improve signal quality in ultra-wideband synthetic aperture radar by suppressing radio frequency interference, random spectral gaps, and contiguous block spectral gaps. By training and applying the networks on raw single-aperture data prior to beamforming, it can work with myriad sensor geometries and different beamforming equations, a crucial requirement in synthetic aperture radar
- …