173 research outputs found
Automatic Estimation of Intelligibility Measure for Consonants in Speech
In this article, we provide a model to estimate a real-valued measure of the
intelligibility of individual speech segments. We trained regression models
based on Convolutional Neural Networks (CNN) for stop consonants
\textipa{/p,t,k,b,d,g/} associated with vowel \textipa{/A/}, to estimate the
corresponding Signal to Noise Ratio (SNR) at which the Consonant-Vowel (CV)
sound becomes intelligible for Normal Hearing (NH) ears. The intelligibility
measure for each sound is called SNR, and is defined to be the SNR level
at which human participants are able to recognize the consonant at least 90\%
correctly, on average, as determined in prior experiments with NH subjects.
Performance of the CNN is compared to a baseline prediction based on automatic
speech recognition (ASR), specifically, a constant offset subtracted from the
SNR at which the ASR becomes capable of correctly labeling the consonant.
Compared to baseline, our models were able to accurately estimate the
SNR~intelligibility measure with less than 2 [dB] Mean Squared Error
(MSE) on average, while the baseline ASR-defined measure computes
SNR~with a variance of 5.2 to 26.6 [dB], depending on the consonant.Comment: 5 pages, 1 figure, 7 tables, submitted to Inter Speech 2020
Conferenc
Deep learning-based denoising streamed from mobile phones improves speech-in-noise understanding for hearing aid users
The hearing loss of almost half a billion people is commonly treated with
hearing aids. However, current hearing aids often do not work well in
real-world noisy environments. We present a deep learning based denoising
system that runs in real time on iPhone 7 and Samsung Galaxy S10 (25ms
algorithmic latency). The denoised audio is streamed to the hearing aid,
resulting in a total delay of around 75ms. In tests with hearing aid users
having moderate to severe hearing loss, our denoising system improves audio
across three tests: 1) listening for subjective audio ratings, 2) listening for
objective speech intelligibility, and 3) live conversations in a noisy
environment for subjective ratings. Subjective ratings increase by more than
40%, for both the listening test and the live conversation compared to a fitted
hearing aid as a baseline. Speech reception thresholds, measuring speech
understanding in noise, improve by 1.6 dB SRT. Ours is the first denoising
system that is implemented on a mobile device, streamed directly to users'
hearing aids using only a single channel as audio input while improving user
satisfaction on all tested aspects, including speech intelligibility. This
includes overall preference of the denoised and streamed signal over the
hearing aid, thereby accepting the higher latency for the significant
improvement in speech understanding
Speech Dereverberation Based on Integrated Deep and Ensemble Learning Algorithm
Reverberation, which is generally caused by sound reflections from walls,
ceilings, and floors, can result in severe performance degradation of acoustic
applications. Due to a complicated combination of attenuation and time-delay
effects, the reverberation property is difficult to characterize, and it
remains a challenging task to effectively retrieve the anechoic speech signals
from reverberation ones. In the present study, we proposed a novel integrated
deep and ensemble learning algorithm (IDEA) for speech dereverberation. The
IDEA consists of offline and online phases. In the offline phase, we train
multiple dereverberation models, each aiming to precisely dereverb speech
signals in a particular acoustic environment; then a unified fusion function is
estimated that aims to integrate the information of multiple dereverberation
models. In the online phase, an input utterance is first processed by each of
the dereverberation models. The outputs of all models are integrated
accordingly to generate the final anechoic signal. We evaluated the IDEA on
designed acoustic environments, including both matched and mismatched
conditions of the training and testing data. Experimental results confirm that
the proposed IDEA outperforms single deep-neural-network-based dereverberation
model with the same model architecture and training data
An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation
Speech enhancement and speech separation are two related tasks, whose purpose
is to extract either one or more target speech signals, respectively, from a
mixture of sounds generated by several sources. Traditionally, these tasks have
been tackled using signal processing and machine learning techniques applied to
the available acoustic signals. Since the visual aspect of speech is
essentially unaffected by the acoustic environment, visual information from the
target speakers, such as lip movements and facial expressions, has also been
used for speech enhancement and speech separation systems. In order to
efficiently fuse acoustic and visual information, researchers have exploited
the flexibility of data-driven approaches, specifically deep learning,
achieving strong performance. The ceaseless proposal of a large number of
techniques to extract features and fuse multimodal information has highlighted
the need for an overview that comprehensively describes and discusses
audio-visual speech enhancement and separation based on deep learning. In this
paper, we provide a systematic survey of this research topic, focusing on the
main elements that characterise the systems in the literature: acoustic
features; visual features; deep learning methods; fusion techniques; training
targets and objective functions. In addition, we review deep-learning-based
methods for speech reconstruction from silent videos and audio-visual sound
source separation for non-speech signals, since these methods can be more or
less directly applied to audio-visual speech enhancement and separation.
Finally, we survey commonly employed audio-visual speech datasets, given their
central role in the development of data-driven approaches, and evaluation
methods, because they are generally used to compare different systems and
determine their performance
Single-Microphone Speech Enhancement and Separation Using Deep Learning
The cocktail party problem comprises the challenging task of understanding a
speech signal in a complex acoustic environment, where multiple speakers and
background noise signals simultaneously interfere with the speech signal of
interest. A signal processing algorithm that can effectively increase the
speech intelligibility and quality of speech signals in such complicated
acoustic situations is highly desirable. Especially for applications involving
mobile communication devices and hearing assistive devices. Due to the
re-emergence of machine learning techniques, today, known as deep learning, the
challenges involved with such algorithms might be overcome. In this PhD thesis,
we study and develop deep learning-based techniques for two sub-disciplines of
the cocktail party problem: single-microphone speech enhancement and
single-microphone multi-talker speech separation. Specifically, we conduct
in-depth empirical analysis of the generalizability capability of modern deep
learning-based single-microphone speech enhancement algorithms. We show that
performance of such algorithms is closely linked to the training data, and good
generalizability can be achieved with carefully designed training data.
Furthermore, we propose uPIT, a deep learning-based algorithm for
single-microphone speech separation and we report state-of-the-art results on a
speaker-independent multi-talker speech separation task. Additionally, we show
that uPIT works well for joint speech separation and enhancement without
explicit prior knowledge about the noise type or number of speakers. Finally,
we show that deep learning-based speech enhancement algorithms designed to
minimize the classical short-time spectral amplitude mean squared error leads
to enhanced speech signals which are essentially optimal in terms of STOI, a
state-of-the-art speech intelligibility estimator.Comment: PhD Thesis. 233 page
- …