10,689 research outputs found
Efficient Implementation of the Room Simulator for Training Deep Neural Network Acoustic Models
In this paper, we describe how to efficiently implement an acoustic room
simulator to generate large-scale simulated data for training deep neural
networks. Even though Google Room Simulator in [1] was shown to be quite
effective in reducing the Word Error Rates (WERs) for far-field applications by
generating simulated far-field training sets, it requires a very large number
of Fast Fourier Transforms (FFTs) of large size. Room Simulator in [1] used
approximately 80 percent of Central Processing Unit (CPU) usage in our CPU +
Graphics Processing Unit (GPU) training architecture [2]. In this work, we
implement an efficient OverLap Addition (OLA) based filtering using the
open-source FFTW3 library. Further, we investigate the effects of the Room
Impulse Response (RIR) lengths. Experimentally, we conclude that we can cut the
tail portions of RIRs whose power is less than 20 dB below the maximum power
without sacrificing the speech recognition accuracy. However, we observe that
cutting RIR tail more than this threshold harms the speech recognition accuracy
for rerecorded test sets. Using these approaches, we were able to reduce CPU
usage for the room simulator portion down to 9.69 percent in CPU/GPU training
architecture. Profiling result shows that we obtain 22.4 times speed-up on a
single machine and 37.3 times speed up on Google's distributed training
infrastructure.Comment: Published at INTERSPEECH 2018.
(https://www.isca-speech.org/archive/Interspeech_2018/abstracts/2566.html
PRESENCE: A human-inspired architecture for speech-based human-machine interaction
Recent years have seen steady improvements in the quality and performance of speech-based human-machine interaction driven by a significant convergence in the methods and techniques employed. However, the quantity of training data required to improve state-of-the-art systems seems to be growing exponentially and performance appears to be asymptotic to a level that may be inadequate for many real-world applications. This suggests that there may be a fundamental flaw in the underlying architecture of contemporary systems, as well as a failure to capitalize on the combinatorial properties of human spoken language. This paper addresses these issues and presents a novel architecture for speech-based human-machine interaction inspired by recent findings in the neurobiology of living systems. Called PRESENCE-"PREdictive SENsorimotor Control and Emulation" - this new architecture blurs the distinction between the core components of a traditional spoken language dialogue system and instead focuses on a recursive hierarchical feedback control structure. Cooperative and communicative behavior emerges as a by-product of an architecture that is founded on a model of interaction in which the system has in mind the needs and intentions of a user and a user has in mind the needs and intentions of the system
SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learning
We introduce SoundSpaces 2.0, a platform for on-the-fly geometry-based audio
rendering for 3D environments. Given a 3D mesh of a real-world environment,
SoundSpaces can generate highly realistic acoustics for arbitrary sounds
captured from arbitrary microphone locations. Together with existing 3D visual
assets, it supports an array of audio-visual research tasks, such as
audio-visual navigation, mapping, source localization and separation, and
acoustic matching. Compared to existing resources, SoundSpaces 2.0 has the
advantages of allowing continuous spatial sampling, generalization to novel
environments, and configurable microphone and material properties. To our
knowledge, this is the first geometry-based acoustic simulation that offers
high fidelity and realism while also being fast enough to use for embodied
learning. We showcase the simulator's properties and benchmark its performance
against real-world audio measurements. In addition, we demonstrate two
downstream tasks -- embodied navigation and far-field automatic speech
recognition -- and highlight sim2real performance for the latter. SoundSpaces
2.0 is publicly available to facilitate wider research for perceptual systems
that can both see and hear.Comment: Camera-ready version. Website: https://soundspaces.org. Project page:
https://vision.cs.utexas.edu/projects/soundspaces
GWA: A Large High-Quality Acoustic Dataset for Audio Processing
We present the Geometric-Wave Acoustic (GWA) dataset, a large-scale audio
dataset of over 2 million synthetic room impulse responses (IRs) and their
corresponding detailed geometric and simulation configurations. Our dataset
samples acoustic environments from over 6.8K high-quality diverse and
professionally designed houses represented as semantically labeled 3D meshes.
We also present a novel real-world acoustic materials assignment scheme based
on semantic matching that uses a sentence transformer model. We compute
high-quality impulse responses corresponding to accurate low-frequency and
high-frequency wave effects by automatically calibrating geometric acoustic
ray-tracing with a finite-difference time-domain wave solver. We demonstrate
the higher accuracy of our IRs by comparing with recorded IRs from complex
real-world environments. The code and the full dataset will be released at the
time of publication. Moreover, we highlight the benefits of GWA on audio deep
learning tasks such as automated speech recognition, speech enhancement, and
speech separation. We observe significant improvement over prior synthetic IR
datasets in all tasks due to using our dataset.Comment: Project webpage https://gamma.umd.edu/pro/sound/gw
Novel-View Acoustic Synthesis
We introduce the novel-view acoustic synthesis (NVAS) task: given the sight
and sound observed at a source viewpoint, can we synthesize the sound of that
scene from an unseen target viewpoint? We propose a neural rendering approach:
Visually-Guided Acoustic Synthesis (ViGAS) network that learns to synthesize
the sound of an arbitrary point in space by analyzing the input audio-visual
cues. To benchmark this task, we collect two first-of-their-kind large-scale
multi-view audio-visual datasets, one synthetic and one real. We show that our
model successfully reasons about the spatial cues and synthesizes faithful
audio on both datasets. To our knowledge, this work represents the very first
formulation, dataset, and approach to solve the novel-view acoustic synthesis
task, which has exciting potential applications ranging from AR/VR to art and
design. Unlocked by this work, we believe that the future of novel-view
synthesis is in multi-modal learning from videos.Comment: Project page: https://vision.cs.utexas.edu/projects/nva
- …