34 research outputs found
Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear
Developing embodied agents in simulation has been a key research topic in
recent years. Exciting new tasks, algorithms, and benchmarks have been
developed in various simulators. However, most of them assume deaf agents in
silent environments, while we humans perceive the world with multiple senses.
We introduce Sonicverse, a multisensory simulation platform with integrated
audio-visual simulation for training household agents that can both see and
hear. Sonicverse models realistic continuous audio rendering in 3D environments
in real-time. Together with a new audio-visual VR interface that allows humans
to interact with agents with audio, Sonicverse enables a series of embodied AI
tasks that need audio-visual perception. For semantic audio-visual navigation
in particular, we also propose a new multi-task learning model that achieves
state-of-the-art performance. In addition, we demonstrate Sonicverse's realism
via sim-to-real transfer, which has not been achieved by other simulators: an
agent trained in Sonicverse can successfully perform audio-visual navigation in
real-world environments. Sonicverse is available at:
https://github.com/StanfordVL/Sonicverse.Comment: In ICRA 2023. Project page:
https://ai.stanford.edu/~rhgao/sonicverse/. Code:
https://github.com/StanfordVL/sonicverse. Gao and Li contributed equally to
this work and are in alphabetical orde
Weakly-Supervised Audio-Visual Segmentation
Audio-visual segmentation is a challenging task that aims to predict
pixel-level masks for sound sources in a video. Previous work applied a
comprehensive manually designed architecture with countless pixel-wise accurate
masks as supervision. However, these pixel-level masks are expensive and not
available in all cases. In this work, we aim to simplify the supervision as the
instance-level annotation, i.e., weakly-supervised audio-visual segmentation.
We present a novel Weakly-Supervised Audio-Visual Segmentation framework,
namely WS-AVS, that can learn multi-scale audio-visual alignment with
multi-scale multiple-instance contrastive learning for audio-visual
segmentation. Extensive experiments on AVSBench demonstrate the effectiveness
of our WS-AVS in the weakly-supervised audio-visual segmentation of
single-source and multi-source scenarios