Binaural audio plays a significant role in constructing immersive augmented
and virtual realities. As it is expensive to record binaural audio from the
real world, synthesizing them from mono audio has attracted increasing
attention. This synthesis process involves not only the basic physical warping
of the mono audio, but also room reverberations and head/ear related
filtrations, which, however, are difficult to accurately simulate in
traditional digital signal processing. In this paper, we formulate the
synthesis process from a different perspective by decomposing the binaural
audio into a common part that shared by the left and right channels as well as
a specific part that differs in each channel. Accordingly, we propose
BinauralGrad, a novel two-stage framework equipped with diffusion models to
synthesize them respectively. Specifically, in the first stage, the common
information of the binaural audio is generated with a single-channel diffusion
model conditioned on the mono audio, based on which the binaural audio is
generated by a two-channel diffusion model in the second stage. Combining this
novel perspective of two-stage synthesis with advanced generative models (i.e.,
the diffusion models),the proposed BinauralGrad is able to generate accurate
and high-fidelity binaural audio samples. Experiment results show that on a
benchmark dataset, BinauralGrad outperforms the existing baselines by a large
margin in terms of both object and subject evaluation metrics (Wave L2: 0.128
vs. 0.157, MOS: 3.80 vs. 3.61). The generated audio samples
(https://speechresearch.github.io/binauralgrad) and code
(https://github.com/microsoft/NeuralSpeech/tree/master/BinauralGrad) are
available online.Comment: NeurIPS 2022 camera versio