The goal of speech enhancement (SE) is to eliminate the background
interference from the noisy speech signal. Generative models such as diffusion
models (DM) have been applied to the task of SE because of better
generalization in unseen noisy scenes. Technical routes for the DM-based SE
methods can be summarized into three types: task-adapted diffusion process
formulation, generator-plus-conditioner (GPC) structures and the multi-stage
frameworks. We focus on the first two approaches, which are constructed under
the GPC architecture and use the task-adapted diffusion process to better deal
with the real noise. However, the performance of these SE models is limited by
the following issues: (a) Non-Gaussian noise estimation in the task-adapted
diffusion process. (b) Conditional domain bias caused by the weak conditioner
design in the GPC structure. (c) Large amount of residual noise caused by
unreasonable interpolation operations during inference. To solve the above
problems, we propose a noise-aware diffusion-based SE model (NADiffuSE) to
boost the SE performance, where the noise representation is extracted from the
noisy speech signal and introduced as a global conditional information for
estimating the non-Gaussian components. Furthermore, the anchor-based inference
algorithm is employed to achieve a compromise between the speech distortion and
noise residual. In order to mitigate the performance degradation caused by the
conditional domain bias in the GPC framework, we investigate three model
variants, all of which can be viewed as multi-stage SE based on the
preprocessing networks for Mel spectrograms. Experimental results show that
NADiffuSE outperforms other DM-based SE models under the GPC infrastructure.
Audio samples are available at: https://square-of-w.github.io/NADiffuSE-demo/