Diffusion-based generative models have achieved remarkable success in various
domains. It trains a model on denoising tasks that encompass different noise
levels simultaneously, representing a form of multi-task learning (MTL).
However, analyzing and improving diffusion models from an MTL perspective
remains under-explored. In particular, MTL can sometimes lead to the well-known
phenomenon of negative transfer, which results in the performance
degradation of certain tasks due to conflicts between tasks. In this paper, we
aim to analyze diffusion training from an MTL standpoint, presenting two key
observations: (O1) the task affinity between denoising tasks
diminishes as the gap between noise levels widens, and (O2) negative
transfer can arise even in the context of diffusion training. Building upon
these observations, our objective is to enhance diffusion training by
mitigating negative transfer. To achieve this, we propose leveraging existing
MTL methods, but the presence of a huge number of denoising tasks makes this
computationally expensive to calculate the necessary per-task loss or gradient.
To address this challenge, we propose clustering the denoising tasks into small
task clusters and applying MTL methods to them. Specifically, based on
(O2), we employ interval clustering to enforce temporal proximity
among denoising tasks within clusters. We show that interval clustering can be
solved with dynamic programming and utilize signal-to-noise ratio, timestep,
and task affinity for clustering objectives. Through this, our approach
addresses the issue of negative transfer in diffusion models by allowing for
efficient computation of MTL methods. We validate the proposed clustering and
its integration with MTL methods through various experiments, demonstrating
improved sample quality of diffusion models.Comment: 22 pages, 12 figures, under revie