Diffusion models (DMs) are widely used for generating high-quality
high-dimensional images in a non-differentially private manner. To address this
challenge, recent papers suggest pre-training DMs with public data, then
fine-tuning them with private data using DP-SGD for a relatively short period.
In this paper, we further improve the current state of DMs with DP by adopting
the Latent Diffusion Models (LDMs). LDMs are equipped with powerful pre-trained
autoencoders that map the high-dimensional pixels into lower-dimensional latent
representations, in which DMs are trained, yielding a more efficient and fast
training of DMs. In our algorithm, DP-LDMs, rather than fine-tuning the entire
DMs, we fine-tune only the attention modules of LDMs at varying layers with
privacy-sensitive data, reducing the number of trainable parameters by roughly
90% and achieving a better accuracy, compared to fine-tuning the entire DMs.
The smaller parameter space to fine-tune with DP-SGD helps our algorithm to
achieve new state-of-the-art results in several public-private benchmark data
pairs.Our approach also allows us to generate more realistic, high-dimensional
images (256x256) and those conditioned on text prompts with differential
privacy, which have not been attempted before us, to the best of our knowledge.
Our approach provides a promising direction for training more powerful, yet
training-efficient differentially private DMs, producing high-quality
high-dimensional DP images