17 research outputs found
Blurring Diffusion Models
Recently, Rissanen et al., (2022) have presented a new type of diffusion
process for generative modeling based on heat dissipation, or blurring, as an
alternative to isotropic Gaussian diffusion. Here, we show that blurring can
equivalently be defined through a Gaussian diffusion process with non-isotropic
noise. In making this connection, we bridge the gap between inverse heat
dissipation and denoising diffusion, and we shed light on the inductive bias
that results from this modeling choice. Finally, we propose a generalized class
of diffusion models that offers the best of both standard Gaussian denoising
diffusion and inverse heat dissipation, which we call Blurring Diffusion
Models
simple diffusion: End-to-end diffusion for high resolution images
Currently, applying diffusion models in pixel space of high resolution images
is difficult. Instead, existing approaches focus on diffusion in lower
dimensional spaces (latent diffusion), or have multiple super-resolution levels
of generation referred to as cascades. The downside is that these approaches
add additional complexity to the diffusion framework.
This paper aims to improve denoising diffusion for high resolution images
while keeping the model as simple as possible. The paper is centered around the
research question: How can one train a standard denoising diffusion models on
high resolution images, and still obtain performance comparable to these
alternate approaches?
The four main findings are: 1) the noise schedule should be adjusted for high
resolution images, 2) It is sufficient to scale only a particular part of the
architecture, 3) dropout should be added at specific locations in the
architecture, and 4) downsampling is an effective strategy to avoid high
resolution feature maps. Combining these simple yet effective techniques, we
achieve state-of-the-art on image generation among diffusion models without
sampling modifiers on ImageNet
Emerging Convolutions for Generative Normalizing Flows
Generative flows are attractive because they admit exact likelihood
optimization and efficient image synthesis. Recently, Kingma & Dhariwal (2018)
demonstrated with Glow that generative flows are capable of generating high
quality images. We generalize the 1 x 1 convolutions proposed in Glow to
invertible d x d convolutions, which are more flexible since they operate on
both channel and spatial axes. We propose two methods to produce invertible
convolutions that have receptive fields identical to standard convolutions:
Emerging convolutions are obtained by chaining specific autoregressive
convolutions, and periodic convolutions are decoupled in the frequency domain.
Our experiments show that the flexibility of d x d convolutions significantly
improves the performance of generative flow models on galaxy images, CIFAR10
and ImageNet.Comment: Accepted at International Conference on Machine Learning (ICML) 201
Learning Likelihoods with Conditional Normalizing Flows
Normalizing Flows (NFs) are able to model complicated distributions p(y) with
strong inter-dimensional correlations and high multimodality by transforming a
simple base density p(z) through an invertible neural network under the change
of variables formula. Such behavior is desirable in multivariate structured
prediction tasks, where handcrafted per-pixel loss-based methods inadequately
capture strong correlations between output dimensions. We present a study of
conditional normalizing flows (CNFs), a class of NFs where the base density to
output space mapping is conditioned on an input x, to model conditional
densities p(y|x). CNFs are efficient in sampling and inference, they can be
trained with a likelihood-based objective, and CNFs, being generative flows, do
not suffer from mode collapse or training instabilities. We provide an
effective method to train continuous CNFs for binary problems and in
particular, we apply these CNFs to super-resolution and vessel segmentation
tasks demonstrating competitive performance on standard benchmark datasets in
terms of likelihood and conventional metrics.Comment: 18 pages, 8 Tables, 9 Figures, Preprin
The Convolution Exponential and Generalized Sylvester Flows
This paper introduces a new method to build linear flows, by taking the
exponential of a linear transformation. This linear transformation does not
need to be invertible itself, and the exponential has the following desirable
properties: it is guaranteed to be invertible, its inverse is straightforward
to compute and the log Jacobian determinant is equal to the trace of the linear
transformation. An important insight is that the exponential can be computed
implicitly, which allows the use of convolutional layers. Using this insight,
we develop new invertible transformations named convolution exponentials and
graph convolution exponentials, which retain the equivariance of their
underlying transformations. In addition, we generalize Sylvester Flows and
propose Convolutional Sylvester Flows which are based on the generalization and
the convolution exponential as basis change. Empirically, we show that the
convolution exponential outperforms other linear transformations in generative
flows on CIFAR10 and the graph convolution exponential improves the performance
of graph normalizing flows. In addition, we show that Convolutional Sylvester
Flows improve performance over residual flows as a generative flow model
measured in log-likelihood
SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flows
Normalizing flows and variational autoencoders are powerful generative models
that can represent complicated density functions. However, they both impose
constraints on the models: Normalizing flows use bijective transformations to
model densities whereas VAEs learn stochastic transformations that are
non-invertible and thus typically do not provide tractable estimates of the
marginal likelihood. In this paper, we introduce SurVAE Flows: A modular
framework of composable transformations that encompasses VAEs and normalizing
flows. SurVAE Flows bridge the gap between normalizing flows and VAEs with
surjective transformations, wherein the transformations are deterministic in
one direction -- thereby allowing exact likelihood computation, and stochastic
in the reverse direction -- hence providing a lower bound on the corresponding
likelihood. We show that several recently proposed methods, including
dequantization and augmented normalizing flows, can be expressed as SurVAE
Flows. Finally, we introduce common operations such as the max value, the
absolute value, sorting and stochastic permutation as composable layers in
SurVAE Flows