33 research outputs found
Emerging Convolutions for Generative Normalizing Flows
Generative flows are attractive because they admit exact likelihood
optimization and efficient image synthesis. Recently, Kingma & Dhariwal (2018)
demonstrated with Glow that generative flows are capable of generating high
quality images. We generalize the 1 x 1 convolutions proposed in Glow to
invertible d x d convolutions, which are more flexible since they operate on
both channel and spatial axes. We propose two methods to produce invertible
convolutions that have receptive fields identical to standard convolutions:
Emerging convolutions are obtained by chaining specific autoregressive
convolutions, and periodic convolutions are decoupled in the frequency domain.
Our experiments show that the flexibility of d x d convolutions significantly
improves the performance of generative flow models on galaxy images, CIFAR10
and ImageNet.Comment: Accepted at International Conference on Machine Learning (ICML) 201
Ordering Dimensions with Nested Dropout Normalizing Flows
The latent space of normalizing flows must be of the same dimensionality as
their output space. This constraint presents a problem if we want to learn
low-dimensional, semantically meaningful representations. Recent work has
provided compact representations by fitting flows constrained to manifolds, but
hasn't defined a density off that manifold. In this work we consider flows with
full support in data space, but with ordered latent variables. Like in PCA, the
leading latent dimensions define a sequence of manifolds that lie close to the
data. We note a trade-off between the flow likelihood and the quality of the
ordering, depending on the parameterization of the flow
Self Normalizing Flows
Efficient gradient computation of the Jacobian determinant term is a core
problem in many machine learning settings, and especially so in the normalizing
flow framework. Most proposed flow models therefore either restrict to a
function class with easy evaluation of the Jacobian determinant, or an
efficient estimator thereof. However, these restrictions limit the performance
of such density models, frequently requiring significant depth to reach desired
performance levels. In this work, we propose Self Normalizing Flows, a flexible
framework for training normalizing flows by replacing expensive terms in the
gradient by learned approximate inverses at each layer. This reduces the
computational complexity of each layer's exact update from
to , allowing for the training of flow architectures which
were otherwise computationally infeasible, while also providing efficient
sampling. We show experimentally that such models are remarkably stable and
optimize to similar data likelihood values as their exact gradient
counterparts, while training more quickly and surpassing the performance of
functionally constrained counterparts