46 research outputs found
Closing the gap: Exact maximum likelihood training of generative autoencoders using invertible layers
In this work, we provide an exact likelihood alternative to the variational
training of generative autoencoders. We show that VAE-style autoencoders can be
constructed using invertible layers, which offer a tractable exact likelihood
without the need for any regularization terms. This is achieved while leaving
complete freedom in the choice of encoder, decoder and prior architectures,
making our approach a drop-in replacement for the training of existing VAEs and
VAE-style models. We refer to the resulting models as Autoencoders within Flows
(AEF), since the encoder, decoder and prior are defined as individual layers of
an overall invertible architecture. We show that the approach results in
strikingly higher performance than architecturally equivalent VAEs in term of
log-likelihood, sample quality and denoising performance. In a broad sense, the
main ambition of this work is to close the gap between the normalizing flow and
autoencoder literature under the common framework of invertibility and exact
maximum likelihood
Maximum Likelihood Training of Autoencoders
Maximum likelihood training has favorable statistical properties and is
popular for generative modeling, especially with normalizing flows. On the
other hand, generative autoencoders promise to be more efficient than
normalizing flows due to the manifold hypothesis. In this work, we introduce
successful maximum likelihood training of unconstrained autoencoders for the
first time, bringing the two paradigms together. To do so, we identify and
overcome two challenges: Firstly, existing maximum likelihood estimators for
free-form networks are unacceptably slow, relying on iteration schemes whose
cost scales linearly with latent dimension. We introduce an improved estimator
which eliminates iteration, resulting in constant cost (roughly double the
runtime per batch of a vanilla autoencoder). Secondly, we demonstrate that
naively applying maximum likelihood to autoencoders can lead to divergent
solutions and use this insight to motivate a stable maximum likelihood training
objective. We perform extensive experiments on toy, tabular and image data,
demonstrating the competitive performance of the resulting model. We call our
model the maximum likelihood autoencoder (MLAE)
Trumpets: Injective Flows for Inference and Inverse Problems
We propose injective generative models called Trumpets that generalize
invertible normalizing flows. The proposed generators progressively increase
dimension from a low-dimensional latent space. We demonstrate that Trumpets can
be trained orders of magnitudes faster than standard flows while yielding
samples of comparable or better quality. They retain many of the advantages of
the standard flows such as training based on maximum likelihood and a fast,
exact inverse of the generator. Since Trumpets are injective and have fast
inverses, they can be effectively used for downstream Bayesian inference. To
wit, we use Trumpet priors for maximum a posteriori estimation in the context
of image reconstruction from compressive measurements, outperforming
competitive baselines in terms of reconstruction quality and speed. We then
propose an efficient method for posterior characterization and uncertainty
quantification with Trumpets by taking advantage of the low-dimensional latent
space.Comment: 16 page