168 research outputs found
Diffusion Variational Autoencoders
A standard Variational Autoencoder, with a Euclidean latent space, is
structurally incapable of capturing topological properties of certain datasets.
To remove topological obstructions, we introduce Diffusion Variational
Autoencoders with arbitrary manifolds as a latent space. A Diffusion
Variational Autoencoder uses transition kernels of Brownian motion on the
manifold. In particular, it uses properties of the Brownian motion to implement
the reparametrization trick and fast approximations to the KL divergence. We
show that the Diffusion Variational Autoencoder is capable of capturing
topological properties of synthetic datasets. Additionally, we train MNIST on
spheres, tori, projective spaces, SO(3), and a torus embedded in R3. Although a
natural dataset like MNIST does not have latent variables with a clear-cut
topological structure, training it on a manifold can still highlight
topological and geometrical properties.Comment: 10 pages, 8 figures Added an appendix with derivation of asymptotic
expansion of KL divergence for heat kernel on arbitrary Riemannian manifolds,
and an appendix with new experiments on binarized MNIST. Added a previously
missing factor in the asymptotic expansion of the heat kernel and corrected a
coefficient in asymptotic expansion KL divergence; further minor edit
The Riemannian Geometry of Deep Generative Models
Deep generative models learn a mapping from a low dimensional latent space to
a high-dimensional data space. Under certain regularity conditions, these
models parameterize nonlinear manifolds in the data space. In this paper, we
investigate the Riemannian geometry of these generated manifolds. First, we
develop efficient algorithms for computing geodesic curves, which provide an
intrinsic notion of distance between points on the manifold. Second, we develop
an algorithm for parallel translation of a tangent vector along a path on the
manifold. We show how parallel translation can be used to generate analogies,
i.e., to transport a change in one data point into a semantically similar
change of another data point. Our experiments on real image data show that the
manifolds learned by deep generative models, while nonlinear, are surprisingly
close to zero curvature. The practical implication is that linear paths in the
latent space closely approximate geodesics on the generated manifold. However,
further investigation into this phenomenon is warranted, to identify if there
are other architectures or datasets where curvature plays a more prominent
role. We believe that exploring the Riemannian geometry of deep generative
models, using the tools developed in this paper, will be an important step in
understanding the high-dimensional, nonlinear spaces these models learn.Comment: 9 page
Continuous Hierarchical Representations with Poincar\'e Variational Auto-Encoders
The variational auto-encoder (VAE) is a popular method for learning a
generative model and embeddings of the data. Many real datasets are
hierarchically structured. However, traditional VAEs map data in a Euclidean
latent space which cannot efficiently embed tree-like structures. Hyperbolic
spaces with negative curvature can. We therefore endow VAEs with a Poincar\'e
ball model of hyperbolic geometry as a latent space and rigorously derive the
necessary methods to work with two main Gaussian generalisations on that space.
We empirically show better generalisation to unseen data than the Euclidean
counterpart, and can qualitatively and quantitatively better recover
hierarchical structures.Comment: Advances in Neural Information Processing System
Adversarial robustness of VAEs through the lens of local geometry
In an unsupervised attack on variational autoencoders (VAEs), an adversary
finds a small perturbation in an input sample that significantly changes its
latent space encoding, thereby compromising the reconstruction for a fixed
decoder. A known reason for such vulnerability is the distortions in the latent
space resulting from a mismatch between approximated latent posterior and a
prior distribution. Consequently, a slight change in an input sample can move
its encoding to a low/zero density region in the latent space resulting in an
unconstrained generation. This paper demonstrates that an optimal way for an
adversary to attack VAEs is to exploit a directional bias of a stochastic
pullback metric tensor induced by the encoder and decoder networks. The
pullback metric tensor of an encoder measures the change in infinitesimal
latent volume from an input to a latent space. Thus, it can be viewed as a lens
to analyse the effect of input perturbations leading to latent space
distortions. We propose robustness evaluation scores using the eigenspectrum of
a pullback metric tensor. Moreover, we empirically show that the scores
correlate with the robustness parameter of the VAE. Since
increasing also degrades reconstruction quality, we demonstrate a
simple alternative using \textit{mixup} training to fill the empty regions in
the latent space, thus improving robustness with improved reconstruction.Comment: International Conference on Artificial Intelligence and Statistics
(AISTATS) 202
Representation Learning via Manifold Flattening and Reconstruction
This work proposes an algorithm for explicitly constructing a pair of neural
networks that linearize and reconstruct an embedded submanifold, from finite
samples of this manifold. Our such-generated neural networks, called Flattening
Networks (FlatNet), are theoretically interpretable, computationally feasible
at scale, and generalize well to test data, a balance not typically found in
manifold-based learning methods. We present empirical results and comparisons
to other models on synthetic high-dimensional manifold data and 2D image data.
Our code is publicly available.Comment: 44 pages, 19 figure
- …