7,399 research outputs found
Tensor Monte Carlo: particle methods for the GPU era
Multi-sample, importance-weighted variational autoencoders (IWAE) give
tighter bounds and more accurate uncertainty estimates than variational
autoencoders (VAE) trained with a standard single-sample objective. However,
IWAEs scale poorly: as the latent dimensionality grows, they require
exponentially many samples to retain the benefits of importance weighting.
While sequential Monte-Carlo (SMC) can address this problem, it is
prohibitively slow because the resampling step imposes sequential structure
which cannot be parallelised, and moreover, resampling is non-differentiable
which is problematic when learning approximate posteriors. To address these
issues, we developed tensor Monte-Carlo (TMC) which gives exponentially many
importance samples by separately drawing samples for each of the latent
variables, then averaging over all possible combinations. While the sum
over exponentially many terms might seem to be intractable, in many cases it
can be computed efficiently as a series of tensor inner-products. We show that
TMC is superior to IWAE on a generative model with multiple stochastic layers
trained on the MNIST handwritten digit database, and we show that TMC can be
combined with standard variance reduction techniques
Denoising Criterion for Variational Auto-Encoding Framework
Denoising autoencoders (DAE) are trained to reconstruct their clean inputs
with noise injected at the input level, while variational autoencoders (VAE)
are trained with noise injected in their stochastic hidden layer, with a
regularizer that encourages this noise injection. In this paper, we show that
injecting noise both in input and in the stochastic hidden layer can be
advantageous and we propose a modified variational lower bound as an improved
objective function in this setup. When input is corrupted, then the standard
VAE lower bound involves marginalizing the encoder conditional distribution
over the input noise, which makes the training criterion intractable. Instead,
we propose a modified training criterion which corresponds to a tractable bound
when input is corrupted. Experimentally, we find that the proposed denoising
variational autoencoder (DVAE) yields better average log-likelihood than the
VAE and the importance weighted autoencoder on the MNIST and Frey Face
datasets.Comment: ICLR conference submissio
Gravity-Inspired Graph Autoencoders for Directed Link Prediction
Graph autoencoders (AE) and variational autoencoders (VAE) recently emerged
as powerful node embedding methods. In particular, graph AE and VAE were
successfully leveraged to tackle the challenging link prediction problem,
aiming at figuring out whether some pairs of nodes from a graph are connected
by unobserved edges. However, these models focus on undirected graphs and
therefore ignore the potential direction of the link, which is limiting for
numerous real-life applications. In this paper, we extend the graph AE and VAE
frameworks to address link prediction in directed graphs. We present a new
gravity-inspired decoder scheme that can effectively reconstruct directed
graphs from a node embedding. We empirically evaluate our method on three
different directed link prediction tasks, for which standard graph AE and VAE
perform poorly. We achieve competitive results on three real-world graphs,
outperforming several popular baselines.Comment: ACM International Conference on Information and Knowledge Management
(CIKM 2019
- …