3 research outputs found
Structured Black Box Variational Inference for Latent Time Series Models
Continuous latent time series models are prevalent in Bayesian modeling;
examples include the Kalman filter, dynamic collaborative filtering, or dynamic
topic models. These models often benefit from structured, non mean field
variational approximations that capture correlations between time steps. Black
box variational inference with reparameterization gradients (BBVI) allows us to
explore a rich new class of Bayesian non-conjugate latent time series models;
however, a naive application of BBVI to such structured variational models
would scale quadratically in the number of time steps. We describe a BBVI
algorithm analogous to the forward-backward algorithm which instead scales
linearly in time. It allows us to efficiently sample from the variational
distribution and estimate the gradients of the ELBO. Finally, we show results
on the recently proposed dynamic word embedding model, which was trained using
our method.Comment: 5 pages, 1 figure; presented at the ICML 2017 Time Series Worksho
GP-VAE: Deep Probabilistic Time Series Imputation
Multivariate time series with missing values are common in areas such as
healthcare and finance, and have grown in number and complexity over the years.
This raises the question whether deep learning methodologies can outperform
classical data imputation methods in this domain. However, naive applications
of deep learning fall short in giving reliable confidence estimates and lack
interpretability. We propose a new deep sequential latent variable model for
dimensionality reduction and data imputation. Our modeling assumption is simple
and interpretable: the high dimensional time series has a lower-dimensional
representation which evolves smoothly in time according to a Gaussian process.
The non-linear dimensionality reduction in the presence of missing data is
achieved using a VAE approach with a novel structured variational
approximation. We demonstrate that our approach outperforms several classical
and deep learning-based data imputation methods on high-dimensional data from
the domains of computer vision and healthcare, while additionally improving the
smoothness of the imputations and providing interpretable uncertainty
estimates.Comment: Accepted for publication at the 23rd International Conference on
Artificial Intelligence and Statistics (AISTATS 2020
Advances in Variational Inference
Many modern unsupervised or semi-supervised machine learning algorithms rely
on Bayesian probabilistic models. These models are usually intractable and thus
require approximate inference. Variational inference (VI) lets us approximate a
high-dimensional Bayesian posterior with a simpler variational distribution by
solving an optimization problem. This approach has been successfully used in
various models and large-scale applications. In this review, we give an
overview of recent trends in variational inference. We first introduce standard
mean field variational inference, then review recent advances focusing on the
following aspects: (a) scalable VI, which includes stochastic approximations,
(b) generic VI, which extends the applicability of VI to a large class of
otherwise intractable models, such as non-conjugate models, (c) accurate VI,
which includes variational models beyond the mean field approximation or with
atypical divergences, and (d) amortized VI, which implements the inference over
local latent variables with inference networks. Finally, we provide a summary
of promising future research directions