100 research outputs found
Conditional Sampling of Variational Autoencoders via Iterated Approximate Ancestral Sampling
Conditional sampling of variational autoencoders (VAEs) is needed in various applications, such as missing data imputation, but is computationally intractable. A principled choice for asymptotically exact conditional sampling is Metropolis-within-Gibbs (MWG). However, we observe that the tendency of VAEs to learn a structured latent space, a commonly desired property, can cause the MWG sampler to get âstuckâ far from the target distribution. This paper mitigates the limitations of MWG: we systematically outline the pitfalls in the context of VAEs, propose two original methods that address these pitfalls, and demonstrate an improved performance of the proposed methods on a set of sampling tasks
Bayesian Optimization with Informative Covariance
Bayesian Optimization is a methodology for global optimization of unknown and
expensive objectives. It combines a surrogate Bayesian regression model with an
acquisition function to decide where to evaluate the objective. Typical
regression models are Gaussian processes with stationary covariance functions,
which, however, are unable to express prior input-dependent information, in
particular information about possible locations of the optimum. The ubiquity of
stationary models has led to the common practice of exploiting prior
information via informative mean functions. In this paper, we highlight that
these models can lead to poor performance, especially in high dimensions. We
propose novel informative covariance functions that leverage nonstationarity to
encode preferences for certain regions of the search space and adaptively
promote local exploration during the optimization. We demonstrate that they can
increase the sample efficiency of the optimization in high dimensions, even
under weak prior information
Conditional Sampling of Variational Autoencoders via Iterated Approximate Ancestral Sampling
Conditional sampling of variational autoencoders (VAEs) is needed in various
applications, such as missing data imputation, but is computationally
intractable. A principled choice for asymptotically exact conditional sampling
is Metropolis-within-Gibbs (MWG). However, we observe that the tendency of VAEs
to learn a structured latent space, a commonly desired property, can cause the
MWG sampler to get "stuck" far from the target distribution. This paper
mitigates the limitations of MWG: we systematically outline the pitfalls in the
context of VAEs, propose two original methods that address these pitfalls, and
demonstrate an improved performance of the proposed methods on a set of
sampling tasks
Improving variational autoencoder estimation from incomplete data with mixture variational families
We consider the task of estimating variational autoencoders (VAEs) when the training data is incomplete. We show that missing data increases the complexity of the modelâs posterior distribution over the latent variables compared to the fully-observed case. The increased complexity may adversely affect the fit of the model due to a mismatch between the variational and model posterior distributions. We introduce two strategies based on (i) finite variational-mixture and (ii) imputation-based variational-mixture distributions to address the increased posterior complexity. Through a comprehensive evaluation of the proposed approaches, we show that variational mixtures are effective at improving the accuracy of VAE estimation from incomplete data
Statistical applications of contrastive learning
The likelihood function plays a crucial role in statistical inference and
experimental design. However, it is computationally intractable for several
important classes of statistical models, including energy-based models and
simulator-based models. Contrastive learning is an intuitive and
computationally feasible alternative to likelihood-based learning. We here
first provide an introduction to contrastive learning and then show how we can
use it to derive methods for diverse statistical problems, namely parameter
estimation for energy-based models, Bayesian inference for simulator-based
models, as well as experimental design.Comment: Accepted to Behaviormetrik
Variational Gibbs inference for statistical model estimation from incomplete data
Statistical models are central to machine learning with broad applicability
across a range of downstream tasks. The models are controlled by free
parameters that are typically estimated from data by maximum-likelihood
estimation or approximations thereof. However, when faced with real-world
datasets many of the models run into a critical issue: they are formulated in
terms of fully-observed data, whereas in practice the datasets are plagued with
missing data. The theory of statistical model estimation from incomplete data
is conceptually similar to the estimation of latent-variable models, where
powerful tools such as variational inference (VI) exist. However, in contrast
to standard latent-variable models, parameter estimation with incomplete data
often requires estimating exponentially-many conditional distributions of the
missing variables, hence making standard VI methods intractable. We address
this gap by introducing variational Gibbs inference (VGI), a new
general-purpose method to estimate the parameters of statistical models from
incomplete data. We validate VGI on a set of synthetic and real-world
estimation tasks, estimating important machine learning models such as VAEs and
normalising flows from incomplete data. The proposed method, whilst
general-purpose, achieves competitive or better performance than existing
model-specific estimation methods.Comment: Improved clarity and references. Added algorithms 2-5. Experiment
results remain unchange
- âŠ