4,326 research outputs found
Predictive Entropy Search for Efficient Global Optimization of Black-box Functions
We propose a novel information-theoretic approach for Bayesian optimization
called Predictive Entropy Search (PES). At each iteration, PES selects the next
evaluation point that maximizes the expected information gained with respect to
the global maximum. PES codifies this intractable acquisition function in terms
of the expected reduction in the differential entropy of the predictive
distribution. This reformulation allows PES to obtain approximations that are
both more accurate and efficient than other alternatives such as Entropy Search
(ES). Furthermore, PES can easily perform a fully Bayesian treatment of the
model hyperparameters while ES cannot. We evaluate PES in both synthetic and
real-world applications, including optimization problems in machine learning,
finance, biotechnology, and robotics. We show that the increased accuracy of
PES leads to significant gains in optimization performance
Semiparametric inference in mixture models with predictive recursion marginal likelihood
Predictive recursion is an accurate and computationally efficient algorithm
for nonparametric estimation of mixing densities in mixture models. In
semiparametric mixture models, however, the algorithm fails to account for any
uncertainty in the additional unknown structural parameter. As an alternative
to existing profile likelihood methods, we treat predictive recursion as a
filter approximation to fitting a fully Bayes model, whereby an approximate
marginal likelihood of the structural parameter emerges and can be used for
inference. We call this the predictive recursion marginal likelihood.
Convergence properties of predictive recursion under model mis-specification
also lead to an attractive construction of this new procedure. We show
pointwise convergence of a normalized version of this marginal likelihood
function. Simulations compare the performance of this new marginal likelihood
approach that of existing profile likelihood methods as well as Dirichlet
process mixtures in density estimation. Mixed-effects models and an empirical
Bayes multiple testing application in time series analysis are also considered
Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
We introduce the Kronecker factored online Laplace approximation for
overcoming catastrophic forgetting in neural networks. The method is grounded
in a Bayesian online learning framework, where we recursively approximate the
posterior after every task with a Gaussian, leading to a quadratic penalty on
changes to the weights. The Laplace approximation requires calculating the
Hessian around a mode, which is typically intractable for modern architectures.
In order to make our method scalable, we leverage recent block-diagonal
Kronecker factored approximations to the curvature. Our algorithm achieves over
90% test accuracy across a sequence of 50 instantiations of the permuted MNIST
dataset, substantially outperforming related methods for overcoming
catastrophic forgetting.Comment: 13 pages, 6 figure
- …