3,828 research outputs found
Bayesian Dark Knowledge
We consider the problem of Bayesian parameter estimation for deep neural
networks, which is important in problem settings where we may have little data,
and/ or where we need accurate posterior predictive densities, e.g., for
applications involving bandits or active learning. One simple approach to this
is to use online Monte Carlo methods, such as SGLD (stochastic gradient
Langevin dynamics). Unfortunately, such a method needs to store many copies of
the parameters (which wastes memory), and needs to make predictions using many
versions of the model (which wastes time).
We describe a method for "distilling" a Monte Carlo approximation to the
posterior predictive density into a more compact form, namely a single deep
neural network. We compare to two very recent approaches to Bayesian neural
networks, namely an approach based on expectation propagation [Hernandez-Lobato
and Adams, 2015] and an approach based on variational Bayes [Blundell et al.,
2015]. Our method performs better than both of these, is much simpler to
implement, and uses less computation at test time.Comment: final version submitted to NIPS 201
A Survey on Bayesian Deep Learning
A comprehensive artificial intelligence system needs to not only perceive the
environment with different `senses' (e.g., seeing and hearing) but also infer
the world's conditional (or even causal) relations and corresponding
uncertainty. The past decade has seen major advances in many perception tasks
such as visual object recognition and speech recognition using deep learning
models. For higher-level inference, however, probabilistic graphical models
with their Bayesian nature are still more powerful and flexible. In recent
years, Bayesian deep learning has emerged as a unified probabilistic framework
to tightly integrate deep learning and Bayesian models. In this general
framework, the perception of text or images using deep learning can boost the
performance of higher-level inference and in turn, the feedback from the
inference process is able to enhance the perception of text or images. This
survey provides a comprehensive introduction to Bayesian deep learning and
reviews its recent applications on recommender systems, topic models, control,
etc. Besides, we also discuss the relationship and differences between Bayesian
deep learning and other related topics such as Bayesian treatment of neural
networks.Comment: To appear in ACM Computing Surveys (CSUR) 202
- …