6,527 research outputs found
Deterministic variational inference for robust Bayesian neural networks
Bayesian neural networks (BNNs) hold great promise as a flexible and
principled solution to deal with uncertainty when learning from finite data.
Among approaches to realize probabilistic inference in deep neural networks,
variational Bayes (VB) is theoretically grounded, generally applicable, and
computationally efficient. With wide recognition of potential advantages, why
is it that variational Bayes has seen very limited practical use for BNNs in
real applications? We argue that variational inference in neural networks is
fragile: successful implementations require careful initialization and tuning
of prior variances, as well as controlling the variance of Monte Carlo gradient
estimates. We provide two innovations that aim to turn VB into a robust
inference tool for Bayesian neural networks: first, we introduce a novel
deterministic method to approximate moments in neural networks, eliminating
gradient variance; second, we introduce a hierarchical prior for parameters and
a novel Empirical Bayes procedure for automatically selecting prior variances.
Combining these two innovations, the resulting method is highly efficient and
robust. On the application of heteroscedastic regression we demonstrate good
predictive performance over alternative approaches
Dropout Inference in Bayesian Neural Networks with Alpha-divergences
To obtain uncertainty estimates with real-world Bayesian deep learning
models, practical inference approximations are needed. Dropout variational
inference (VI) for example has been used for machine vision and medical
applications, but VI can severely underestimates model uncertainty.
Alpha-divergences are alternative divergences to VI's KL objective, which are
able to avoid VI's uncertainty underestimation. But these are hard to use in
practice: existing techniques can only use Gaussian approximating
distributions, and require existing models to be changed radically, thus are of
limited use for practitioners. We propose a re-parametrisation of the
alpha-divergence objectives, deriving a simple inference technique which,
together with dropout, can be easily implemented with existing models by simply
changing the loss of the model. We demonstrate improved uncertainty estimates
and accuracy compared to VI in dropout networks. We study our model's epistemic
uncertainty far away from the data using adversarial images, showing that these
can be distinguished from non-adversarial images by examining our model's
uncertainty
Lightweight Probabilistic Deep Networks
Even though probabilistic treatments of neural networks have a long history,
they have not found widespread use in practice. Sampling approaches are often
too slow already for simple networks. The size of the inputs and the depth of
typical CNN architectures in computer vision only compound this problem.
Uncertainty in neural networks has thus been largely ignored in practice,
despite the fact that it may provide important information about the
reliability of predictions and the inner workings of the network. In this
paper, we introduce two lightweight approaches to making supervised learning
with probabilistic deep networks practical: First, we suggest probabilistic
output layers for classification and regression that require only minimal
changes to existing networks. Second, we employ assumed density filtering and
show that activation uncertainties can be propagated in a practical fashion
through the entire network, again with minor changes. Both probabilistic
networks retain the predictive power of the deterministic counterpart, but
yield uncertainties that correlate well with the empirical error induced by
their predictions. Moreover, the robustness to adversarial examples is
significantly increased.Comment: To appear at CVPR 201
- …