37,221 research outputs found
Free energy of Bayesian Convolutional Neural Network with Skip Connection
Since the success of Residual Network(ResNet), many of architectures of
Convolutional Neural Networks(CNNs) have adopted skip connection. While the
generalization performance of CNN with skip connection has been explained
within the framework of Ensemble Learning, the dependency on the number of
parameters have not been revealed. In this paper, we show that Bayesian free
energy of Convolutional Neural Network both with and without skip connection in
Bayesian learning. The upper bound of free energy of Bayesian CNN with skip
connection does not depend on the oveparametrization and, the generalization
error of Bayesian CNN has similar property.Comment: 16 pages, 4 figure
Specifying Weight Priors in Bayesian Deep Neural Networks with Empirical Bayes
Stochastic variational inference for Bayesian deep neural network (DNN)
requires specifying priors and approximate posterior distributions over neural
network weights. Specifying meaningful weight priors is a challenging problem,
particularly for scaling variational inference to deeper architectures
involving high dimensional weight space. We propose MOdel Priors with Empirical
Bayes using DNN (MOPED) method to choose informed weight priors in Bayesian
neural networks. We formulate a two-stage hierarchical modeling, first find the
maximum likelihood estimates of weights with DNN, and then set the weight
priors using empirical Bayes approach to infer the posterior with variational
inference. We empirically evaluate the proposed approach on real-world tasks
including image classification, video activity recognition and audio
classification with varying complex neural network architectures. We also
evaluate our proposed approach on diabetic retinopathy diagnosis task and
benchmark with the state-of-the-art Bayesian deep learning techniques. We
demonstrate MOPED method enables scalable variational inference and provides
reliable uncertainty quantification.Comment: To be published at AAAI 2020 conferenc
- …