1,715 research outputs found
PAC-Bayesian Theory Meets Bayesian Inference
We exhibit a strong link between frequentist PAC-Bayesian risk bounds and the
Bayesian marginal likelihood. That is, for the negative log-likelihood loss
function, we show that the minimization of PAC-Bayesian generalization risk
bounds maximizes the Bayesian marginal likelihood. This provides an alternative
explanation to the Bayesian Occam's razor criteria, under the assumption that
the data is generated by an i.i.d distribution. Moreover, as the negative
log-likelihood is an unbounded loss function, we motivate and propose a
PAC-Bayesian theorem tailored for the sub-gamma loss family, and we show that
our approach is sound on classical Bayesian linear regression tasks.Comment: Published at NIPS 2015
(http://papers.nips.cc/paper/6569-pac-bayesian-theory-meets-bayesian-inference
The fine print on tempered posteriors
We conduct a detailed investigation of tempered posteriors and uncover a
number of crucial and previously undiscussed points. Contrary to previous
results, we first show that for realistic models and datasets and the tightly
controlled case of the Laplace approximation to the posterior, stochasticity
does not in general improve test accuracy. The coldest temperature is often
optimal. One might think that Bayesian models with some stochasticity can at
least obtain improvements in terms of calibration. However, we show empirically
that when gains are obtained this comes at the cost of degradation in test
accuracy. We then discuss how targeting Frequentist metrics using Bayesian
models provides a simple explanation of the need for a temperature parameter
in the optimization objective. Contrary to prior works, we finally
show through a PAC-Bayesian analysis that the temperature cannot be
seen as simply fixing a misspecified prior or likelihood
- …