446 research outputs found
Gaussian mean field regularizes by limiting learned information
Variational inference with a factorized Gaussian posterior estimate is a widely-used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information between learned parameters and the data through noise. We quantify a maximum capacity when the posterior variance is either fixed or learned and connect it to generalization error, even when the KL-divergence in the objective is scaled by a constant. Our experiments suggest that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks
The Asymptotic Performance of Linear Echo State Neural Networks
In this article, a study of the mean-square error (MSE) performance of linear
echo-state neural networks is performed, both for training and testing tasks.
Considering the realistic setting of noise present at the network nodes, we
derive deterministic equivalents for the aforementioned MSE in the limit where
the number of input data and network size both grow large. Specializing
then the network connectivity matrix to specific random settings, we further
obtain simple formulas that provide new insights on the performance of such
networks
- …