251 research outputs found
Backpropagation at the Infinitesimal Inference Limit of Energy-Based Models: Unifying Predictive Coding, Equilibrium Propagation, and Contrastive Hebbian Learning
How the brain performs credit assignment is a fundamental unsolved problem in
neuroscience. Many `biologically plausible' algorithms have been proposed,
which compute gradients that approximate those computed by backpropagation
(BP), and which operate in ways that more closely satisfy the constraints
imposed by neural circuitry. Many such algorithms utilize the framework of
energy-based models (EBMs), in which all free variables in the model are
optimized to minimize a global energy function. However, in the literature,
these algorithms exist in isolation and no unified theory exists linking them
together. Here, we provide a comprehensive theory of the conditions under which
EBMs can approximate BP, which lets us unify many of the BP approximation
results in the literature (namely, predictive coding, equilibrium propagation,
and contrastive Hebbian learning) and demonstrate that their approximation to
BP arises from a simple and general mathematical property of EBMs at free-phase
equilibrium. This property can then be exploited in different ways with
different energy functions, and these specific choices yield a family of
BP-approximating algorithms, which both includes the known results in the
literature and can be used to derive new ones.Comment: 31/05/22 initial upload; 22/06/22 change corresponding author;
03/08/22 revision
Convolutional Neural Generative Coding: Scaling Predictive Coding to Natural Images
In this work, we develop convolutional neural generative coding (Conv-NGC), a
generalization of predictive coding to the case of
convolution/deconvolution-based computation. Specifically, we concretely
implement a flexible neurobiologically-motivated algorithm that progressively
refines latent state maps in order to dynamically form a more accurate internal
representation/reconstruction model of natural images. The performance of the
resulting sensory processing system is evaluated on several benchmark datasets
such as Color-MNIST, CIFAR-10, and Street House View Numbers (SVHN). We study
the effectiveness of our brain-inspired neural system on the tasks of
reconstruction and image denoising and find that it is competitive with
convolutional auto-encoding systems trained by backpropagation of errors and
notably outperforms them with respect to out-of-distribution reconstruction
(including on the full 90k CINIC-10 test set)
Contrastive Learning for Lifted Networks
In this work we address supervised learning of neural networks via lifted
network formulations. Lifted networks are interesting because they allow
training on massively parallel hardware and assign energy models to
discriminatively trained neural networks. We demonstrate that the training
methods for lifted networks proposed in the literature have significant
limitations and show how to use a contrastive loss to address those
limitations. We demonstrate that this contrastive training approximates
back-propagation in theory and in practice and that it is superior to the
training objective regularly used for lifted networks.Comment: 9 pages, BMVC 201
Predictive Coding: a Theoretical and Experimental Review
Predictive coding offers a potentially unifying account of cortical function
-- postulating that the core function of the brain is to minimize prediction
errors with respect to a generative model of the world. The theory is closely
related to the Bayesian brain framework and, over the last two decades, has
gained substantial influence in the fields of theoretical and cognitive
neuroscience. A large body of research has arisen based on both empirically
testing improved and extended theoretical and mathematical models of predictive
coding, as well as in evaluating their potential biological plausibility for
implementation in the brain and the concrete neurophysiological and
psychological predictions made by the theory. Despite this enduring popularity,
however, no comprehensive review of predictive coding theory, and especially of
recent developments in this field, exists. Here, we provide a comprehensive
review both of the core mathematical structure and logic of predictive coding,
thus complementing recent tutorials in the literature. We also review a wide
range of classic and recent work within the framework, ranging from the
neurobiologically realistic microcircuits that could implement predictive
coding, to the close relationship between predictive coding and the widely-used
backpropagation of error algorithm, as well as surveying the close
relationships between predictive coding and modern machine learning techniques.Comment: 27/07/21 initial upload; 14/01/22 maths fix; 05/07/22 maths fix;
12/07/22 text fixe
- …