2 research outputs found
Associative content-addressable networks with exponentially many robust stable states
The brain must robustly store a large number of memories, corresponding to
the many events encountered over a lifetime. However, the number of memory
states in existing neural network models either grows weakly with network size
or recall fails catastrophically with vanishingly little noise. We construct an
associative content-addressable memory with exponentially many stable states
and robust error-correction. The network possesses expander graph connectivity
on a restricted Boltzmann machine architecture. The expansion property allows
simple neural network dynamics to perform at par with modern error-correcting
codes. Appropriate networks can be constructed with sparse random connections,
glomerular nodes, and associative learning using low dynamic-range weights.
Thus, sparse quasi-random structures---characteristic of important
error-correcting codes---may provide for high-performance computation in
artificial neural networks and the brain.Comment: 42 pages, 8 figure
Convolutional Bipartite Attractor Networks
In human perception and cognition, a fundamental operation that brains
perform is interpretation: constructing coherent neural states from noisy,
incomplete, and intrinsically ambiguous evidence. The problem of interpretation
is well matched to an early and often overlooked architecture, the attractor
network---a recurrent neural net that performs constraint satisfaction,
imputation of missing features, and clean up of noisy data via energy
minimization dynamics. We revisit attractor nets in light of modern deep
learning methods and propose a convolutional bipartite architecture with a
novel training loss, activation function, and connectivity constraints. We
tackle larger problems than have been previously explored with attractor nets
and demonstrate their potential for image completion and super-resolution. We
argue that this architecture is better motivated than ever-deeper feedforward
models and is a viable alternative to more costly sampling-based generative
methods on a range of supervised and unsupervised tasks