601 research outputs found
Non-Convex Multi-species Hopfield models
In this work we introduce a multi-species generalization of the Hopfield
model for associative memory, where neurons are divided into groups and both
inter-groups and intra-groups pair-wise interactions are considered, with
different intensities. Thus, this system contains two of the main ingredients
of modern Deep neural network architectures: Hebbian interactions to store
patterns of information and multiple layers coding different levels of
correlations. The model is completely solvable in the low-load regime with a
suitable generalization of the Hamilton-Jacobi technique, despite the
Hamiltonian can be a non-definite quadratic form of the magnetizations. The
family of multi-species Hopfield model includes, as special cases, the 3-layers
Restricted Boltzmann Machine (RBM) with Gaussian hidden layer and the
Bidirectional Associative Memory (BAM) model.Comment: This is a pre-print of an article published in J. Stat. Phy
Long Sequence Hopfield Memory
Sequence memory is an essential attribute of natural and artificial
intelligence that enables agents to encode, store, and retrieve complex
sequences of stimuli and actions. Computational models of sequence memory have
been proposed where recurrent Hopfield-like neural networks are trained with
temporally asymmetric Hebbian rules. However, these networks suffer from
limited sequence capacity (maximal length of the stored sequence) due to
interference between the memories. Inspired by recent work on Dense Associative
Memories, we expand the sequence capacity of these models by introducing a
nonlinear interaction term, enhancing separation between the patterns. We
derive novel scaling laws for sequence capacity with respect to network size,
significantly outperforming existing scaling laws for models based on
traditional Hopfield networks, and verify these theoretical results with
numerical simulation. Moreover, we introduce a generalized pseudoinverse rule
to recall sequences of highly correlated patterns. Finally, we extend this
model to store sequences with variable timing between states' transitions and
describe a biologically-plausible implementation, with connections to motor
neuroscience.Comment: NeurIPS 2023 Camera-Ready, 41 page
Free energies of Boltzmann Machines: self-averaging, annealed and replica symmetric approximations in the thermodynamic limit
Restricted Boltzmann machines (RBMs) constitute one of the main models for
machine statistical inference and they are widely employed in Artificial
Intelligence as powerful tools for (deep) learning. However, in contrast with
countless remarkable practical successes, their mathematical formalization has
been largely elusive: from a statistical-mechanics perspective these systems
display the same (random) Gibbs measure of bi-partite spin-glasses, whose
rigorous treatment is notoriously difficult. In this work, beyond providing a
brief review on RBMs from both the learning and the retrieval perspectives, we
aim to contribute to their analytical investigation, by considering two
distinct realizations of their weights (i.e., Boolean and Gaussian) and
studying the properties of their related free energies. More precisely,
focusing on a RBM characterized by digital couplings, we first extend the
Pastur-Shcherbina-Tirozzi method (originally developed for the Hopfield model)
to prove the self-averaging property for the free energy, over its quenched
expectation, in the infinite volume limit, then we explicitly calculate its
simplest approximation, namely its annealed bound. Next, focusing on a RBM
characterized by analogical weights, we extend Guerra's interpolating scheme to
obtain a control of the quenched free-energy under the assumption of replica
symmetry: we get self-consistencies for the order parameters (in full agreement
with the existing Literature) as well as the critical line for ergodicity
breaking that turns out to be the same obtained in AGS theory. As we discuss,
this analogy stems from the slow-noise universality. Finally, glancing beyond
replica symmetry, we analyze the fluctuations of the overlaps for an estimate
of the (slow) noise affecting the retrieval of the signal, and by a stability
analysis we recover the Aizenman-Contucci identities typical of glassy systems.Comment: 21 pages, 1 figur
Optimisation in ‘Self-modelling’ Complex Adaptive Systems
When a dynamical system with multiple point attractors is released from an arbitrary initial condition it will relax into a configuration that locally resolves the constraints or opposing forces between interdependent state variables. However, when there are many conflicting interdependencies between variables, finding a configuration that globally optimises these constraints by this method is unlikely, or may take many attempts. Here we show that a simple distributed mechanism can incrementally alter a dynamical system such that it finds lower energy configurations, more reliably and more quickly. Specifically, when Hebbian learning is applied to the connections of a simple dynamical system undergoing repeated relaxation, the system will develop an associative memory that amplifies a subset of its own attractor states. This modifies the dynamics of the system such that its ability to find configurations that minimise total system energy, and globally resolve conflicts between interdependent variables, is enhanced. Moreover, we show that the system is not merely ‘recalling’ low energy states that have been previously visited but ‘predicting’ their location by generalising over local attractor states that have already been visited. This ‘self-modelling’ framework, i.e. a system that augments its behaviour with an associative memory of its own attractors, helps us better-understand the conditions under which a simple locally-mediated mechanism of self-organisation can promote significantly enhanced global resolution of conflicts between the components of a complex adaptive system. We illustrate this process in random and modular network constraint problems equivalent to graph colouring and distributed task allocation problems
Attractors in fully asymmetric neural networks
The statistical properties of the length of the cycles and of the weights of
the attraction basins in fully asymmetric neural networks (i.e. with completely
uncorrelated synapses) are computed in the framework of the annealed
approximation which we previously introduced for the study of Kauffman
networks. Our results show that this model behaves essentially as a Random Map
possessing a reversal symmetry. Comparison with numerical results suggests that
the approximation could become exact in the infinite size limit.Comment: 23 pages, 6 figures, Latex, to appear on J. Phys.
An associative network with spatially organized connectivity
We investigate the properties of an autoassociative network of
threshold-linear units whose synaptic connectivity is spatially structured and
asymmetric. Since the methods of equilibrium statistical mechanics cannot be
applied to such a network due to the lack of a Hamiltonian, we approach the
problem through a signal-to-noise analysis, that we adapt to spatially
organized networks. The conditions are analyzed for the appearance of stable,
spatially non-uniform profiles of activity with large overlaps with one of the
stored patterns. It is also shown, with simulations and analytic results, that
the storage capacity does not decrease much when the connectivity of the
network becomes short range. In addition, the method used here enables us to
calculate exactly the storage capacity of a randomly connected network with
arbitrary degree of dilution.Comment: 27 pages, 6 figures; Accepted for publication in JSTA
- …