5,403 research outputs found
Recognition of partially occluded threat objects using the annealed Hopefield network
Recognition of partially occluded objects has been an important issue to airport security because occlusion causes significant problems in identifying and locating objects during baggage inspection. The neural network approach is suitable for the problems in the sense that the inherent parallelism of neural networks pursues many hypotheses in parallel resulting in high computation rates. Moreover, they provide a greater degree of robustness or fault tolerance than conventional computers. The annealed Hopfield network which is derived from the mean field annealing (MFA) has been developed to find global solutions of a nonlinear system. In the study, it has been proven that the system temperature of MFA is equivalent to the gain of the sigmoid function of a Hopfield network. In our early work, we developed the hybrid Hopfield network (HHN) for fast and reliable matching. However, HHN doesn't guarantee global solutions and yields false matching under heavily occluded conditions because HHN is dependent on initial states by its nature. In this paper, we present the annealed Hopfield network (AHN) for occluded object matching problems. In AHN, the mean field theory is applied to the hybird Hopfield network in order to improve computational complexity of the annealed Hopfield network and provide reliable matching under heavily occluded conditions. AHN is slower than HHN. However, AHN provides near global solutions without initial restrictions and provides less false matching than HHN. In conclusion, a new algorithm based upon a neural network approach was developed to demonstrate the feasibility of the automated inspection of threat objects from x-ray images. The robustness of the algorithm is proved by identifying occluded target objects with large tolerance of their features
A walk in the statistical mechanical formulation of neural networks
Neural networks are nowadays both powerful operational tools (e.g., for
pattern recognition, data mining, error correction codes) and complex
theoretical models on the focus of scientific investigation. As for the
research branch, neural networks are handled and studied by psychologists,
neurobiologists, engineers, mathematicians and theoretical physicists. In
particular, in theoretical physics, the key instrument for the quantitative
analysis of neural networks is statistical mechanics. From this perspective,
here, we first review attractor networks: starting from ferromagnets and
spin-glass models, we discuss the underlying philosophy and we recover the
strand paved by Hopfield, Amit-Gutfreund-Sompolinky. One step forward, we
highlight the structural equivalence between Hopfield networks (modeling
retrieval) and Boltzmann machines (modeling learning), hence realizing a deep
bridge linking two inseparable aspects of biological and robotic spontaneous
cognition. As a sideline, in this walk we derive two alternative (with respect
to the original Hebb proposal) ways to recover the Hebbian paradigm, stemming
from ferromagnets and from spin-glasses, respectively. Further, as these notes
are thought of for an Engineering audience, we highlight also the mappings
between ferromagnets and operational amplifiers and between antiferromagnets
and flip-flops (as neural networks -built by op-amp and flip-flops- are
particular spin-glasses and the latter are indeed combinations of ferromagnets
and antiferromagnets), hoping that such a bridge plays as a concrete
prescription to capture the beauty of robotics from the statistical mechanical
perspective.Comment: Contribute to the proceeding of the conference: NCTA 2014. Contains
12 pages,7 figure
Non-Convex Multi-species Hopfield models
In this work we introduce a multi-species generalization of the Hopfield
model for associative memory, where neurons are divided into groups and both
inter-groups and intra-groups pair-wise interactions are considered, with
different intensities. Thus, this system contains two of the main ingredients
of modern Deep neural network architectures: Hebbian interactions to store
patterns of information and multiple layers coding different levels of
correlations. The model is completely solvable in the low-load regime with a
suitable generalization of the Hamilton-Jacobi technique, despite the
Hamiltonian can be a non-definite quadratic form of the magnetizations. The
family of multi-species Hopfield model includes, as special cases, the 3-layers
Restricted Boltzmann Machine (RBM) with Gaussian hidden layer and the
Bidirectional Associative Memory (BAM) model.Comment: This is a pre-print of an article published in J. Stat. Phy
Meta-stable states in the hierarchical Dyson model drive parallel processing in the hierarchical Hopfield network
In this paper we introduce and investigate the statistical mechanics of
hierarchical neural networks: First, we approach these systems \`a la Mattis,
by thinking at the Dyson model as a single-pattern hierarchical neural network
and we discuss the stability of different retrievable states as predicted by
the related self-consistencies obtained from a mean-field bound and from a
bound that bypasses the mean-field limitation. The latter is worked out by
properly reabsorbing fluctuations of the magnetization related to higher levels
of the hierarchy into effective fields for the lower levels. Remarkably, mixing
Amit's ansatz technique (to select candidate retrievable states) with the
interpolation procedure (to solve for the free energy of these states) we prove
that (due to gauge symmetry) the Dyson model accomplishes both serial and
parallel processing. One step forward, we extend this scenario toward multiple
stored patterns by implementing the Hebb prescription for learning within the
couplings. This results in an Hopfield-like networks constrained on a
hierarchical topology, for which, restricting to the low storage regime (where
the number of patterns grows at most logarithmical with the amount of neurons),
we prove the existence of the thermodynamic limit for the free energy and we
give an explicit expression of its mean field bound and of the related improved
boun
How glassy are neural networks?
In this paper we continue our investigation on the high storage regime of a
neural network with Gaussian patterns. Through an exact mapping between its
partition function and one of a bipartite spin glass (whose parties consist of
Ising and Gaussian spins respectively), we give a complete control of the whole
annealed region. The strategy explored is based on an interpolation between the
bipartite system and two independent spin glasses built respectively by
dichotomic and Gaussian spins: Critical line, behavior of the principal
thermodynamic observables and their fluctuations as well as overlap
fluctuations are obtained and discussed. Then, we move further, extending such
an equivalence beyond the critical line, to explore the broken ergodicity phase
under the assumption of replica symmetry and we show that the quenched free
energy of this (analogical) Hopfield model can be described as a linear
combination of the two quenched spin-glass free energies even in the replica
symmetric framework
Transient Dynamics of Sparsely Connected Hopfield Neural Networks with Arbitrary Degree Distributions
Using probabilistic approach, the transient dynamics of sparsely connected
Hopfield neural networks is studied for arbitrary degree distributions. A
recursive scheme is developed to determine the time evolution of overlap
parameters. As illustrative examples, the explicit calculations of dynamics for
networks with binomial, power-law, and uniform degree distribution are
performed. The results are good agreement with the extensive numerical
simulations. It indicates that with the same average degree, there is a gradual
improvement of network performance with increasing sharpness of its degree
distribution, and the most efficient degree distribution for global storage of
patterns is the delta function.Comment: 11 pages, 5 figures. Any comments are favore
The Relativistic Hopfield network: rigorous results
The relativistic Hopfield model constitutes a generalization of the standard
Hopfield model that is derived by the formal analogy between the
statistical-mechanic framework embedding neural networks and the Lagrangian
mechanics describing a fictitious single-particle motion in the space of the
tuneable parameters of the network itself. In this analogy the cost-function of
the Hopfield model plays as the standard kinetic-energy term and its related
Mattis overlap (naturally bounded by one) plays as the velocity. The
Hamiltonian of the relativisitc model, once Taylor-expanded, results in a
P-spin series with alternate signs: the attractive contributions enhance the
information-storage capabilities of the network, while the repulsive
contributions allow for an easier unlearning of spurious states, conferring
overall more robustness to the system as a whole. Here we do not deepen the
information processing skills of this generalized Hopfield network, rather we
focus on its statistical mechanical foundation. In particular, relying on
Guerra's interpolation techniques, we prove the existence of the infinite
volume limit for the model free-energy and we give its explicit expression in
terms of the Mattis overlaps. By extremizing the free energy over the latter we
get the generalized self-consistent equations for these overlaps, as well as a
picture of criticality that is further corroborated by a fluctuation analysis.
These findings are in full agreement with the available previous results.Comment: 11 pages, 1 figur
Neural Networks retrieving Boolean patterns in a sea of Gaussian ones
Restricted Boltzmann Machines are key tools in Machine Learning and are
described by the energy function of bipartite spin-glasses. From a statistical
mechanical perspective, they share the same Gibbs measure of Hopfield networks
for associative memory. In this equivalence, weights in the former play as
patterns in the latter. As Boltzmann machines usually require real weights to
be trained with gradient descent like methods, while Hopfield networks
typically store binary patterns to be able to retrieve, the investigation of a
mixed Hebbian network, equipped with both real (e.g., Gaussian) and discrete
(e.g., Boolean) patterns naturally arises. We prove that, in the challenging
regime of a high storage of real patterns, where retrieval is forbidden, an
extra load of Boolean patterns can still be retrieved, as long as the ratio
among the overall load and the network size does not exceed a critical
threshold, that turns out to be the same of the standard
Amit-Gutfreund-Sompolinsky theory. Assuming replica symmetry, we study the case
of a low load of Boolean patterns combining the stochastic stability and
Hamilton-Jacobi interpolating techniques. The result can be extended to the
high load by a non rigorous but standard replica computation argument.Comment: 16 pages, 1 figur
Hierarchical neural networks perform both serial and parallel processing
In this work we study a Hebbian neural network, where neurons are arranged
according to a hierarchical architecture such that their couplings scale with
their reciprocal distance. As a full statistical mechanics solution is not yet
available, after a streamlined introduction to the state of the art via that
route, the problem is consistently approached through signal- to-noise
technique and extensive numerical simulations. Focusing on the low-storage
regime, where the amount of stored patterns grows at most logarithmical with
the system size, we prove that these non-mean-field Hopfield-like networks
display a richer phase diagram than their classical counterparts. In
particular, these networks are able to perform serial processing (i.e. retrieve
one pattern at a time through a complete rearrangement of the whole ensemble of
neurons) as well as parallel processing (i.e. retrieve several patterns
simultaneously, delegating the management of diff erent patterns to diverse
communities that build network). The tune between the two regimes is given by
the rate of the coupling decay and by the level of noise affecting the system.
The price to pay for those remarkable capabilities lies in a network's capacity
smaller than the mean field counterpart, thus yielding a new budget principle:
the wider the multitasking capabilities, the lower the network load and
viceversa. This may have important implications in our understanding of
biological complexity
- …