We study the representation of static patterns and temporal sequences in neural networks with signal delays and a stochastic parallel dynamics. For a wide class of delay distributions, the asymptotic network behavior can be described by a generalized Gibbs distribution, generated by a novel Lyapunov functional for the determination dynamics. We extend techniques of equilibrium statistical mechanics so as to deal with time-dependent phenomena, derive analytic results for both retrieval quality and storage capacity, and compare them with numerical simulations

A. C. C. Coolen

A. V. M. Herz

C. M. Marcus

D. J. Amit

D. Kleinfeld

D. O. Hebb

H. Sompolinsky

J. F. Fontanari

J. J. Hopfield

J. L. van Hemmen

M. Kerszberg

P. Peretto

W. A. Little

Z. Li

English

Herz, Andreas V. M.

Li, Z.

Hemmen, Jan Leo van

Universität München: Elektronischen Publikationen

VOLUME 66, NUMBER 10 PHYSICAL REVIEW LETTERS 11 MARcH 1991
Statistical Mechanics of Temporal Association an Neural Networks with Transmission Delays
A. V. M. Herz, ' Z. Li, and J. L. van Hemmen
"iDivision of Chemistry, 139-74, California Institute of Technology, Pasadena, California 91I25
School of Natural Sciences, Institute for Advanced Study, Princeton, New Jersey 08540' Physik-Department der Technische Universitat Mu'nchen, D-8046 Garching, Germany
(Received 4 September 1990)
We study the representation of static patterns and temporal sequences in neural networks with signal
delays and a stochastic parallel dynamics. For a wide class of delay distributions, the asymptotic net-
work behavior can be described by a generalized Gibbs distribution, generated by a novel Lyapunov
functional for the deterministic dynamics. We extend techniques of equilibrium statistical mechanics so
as to deal with time-dependent phenomena, derive analytic results for both retrieval quality and storage
capacity, and compare them with numerical simulations.
PACS numbers: 87.10.+e, 64.60.Cn, 89.70.+c
Associative reconstruction of noisy data in a neural
network can be accomplished by endowing a pattern to
be memorized with a basin of attraction for the system's
retrieval dynamics. ' If the dynamics is governed by a
Lyapunov function, a simple intuitive understanding of
the global computation becomes possible: The system
performs a downhill motion in an energy landscape
created by the stored information.
For networks with static patterns, there is a Lyapunov
function if the interactions between single neurons are
instantaneous and mediated by symmetric couplings.
Methods of equilibrium statistical mechanics may then
be applied and permit a quantitative analysis of the
network's performance in terms of the retrieval quality
and storage capacity. The existence of a Lyapunov
function is thus of great conceptual as well as technical
importance.
In general, external inputs to a neural network are not
limited to static memories but provide information in
both space and time. To code the temporal aspects of se-
quences of patterns to be learned, additional asymmetric
couplings with transmission delays may be introduced.
This approach can be generalized to networks with a
broad distribution of signal delays where Hebb's neuro-
physiological principle for learning naturally leads to a
joint representation of spatial and temporal informa-
tion. However, no description in terms of a Lyapunov
function has been given so far.
In this Letter such a description is developed for a cer-
tain class of networks with transmission delays. We
present a Lyapunov functional for the deterministic
parallel dynamics, generalize the formalism of equilibri-
um statistical mechanics so as to deal with thermal noise
in systems with delayed interactions, and, hence, make
the domain of time-dependent phenomena accessible to
powerful free-energy techniques.
We follow Refs. 1-3 and model single neurons by Is-
ing spins 5;, 1 ~i ~ N. They represent a firing state for
5; =+1 and a quiescent one for 5; = —1. The neurons
are connected by synapses with modifiable efficacies
J;, (r). Here r denotes a fixed time delay for the infor-
mation transport from j to i. We focus on a solitonlike
propagation of neural signals, characteristic for the (ax-
onal) transmission of action potentials, and consider a
model where each pair of neurons is linked by several ax-
ons with delays 0~ ~~ r,„. External stimuli are fed
into the system via two-state receptors cr; = + 1. The lo-
cal fields (postsynaptic potentials) are then given by
N max
where the parameter y measures the system's input sen-
sitivity. Other transport mechanisms and network archi-
tectures have been studied elsewhere; see also Refs. 4,
5, 8, and 9.
We concentrate on synchronous (parallel) dynamics
and set the basic time step equal to unity. Consequently,
the signal delays take non-negative integer values. The
inhuence of synaptic noise may be described by a sto-
chastic Glauber dynamics' at finite inverse temperature
p =T '. In that case,
Prob[S;(t+1) = ~ 1] = —,' [1 ~ tanh[ph;(t)]j, (2)
which reduces in the deterministic limit, p ~, to
S;(t+1)=sgn[h;(t)]. The retrieval dynamics can be
studied in the bulk limit N ~ if the number and
length of the stored sequences remain finite.
In the present Letter, we surpass the limits of this kind
of analysis and show that an approach in terms of statist-
ical mechanics is feasible for networks whose couplings
exhibit an "extended synaptic symmetry, " ' '
(3)
for some (fixed) integer D ) 2. Links with r )D —1
are assumed to vanish, i.e., r,.„~D —2.
The condition (3) can be met by simple training algo-
rithms for the synaptic e%cacies. As an example, we
1370 1991 The American Physical Society
VOLUME 66, NUMBER 10 PHYSICAL REVIEW LETTERS 11 MARcH 1991
AH(t) = —Q [S;(t)—S;(t—D)]h;(t —1) . (6)
Deterministic dynamics, i.e., S;(t) =sgn[h;(t —1)], im-
plies that the right-hand side will only take nonpositive
values. For finite N, H is bounded from below and hH
has to vanish for t ~. This can be realized only if the
system settles into a state with S; (t) =S;(t —D) for all i
We have thus exposed two important facts: (a) The
retrieval dynamics of a certain class of networks with de-
layed interactions is governed by a Lyapunov functional,
and (b) the system relaxes to static states or limit cycles
with S;(t) =S; (t —D )—oscillatory solutions with the
same period as that of the taught sequences (or a period
equal to an integer fraction of D). It remains to be
shown that such a limit cycle of the retrieval dynamics
indeed resembles a stored sequence for appropriate ini-
tial conditions.
In order to prove the above assertion, we proceed in
three steps. First, we demonstrate that our task concern-
ing temporal associations with a cyclic structure can be
mapped onto a symmetric network without signal delays.
present a linear Hebbian rule. During a learning session,
a coupling strength J;J(i) is altered after each time step
by an amount proportional to the product of SJ(t —i)
and S; (t+1)—due to the delay i in (1) and the discrete
dynamics (2) it takes i+1 units of time until neuron j
inAuences the state of neuron i through that specific
synapse. After P learning sessions, labeled by p and
each of duration D„, we thus obtain1'
J;,(i) =e(i) g—g S;(t„+1)S,(t„—i). (4)
Np 1' 1
The a priori weights e(i) are non-negative, normalized
such that p, -'*De(i) =1, and may be used to define vari-
ous network architectures. For a "clamped" learning
scenario with y= 1 in (1), the system evolves strictly ac-
cording to the external inputs, S;(t„)=o;(t„—1). Equa-
tion (3) holds once we teach cyclic associations with a
common period D to networks whose weight distributions
obey e(i) =e(D —2 —i)
Returning to the general case, given by (3), we now
establish that there is a Lyapunov functional for the
noiseless retrieval dynamics. We focus on a free evolu-
tion of the system, i.e., we set y=0 in (1), generalize the
results of Ref. 12, and take
1
N D 1 rmax
H(t)=- ——g g g J;,(i)S, (t —a)ij 1a Or 0
xS (t —(a+i+1)(modD)) .
(5)
The functional H depends on all network states between
t+1 —D and t so that solutions with constant H need not
be static fixed points of the retrieval dynamics. The
difference AH(t) =H(t) —H(t —1) is given by
Second, we apply equilibrium statistical mechanics to
study such "equivalent systems. " Third, we focus on a
special case, the Hebbian learning rule (4), and derive
analytic results for the retrieval quality and storage
capacity.
D-periodic oscillatory solutions of the retrieval dynam-
ics can be interpreted as static states in a "D-plicated"
system. Let us therefore consider such a fictitious sys-
tem with D columns and N rows of ~ 1 neurons whose
activities will be denoted by S;,. A network state will be
written 8 =(AO, A~, . . . , Ao-~) with A, =—tS;„1~i
~ N}. To reproduce the parallel dynamics of the origi-
nal system, neurons S;, with a =t (modD) are updated at
time t. The time evolution of the new network therefore
has a pseudosequential characteristic: synchronous
within single columns and sequentially ordered with
respect to these columns. Accordingly, the neural activi-
ties at time t are given by S;,(t) —=S;(a+n, ) for a
~ t(modD) and S;,(t) =S;(a+n, D) for —a & t
(modD), where n, is defined through t= n, +t(m—odD).
Because of (3), we may construct symmetric efficacies
JJ"=J~ for the new system by putting JJ'=0 and
J~j~=JJ((b —a —1)(modD)) for aWb,
so that there is a well-defined Hamiltonian,
H = ——g g Jt'gS;, Sjb,1
N D —1
(g)2ij 1ab 0
equal to that of a Hopfield network of size ND.
An evaluation of {8) in terms of the former state vari-
ables reveals that it is identical with the Lyapunov func-
tional (5). We may once again calculate AH(t) and ar-
rive at a formula corresponding to (6). The interpreta-
tion, however, is changed: a limit cycle of period D in
the original network corresponds to a fixed point of the
new system of size ND. %"e have thus shown that the
deterministic parallel dynamics of a delay network with
extended symmetry can be understood in terms of a
downhill motion in the energy landscape of its equivalent
system.
If, in a second step, we now s~itch to a stochastic
Glauber dynamics (2), the important question arises
whether H also determines the invariant distribution p of
the network. This need not be true since the column-
wise updating procedure of the equivalent system differs
from both the Little' and the Hopfield model. It turns
out, however, that there is indeed an equilibrium distri-
bution i la Gibbs, "
p(A) =Z ' exp[ PH(A)] . —
Here Z—:Tr~ exp[ PH(A)] denotes the pa—rtition func-
tion. The exact proof is elaborate since due to the non-
Markovian structure of the dynamics (1) and (2), the
principle of detailed balance does not apply to single-step
transition probabilities.
In passing we note that for D=2 there are only links
1371
VOLUME 66, NUMBER 10 PHYSICAL REVIEW LETTERS 11 MARCH 1991
with zero delay. By (3) we have J;&(0) =Jz, (0), i.e., we
are dealing with a symmetric Little model. We may in-
troduce a reduced probability distribution p for this spe-
cial case, P(A() =Tr~, p(ApA(), and obtain P(A()
=Z 'exp[ pH—(A()] with
say s, cycles of the original system (corresponding to sD
static patterns for the equivalent model of size ND) and
assume that the network is in a state highly correlated
with these spatiotemporal objects. The remaining, ex-
tensively many cycles are described as a noise term. We
consider retrieval solutions and make the ansatz
N N
H= —P— ' g ln 2cosh P g JJS, (10) m~g =N'—ggP;SP, =m "8,p,
qgP
=N—'g S;,Sg =B,g [Sp (1 —q ) +q],We thus have recovered both the eAective Hamiltonian
of the Little model as derived by Peretto' and the
duplicated-system (D =2) technique of Ref. 13.
Finally, we finish our argument by turning to quanti-
tative results. We focus on synaptic couplings generated
by the Hebbian rule (4) and investigate the case where
each of the P learning sessions corresponds to teaching a
(different) cycle of D patterns [y,~; I (i ~ N], 1 (p
~D, i.e., each pattern lasts for a single time step,
o; (t„)= y,",„(,dD). In accordance with (4) and (8),
there are PD patterns [g,".,'; 1(i (N, 0(a(D —I]
for the equivalent network, 1 ~ p ~ P, 1 ~ e ~ D, given
by g;, =y;, and (ra 4i(a —a),(modD). In what follows, wepo p pa po
work with unbiased random patterns where y,~ =+ 1
with equal probability, and study our network at a finite
storage level a =lim)v (P/N) )0.
As in the replica-symmetric theory of Amit, Gut-
freund, and Sompolinsky we single out finitely many,
for the relevant order parameters of the n-fold replicated
network, 1 ~ p, o ~ n. By applying the techniques of
Refs. 3 and 14, we arrive at the fixed-point equations
(13)
(i4)q= tanh, m' + ar z,
PI, (( )l'r=q-
k=( [1 —p(1 —q)kl, (8)]
(is)
The double angular brackets represent an average with
respect to both the condensed cycles and the normalized
Gaussian random variable z. The kk(C) are eigenvalues
of the matrix 8 with A, (, =e((b —a——1)(modD)). At
the saddle points given by (13)-(15),the free energy is
qpx, (e)
i —p(i —q)x, (c)
+ —aP'(q —l)r+ 1n2cosh P, gm "0 +der z, ) .2
—Pf(P) = —~P(m') ——aP in[1 —P(1 —q)Xk (8)]—1
2
I(@)
I(Hopfield)
Da, (C)
da, (Hopfield) '
where d denotes the number of diAerent delay lines be-
tween each pair of neurons, 1 ~ d ~ D —I. A numerical
The preceding equations closely resemble their counter-
parts for the Hopfield (or Little) model and become
identical to them for a "maximally uniform" distribu-
tion, (,q = (D —1) (1 —6,b), in the limit D ~. For
general period D and weight distribution 8, there is at
zero temperature a first-order phase transition between
the retrieval state and a spin-glass phase as in the
Hopfield case. The critical storage level e, and the cor-
responding overlap m, are difI'erent, though.
It should be noted that each cycle consists of D in-
dependent patterns so that the storage capacity for single
patterns is a, =Da, . During the recognition process,
however, each of them will trigger the cycle it belongs to
and cannot be retrieved as a static pattern.
As a second measure of the performance, we include
the information content I~(8), measured per synapse
and relative to that of the Hopfield model,
D
0.100
0.110
0.116
0.120
m,
0.93
0.95
0.96
0.96
1.45
1.20
1.12
1.09
The above findings agree well with estimates from a
finite-size analysis (N(3000) of data from numerical
simulations as shown by two examples. For D=3, we
have found a, =0.120%0.015, and for D =4, a, =0.125
L- 0.015.
Our results demonstrate that the storage capacity for
temporal associations is comparable to that for static
memories. As an example, we take D =2, i.e., the Little
model. In the limit of large N, we see that 0.100N two-
cycles of the form y,"~ —y,". 2 may be recalled as compared
to 0.138N static patterns;' this leads to a 1.45-fold in-
t solution of the saddle-point equations at T=O leads to
the following results for networks with a "maximally
uniform" distribution:
1372
VOLUME 66, NUMBER 10 PHYSICAL REVIEW LETTERS 11 MARcH 1991
crease of the information content per synapse. At the
same time, the retrieval overlap drops slightly from 0.97
to 0.93. The influence of the weight distribution on the
network behavior may be demonstrated by some choices
of e(z) for D =4:
0 1 2 3 a, IR
e(z)
e(r)
e(z)
1
3
I
2
0I
0I
0 0
0.116 0.96
0.100 0.93
0.050 0.93
1.12
1.45
1.45
The uniform distribution leads to the largest a, but
smallest Ig. The other two networks have the same
value of IR as the (unique) D =2 system due to the par-
ticular structure of their eigenvalue spectrum. Further-
more, one obtains IR =1.45 independently of D for all
networks with a "minimal connectivity" where only one
synapse links two neurons (D even). Simulation data
show once again slightly higher values of e,—possibly
indicating eAects of replica-symmetry breaking as in the
Hopfield model.
In summary, we have introduced a Lyapunov func-
tional for a wide class of neural networks with signal de-
lays and have generalized equilibrium statistical
mechanics so as to analyze the network behavior under a
stochastic dynamics. Quantitative results have been ob-
tained for a Hebbian learning rule. They prove that an
extensive number of temporal associations can be stored
as spatiotemporal attractors for the retrieval dynamics.
Even at the critical storage level, retrieved sequences
contain less than 3.5% errors for all studied architec-
tures. Our results indicate that dynamical systems with
delayed interactions can be programmed in a very
efFicient manner to perform associative computations in
the space-time domain.
The current approach provides a new framework for
the study of temporal associations in delayed networks.
Though limited to networks whose couplings exhibit an
"extended symmetry, " it may be generalized in various
directions. To mention a few examples, first, more so-
phisticated learning prescriptions like a spatiotemporal
version of the pseudoinverse rule can be investigated.
Second, one may wish to switch to continuous state vari-
ables and study temporal associations in "iterated-map
networks. "' It turns out that a generalization of the
Lyapunov functional (5) covers that case as well, ' al-
lowing for a direct comparison of theoretical predictions
with experimental results from hardware implementa-
tions. Finally, one could try to develop a Lyapunov func-
tional for a continuous-time dynamics including signal
delays which seems to be rather significant for applica-
tions as well as for the general theory of functional
differential equations and dynamical systems.
We thank J. J. Hopfield and R. Kiihn for many stimu-
lating discussions. Both of them as well as J. Buhmann
and D. MacKay helped us with constructive comments
on the manuscript. A.V.M.H. acknowledges support
from the Studienstiftung des Deutschen Volkes. Z.L. is
supported in part through a grant from the Seaver Insti-
tute.
'W. A. Little, Math. Biosci. 19, 101 (1974).
2J. J. Hopfield, Proc. Natl. Acad. Sci. U.S.A. 79, 2554
(1982).
D. J. Amit, H. Gutfreund, and H. Sompolinsky, Phys. Rev.
A 32, 1007 (1985); Ann. Phys. (N.Y.) 173, 30 (1987).
4D. Kleinfeld, Proc. Natl. Acad. Sci. U.S.A. 83, 9469
(1986).
~H. Sompolinsky and I. Kanter, Phys. Rev. Lett. 57, 2861
(1986).
6D. O. Hebb, The Organization of Behavior (Wiley, New
York, 1949), p. 62.
7A. V. M. Herz, B. Sulzer, R. Kuhn, and J. L. van Hemmen,
Europhys. Lett. 7, 663 (1988); Biol. Cybern. 60, 457 (1989).
8A. C. C. Coolen and C. C. A. M. Gielen, Europhys. Lett. 7,
281 (1988).
9M. Kerszberg and A. Zippelius, Phys. Scr. T33, 54 (1990).
' oP. Peret to, Biol. Cybern. 50, 51 (1984).
' 'A. V. M. Herz, Ph. D. thesis, Heidelberg, 1990 (unpub-
lished); Z. Li and A. V. M. Herz, in IVeural iVetworks,
Proceedings of the Eleventh Sitges Conference, Lecture Notes
in Physics Vol. 368, edited by L. Garrido (Springer-Verlag,
Berlin, 1990), pp. 287-302.
'zZ. Li (unpublished).
'3J. L. van Hemmen, Phys. Rev. A 34, 3435 (1986).
'4J. L. van Hemmen and R. Kuhn, Phys. Rev. Lett. 57, 913
(1986); J. L. van Hemmen and V. A. Zagrebnov, J. Phys. A
20, 3989 (1987).
'5J. F. Fontanari and R. Koberle, Phys. Rev. A 36, 2475
(1987),
' C. M. Marcus and R. M. Westervelt, Phys. Rev. A 42,
2410 (1990).
'7A. V. M. Herz (to be published).
1373


Statistical mechanics of temporal association in neural networks with transmission delays

Open Access LMU

VOLUME 66, NUMBER 10 PHYSICAL REVIEW LETTERS 11 MARcH 1991Statistical Mechanics of Temporal Association an Neural Networks with Transmission DelaysA. V. M. Herz, ' Z. Li, and J. L. van Hemmen"iDivision of Chemistry, 139-74, California Institute of Technology, Pasadena, California 91I25School of Natural Sciences, Institute for Advanced Study, Princeton, New Jersey 08540' Physik-Department der Technische Universitat Mu'nchen, D-8046 Garching, Germany(Received 4 September 1990)We study the representation of static patterns and temporal sequences in neural networks with signaldelays and a stochastic parallel dynamics. For a wide class of delay distributions, the asymptotic net-work behavior can be described by a generalized Gibbs distribution, generated by a novel Lyapunovfunctional for the deterministic dynamics. We extend techniques of equilibrium statistical mechanics soas to deal with time-dependent phenomena, derive analytic results for both retrieval quality and storagecapacity, and compare them with numerical simulations.PACS numbers: 87.10.+e, 64.60.Cn, 89.70.+cAssociative reconstruction of noisy data in a neuralnetwork can be accomplished by endowing a pattern tobe memorized with a basin of attraction for the system'sretrieval dynamics. ' If the dynamics is governed by aLyapunov function, a simple intuitive understanding ofthe global computation becomes possible: The systemperforms a downhill motion in an energy landscapecreated by the stored information.For networks with static patterns, there is a Lyapunovfunction if the interactions between single neurons areinstantaneous and mediated by symmetric couplings.Methods of equilibrium statistical mechanics may thenbe applied and permit a quantitative analysis of thenetwork's performance in terms of the retrieval qualityand storage capacity. The existence of a Lyapunovfunction is thus of great conceptual as well as technicalimportance.In general, external inputs to a neural network are notlimited to static memories but provide information inboth space and time. To code the temporal aspects of se-quences of patterns to be learned, additional asymmetriccouplings with transmission delays may be introduced.This approach can be generalized to networks with abroad distribution of signal delays where Hebb's neuro-physiological principle for learning naturally leads to ajoint representation of spatial and temporal informa-tion. However, no description in terms of a Lyapunovfunction has been given so far.In this Letter such a description is developed for a cer-tain class of networks with transmission delays. Wepresent a Lyapunov functional for the deterministicparallel dynamics, generalize the formalism of equilibri-um statistical mechanics so as to deal with thermal noisein systems with delayed interactions, and, hence, makethe domain of time-dependent phenomena accessible topowerful free-energy techniques.We follow Refs. 1-3 and model single neurons by Is-ing spins 5;, 1 ~i ~ N. They represent a firing state for5; =+1 and a quiescent one for 5; = —1. The neuronsare connected by synapses with modifiable efficaciesJ;, (r). Here r denotes a fixed time delay for the infor-mation transport from j to i. We focus on a solitonlikepropagation of neural signals, characteristic for the (ax-onal) transmission of action potentials, and consider amodel where each pair of neurons is linked by several ax-ons with delays 0~ ~~ r,„. External stimuli are fedinto the system via two-state receptors cr; = + 1. The lo-cal fields (postsynaptic potentials) are then given byN maxwhere the parameter y measures the system's input sen-sitivity. Other transport mechanisms and network archi-tectures have been studied elsewhere; see also Refs. 4,5, 8, and 9.We concentrate on synchronous (parallel) dynamicsand set the basic time step equal to unity. Consequently,the signal delays take non-negative integer values. Theinhuence of synaptic noise may be described by a sto-chastic Glauber dynamics' at finite inverse temperaturep =T '. In that case,Prob[S;(t+1) = ~ 1] = —,' [1 ~ tanh[ph;(t)]j, (2)which reduces in the deterministic limit, p ~, toS;(t+1)=sgn[h;(t)]. The retrieval dynamics can bestudied in the bulk limit N ~ if the number andlength of the stored sequences remain finite.In the present Letter, we surpass the limits of this kindof analysis and show that an approach in terms of statist-ical mechanics is feasible for networks whose couplingsexhibit an "extended synaptic symmetry, " ' '(3)for some (fixed) integer D ) 2. Links with r )D —1are assumed to vanish, i.e., r,.„~D —2.The condition (3) can be met by simple training algo-rithms for the synaptic e%cacies. As an example, we1370 1991 The American Physical SocietyVOLUME 66, NUMBER 10 PHYSICAL REVIEW LETTERS 11 MARcH 1991AH(t) = —Q [S;(t)—S;(t—D)]h;(t —1) . (6)Deterministic dynamics, i.e., S;(t) =sgn[h;(t —1)], im-plies that the right-hand side will only take nonpositivevalues. For finite N, H is bounded from below and hHhas to vanish for t ~. This can be realized only if thesystem settles into a state with S; (t) =S;(t —D) for all iWe have thus exposed two important facts: (a) Theretrieval dynamics of a certain class of networks with de-layed interactions is governed by a Lyapunov functional,and (b) the system relaxes to static states or limit cycleswith S;(t) =S; (t —D )—oscillatory solutions with thesame period as that of the taught sequences (or a periodequal to an integer fraction of D). It remains to beshown that such a limit cycle of the retrieval dynamicsindeed resembles a stored sequence for appropriate ini-tial conditions.In order to prove the above assertion, we proceed inthree steps. First, we demonstrate that our task concern-ing temporal associations with a cyclic structure can bemapped onto a symmetric network without signal delays.present a linear Hebbian rule. During a learning session,a coupling strength J;J(i) is altered after each time stepby an amount proportional to the product of SJ(t —i)and S; (t+1)—due to the delay i in (1) and the discretedynamics (2) it takes i+1 units of time until neuron jinAuences the state of neuron i through that specificsynapse. After P learning sessions, labeled by p andeach of duration D„, we thus obtain1'J;,(i) =e(i) g—g S;(t„+1)S,(t„—i). (4)Np 1' 1The a priori weights e(i) are non-negative, normalizedsuch that p, -'*De(i) =1, and may be used to define vari-ous network architectures. For a "clamped" learningscenario with y= 1 in (1), the system evolves strictly ac-cording to the external inputs, S;(t„)=o;(t„—1). Equa-tion (3) holds once we teach cyclic associations with acommon period D to networks whose weight distributionsobey e(i) =e(D —2 —i)Returning to the general case, given by (3), we nowestablish that there is a Lyapunov functional for thenoiseless retrieval dynamics. We focus on a free evolu-tion of the system, i.e., we set y=0 in (1), generalize theresults of Ref. 12, and take1N D 1 rmaxH(t)=- ——g g g J;,(i)S, (t —a)ij 1a Or 0xS (t —(a+i+1)(modD)) .(5)The functional H depends on all network states betweent+1 —D and t so that solutions with constant H need notbe static fixed points of the retrieval dynamics. Thedifference AH(t) =H(t) —H(t —1) is given bySecond, we apply equilibrium statistical mechanics tostudy such "equivalent systems. " Third, we focus on aspecial case, the Hebbian learning rule (4), and deriveanalytic results for the retrieval quality and storagecapacity.D-periodic oscillatory solutions of the retrieval dynam-ics can be interpreted as static states in a "D-plicated"system. Let us therefore consider such a fictitious sys-tem with D columns and N rows of ~ 1 neurons whoseactivities will be denoted by S;,. A network state will bewritten 8 =(AO, A~, . . . , Ao-~) with A, =—tS;„1~i~ N}. To reproduce the parallel dynamics of the origi-nal system, neurons S;, with a =t (modD) are updated attime t. The time evolution of the new network thereforehas a pseudosequential characteristic: synchronouswithin single columns and sequentially ordered withrespect to these columns. Accordingly, the neural activi-ties at time t are given by S;,(t) —=S;(a+n, ) for a~ t(modD) and S;,(t) =S;(a+n, D) for —a & t(modD), where n, is defined through t= n, +t(m—odD).Because of (3), we may construct symmetric efficaciesJJ"=J~ for the new system by putting JJ'=0 andJ~j~=JJ((b —a —1)(modD)) for aWb,so that there is a well-defined Hamiltonian,H = ——g g Jt'gS;, Sjb,1 N D—1(g)2ij 1ab 0equal to that of a Hopfield network of size ND.An evaluation of {8) in terms of the former state vari-ables reveals that it is identical with the Lyapunov func-tional (5). We may once again calculate AH(t) and ar-rive at a formula corresponding to (6). The interpreta-tion, however, is changed: a limit cycle of period D inthe original network corresponds to a fixed point of thenew system of size ND. %"e have thus shown that thedeterministic parallel dynamics of a delay network withextended symmetry can be understood in terms of adownhill motion in the energy landscape of its equivalentsystem.If, in a second step, we now s~itch to a stochasticGlauber dynamics (2), the important question ariseswhether H also determines the invariant distribution p ofthe network. This need not be true since the column-wise updating procedure of the equivalent system differsfrom both the Little' and the Hopfield model. It turnsout, however, that there is indeed an equilibrium distri-bution i la Gibbs, "p(A) =Z ' exp[ PH(A)] . —Here Z—:Tr~ exp[ PH(A)] denotes the pa—rtition func-tion. The exact proof is elaborate since due to the non-Markovian structure of the dynamics (1) and (2), theprinciple of detailed balance does not apply to single-steptransition probabilities.In passing we note that for D=2 there are only links1371VOLUME 66, NUMBER 10 PHYSICAL REVIEW LETTERS 11 MARCH 1991with zero delay. By (3) we have J;&(0) =Jz, (0), i.e., weare dealing with a symmetric Little model. We may in-troduce a reduced probability distribution p for this spe-cial case, P(A() =Tr~, p(ApA(), and obtain P(A()=Z 'exp[ pH—(A()] withsay s, cycles of the original system (corresponding to sDstatic patterns for the equivalent model of size ND) andassume that the network is in a state highly correlatedwith these spatiotemporal objects. The remaining, ex-tensively many cycles are described as a noise term. Weconsider retrieval solutions and make the ansatzN NH= —P— ' g ln 2cosh P g JJS, (10) m~g =N'—ggP;SP, =m "8,p,qgP =N—'g S;,Sg =B,g [Sp (1 —q ) +q],We thus have recovered both the eAective Hamiltonianof the Little model as derived by Peretto' and theduplicated-system (D =2) technique of Ref. 13.Finally, we finish our argument by turning to quanti-tative results. We focus on synaptic couplings generatedby the Hebbian rule (4) and investigate the case whereeach of the P learning sessions corresponds to teaching a(different) cycle of D patterns [y,~; I (i ~ N], 1 (p~D, i.e., each pattern lasts for a single time step,o; (t„)= y,",„(,dD). In accordance with (4) and (8),there are PD patterns [g,".,'; 1(i (N, 0(a(D —I]for the equivalent network, 1 ~ p ~ P, 1 ~ e ~ D, givenby g;, =y;, and (ra 4i(a —a),(modD). In what follows, wepo p pa powork with unbiased random patterns where y,~=+ 1with equal probability, and study our network at a finitestorage level a =lim)v (P/N) )0.As in the replica-symmetric theory of Amit, Gut-freund, and Sompolinsky we single out finitely many,for the relevant order parameters of the n-fold replicatednetwork, 1 ~ p, o ~ n. By applying the techniques ofRefs. 3 and 14, we arrive at the fixed-point equations(13)(i4)q= tanh, m' + ar z,PI, (( )l'r=q-k=( [1 —p(1 —q)kl, (8)] (is)The double angular brackets represent an average withrespect to both the condensed cycles and the normalizedGaussian random variable z. The kk(C) are eigenvaluesof the matrix 8 with A, (, =e((b —a——1)(modD)). Atthe saddle points given by (13)-(15),the free energy isqpx, (e)i —p(i —q)x, (c)+ —aP'(q —l)r+ 1n2cosh P, gm "0 +der z, ) .2—Pf(P) = —~P(m') ——aP in[1 —P(1 —q)Xk (8)]—12I(@)I(Hopfield)Da, (C)da, (Hopfield) 'where d denotes the number of diAerent delay lines be-tween each pair of neurons, 1 ~ d ~ D —I. A numericalThe preceding equations closely resemble their counter-parts for the Hopfield (or Little) model and becomeidentical to them for a "maximally uniform" distribu-tion, (,q = (D —1) (1 —6,b), in the limit D ~. Forgeneral period D and weight distribution 8, there is atzero temperature a first-order phase transition betweenthe retrieval state and a spin-glass phase as in theHopfield case. The critical storage level e, and the cor-responding overlap m, are difI'erent, though.It should be noted that each cycle consists of D in-dependent patterns so that the storage capacity for singlepatterns is a, =Da, . During the recognition process,however, each of them will trigger the cycle it belongs toand cannot be retrieved as a static pattern.As a second measure of the performance, we includethe information content I~(8), measured per synapseand relative to that of the Hopfield model,D0.1000.1100.1160.120m,0.930.950.960.961.451.201.121.09The above findings agree well with estimates from afinite-size analysis (N(3000) of data from numericalsimulations as shown by two examples. For D=3, wehave found a, =0.120%0.015, and for D =4, a, =0.125L- 0.015.Our results demonstrate that the storage capacity fortemporal associations is comparable to that for staticmemories. As an example, we take D =2, i.e., the Littlemodel. In the limit of large N, we see that 0.100N two-cycles of the form y,"~ —y,". 2 may be recalled as comparedto 0.138N static patterns;' this leads to a 1.45-fold in-tsolution of the saddle-point equations at T=O leads tothe following results for networks with a "maximallyuniform" distribution:1372VOLUME 66, NUMBER 10 PHYSICAL REVIEW LETTERS 11 MARcH 1991crease of the information content per synapse. At thesame time, the retrieval overlap drops slightly from 0.97to 0.93. The influence of the weight distribution on thenetwork behavior may be demonstrated by some choicesof e(z) for D =4:0 1 2 3 a, IRe(z)e(r)e(z)13I20I0I0 00.116 0.960.100 0.930.050 0.931.121.451.45The uniform distribution leads to the largest a, butsmallest Ig. The other two networks have the samevalue of IR as the (unique) D =2 system due to the par-ticular structure of their eigenvalue spectrum. Further-more, one obtains IR =1.45 independently of D for allnetworks with a "minimal connectivity" where only onesynapse links two neurons (D even). Simulation datashow once again slightly higher values of e,—possiblyindicating eAects of replica-symmetry breaking as in theHopfield model.In summary, we have introduced a Lyapunov func-tional for a wide class of neural networks with signal de-lays and have generalized equilibrium statisticalmechanics so as to analyze the network behavior under astochastic dynamics. Quantitative results have been ob-tained for a Hebbian learning rule. They prove that anextensive number of temporal associations can be storedas spatiotemporal attractors for the retrieval dynamics.Even at the critical storage level, retrieved sequencescontain less than 3.5% errors for all studied architec-tures. Our results indicate that dynamical systems withdelayed interactions can be programmed in a veryefFicient manner to perform associative computations inthe space-time domain.The current approach provides a new framework forthe study of temporal associations in delayed networks.Though limited to networks whose couplings exhibit an"extended symmetry, " it may be generalized in variousdirections. To mention a few examples, first, more so-phisticated learning prescriptions like a spatiotemporalversion of the pseudoinverse rule can be investigated.Second, one may wish to switch to continuous state vari-ables and study temporal associations in "iterated-mapnetworks. "' It turns out that a generalization of theLyapunov functional (5) covers that case as well, ' al-lowing for a direct comparison of theoretical predictionswith experimental results from hardware implementa-tions. Finally, one could try to develop a Lyapunov func-tional for a continuous-time dynamics including signaldelays which seems to be rather significant for applica-tions as well as for the general theory of functionaldifferential equations and dynamical systems.We thank J. J. Hopfield and R. Kiihn for many stimu-lating discussions. Both of them as well as J. Buhmannand D. MacKay helped us with constructive commentson the manuscript. A.V.M.H. acknowledges supportfrom the Studienstiftung des Deutschen Volkes. Z.L. issupported in part through a grant from the Seaver Insti-tute.'W. A. Little, Math. Biosci. 19, 101 (1974).2J. J. Hopfield, Proc. Natl. Acad. Sci. U.S.A. 79, 2554(1982).D. J. Amit, H. Gutfreund, and H. Sompolinsky, Phys. Rev.A 32, 1007 (1985); Ann. Phys. (N.Y.) 173, 30 (1987).4D. Kleinfeld, Proc. Natl. Acad. Sci. U.S.A. 83, 9469(1986).~H. Sompolinsky and I. Kanter, Phys. Rev. Lett. 57, 2861(1986).6D. O. Hebb, The Organization of Behavior (Wiley, NewYork, 1949), p. 62.7A. V. M. Herz, B. Sulzer, R. Kuhn, and J. L. van Hemmen,Europhys. Lett. 7, 663 (1988); Biol. Cybern. 60, 457 (1989).8A. C. C. Coolen and C. C. A. M. Gielen, Europhys. Lett. 7,281 (1988).9M. Kerszberg and A. Zippelius, Phys. Scr. T33, 54 (1990).' oP. Peret to, Biol. Cybern. 50, 51 (1984).' 'A. V. M. Herz, Ph. D. thesis, Heidelberg, 1990 (unpub-lished); Z. Li and A. V. M. Herz, in IVeural iVetworks,Proceedings of the Eleventh Sitges Conference, Lecture Notesin Physics Vol. 368, edited by L. Garrido (Springer-Verlag,Berlin, 1990), pp. 287-302.'zZ. Li (unpublished).'3J. L. van Hemmen, Phys. Rev. A 34, 3435 (1986).'4J. L. van Hemmen and R. Kuhn, Phys. Rev. Lett. 57, 913(1986); J. L. van Hemmen and V. A. Zagrebnov, J. Phys. A20, 3989 (1987).'5J. F. Fontanari and R. Koberle, Phys. Rev. A 36, 2475(1987),' C. M. Marcus and R. M. Westervelt, Phys. Rev. A 42,2410 (1990).'7A. V. M. Herz (to be published).1373

Statistical mechanics of temporal association in neural networks with transmission delays

Abstract

Similar works

Full text

Available Versions

Universität München: Elektronischen Publikationen

Open Access LMU

Crossref

MPG.PuRe