We analyze the retrieval dynamics of analog ‘‘neural’’ networks with clocked sigmoid elements and multiple signal delays. Proving a conjecture by Marcus and Westervelt, we show that for delay-independent symmetric coupling strengths, the only attractors are fixed points and periodic limit cycles. The same result applies to a larger class of asymmetric networks that may be utilized to store temporal associations with a cyclic structure. We discuss implications for various learning schemes in the space-time domain

Herz, Andreas V. M.

English

A. V. M. Herz

Crossref

Physical Review A

Global analysis of parallel analog networks with retarded feedback

Open Access LMU

PHYSICAL REVIEW A VOLUME 44, NUMBER 2 15 JULY 1991Global analysis of parallel analog networks with retarded feedbackA. V. M. HerzDiuision of Chemistry, California Institute of Technology 139-74, Pasadena, California 91125(Received 6 March 1991jWe analyze the retrieval dynamics of analog "neural" networks with clocked sigmoid elements andmultiple signal delays. Proving a conjecture by Marcus and Westervelt, we show that for delay-independent symmetric coupling strengths, the only attractors are fixed points and periodic limit cycles.The same result applies to a larger class of asymmetric networks that may be utilized to store temporalassociations with a cyclic structure. We discuss implications for various learning schemes in the space-time domain.Signal transmission processes play an important rolefor the dynamics of the nervous system [1]. The underly-ing electrochemical transport mechanisms are quite intri-cate and cannot be modeled explicitly within the mesos-copic approach [2] to biological information processing.On this level of description, single neutrons are treated aselementary building blocks, most commonly in the formof binary threshold units or sigmoid analog elements, andsignal propagation processes are incorporated into thedynamical description as retarded interactions [3]. Nu-merical and analytical studies show that a chaotic timeevolution is rather common in systems with delayedfeedback —in single cells [4] as well as in large assemblies[5].In contrast to the above observation, some neurobio-logical central pattern generators utilize delayed feedbackto initiate precise rhythmic activity of motor neutrons[6]. Furthermore, artificial networks with retarded in-teractions have proven to operate very efFectively as asso-ciative memories in the space-time domain once specialconfigurations of synaptic coupling strengths have beenimplemented by hand [7,8] or memorized by some ap-propriate learning rule [9—12]. However, even these sys-tems exhibit irregular temporal behavior for certain pa-rameter values [13,14], greatly affecting their capabilitiesas content addressable memories.It is therefore of broad interest to determine generalconditions which exclude spurious chaotic motion indelayed-feedback networks but still allow for nontrivialspatiotemporal phenomena. As is well known, a localstability analysis cannot deal with this question [15], andinstead, global techniques have to be applied.So far, two similar approaches have been reported:For systems with binary elements and a parallel deter-ministic updating rule, a Lyapunov functional for the re-trieval dynamics has been introduced in Ref. [16]. It gen-eralizes Hopfield's concept [17] of an "energy landscape"to the space-time domain, applies to a wide class of net-works that store both static patterns and temporal associ-ations, and assures that the dynamics relaxes to fixedpoints or periodic limit cycles. A slightly different routehas been taken by Marcus and Westervelt [18]. These au-thors have focused on systems with analog elements anddelay-independent symmetric interactions. Extendingh, (t) = g g J,, (r)V, (t r)+I, (t) . —j=1 T=O (2)Here, V; denotes the activity of the ith cell, 1 ~i ~ %, anddepends through g; on the local field (postsynaptic poten-tial) h, , which in turn is determined by the external inputI, and previous neural activities, weighted by the synapticcoupling strengths J; (r). For simplicity, the basic timestep (clock cycle) has been set equal to unity. Conse-quently, the signal delays take nonnegative integer values.For the special case of delay-independent symmetricweights, i.e., J;1(r) =J;1=JJ; for 0 ~ r ~r,„, we recoverthe description of Ref. [18].Throughout the present paper, it is generally assumedthat the input-output characteristics g,- are single valued,monotonically increasing, and differentiable, so that thereare we11-defined inverse functions g; ' with the sameproperties. We will call transfer functions g; "sigmoid, "if in addition they also grow in magnitude more slowlythan linear for large positive or negative arguments. Weintroduce auxiliary quantities G, ( V, ),prior work on graded-response neurons [19,20] anditerated maps [21], they showed that clocked networksconverge to fixed-point attractors under certain condi-tions for the synaptic strengths. Guided by some evi-dence from systems with small maximum delay, they con-jectured that if those restrictions for the synaptic weightsare not met, the only new attractors are periodic limit cy-cles.In the present paper, essential ingredients of both ap-proaches are combined to form a more general frame-work for the study of analog networks. We focus on syn-chronous systems, introduce a novel Lyapunov function-al, and prove the conjecture of Marcus and Westervelt.We then turn to a larger group of asymmetric networksand discuss both unsupervised and teacher-forced learn-ing schemes for temporal sequences.The time evolution of analog networks with a synchro-nous updating rule is defined by a set of coupleddifference equations,V, (t +1)=g, (h, (t))withN max1415 1991 The American Physical Society1416 BRIEF REPORTSV,.6;(V, )—=I dxg; '(x),0which for sigmoid g,- and large absolute V; increase fasterthen V; by construction. Following the notation of Ref.[18],we denote the maximum slope of g; by P;.We now prove that there is a Lyapunov functional forthe dynamical system (1),(2) if the synaptic couplings ex-hibit an "extended synaptic symmetry" [21] for someI(fixed) positive integer D,J~.;(r) for r=D —1J (r)= ' J,, (D —r —2) for 0&r&D —1.Links with w~D are assumed to vanish, i.e., ~, (D.Notice that the coupling strengths might be asymmetricfor r&D —1 in the sense that JJ(r)W JJ;(r).Consider the functionalN D —1 N D —1H(t)= ——,' g g J,,(r)V(t —. a)V, (t —(a+r+1)modD)+ g g [6;(V (t r)) —I;(t —1 r)V;(t—r)] .i j =1 a, ~=0 i =1 &=0External inputs are supposed to be D periodic, a condition that is satisfied in a trivial way for the standard case of time-independent inputs. The difference bH(t) =H(t) H— (t —1) is then given byN NAH(t)= —g AZ;(t)h, (t —1)——,' g J, (D —1)bZ, (t)bZ (t)+ g [6;(V(t))—6;(V;(t —D))],D —1Z, (t)—= g V, (t —r) (7)andb Z; (t) =Z; (t) Z; (t —1)=—V; (t) V; (t D) . ——The right-hand side of (6) is zero if b,Z, (t) =0 for all iIn what follows, we analyze the case where V;(t)WV;(t D) for at —least some i Since. g, is single valued and mono-tonically increasing by assumption, 6, is strictly convex. Through a Taylor expansion of 6;(V;(t —D)) around6;( V, (t)), we obtain the following upper bound for those i in the last term of (6):6,.(V(t)) —G, (V(t —D)) &AZ, (t)G,'(V(r)) ,'bZ, (t) G,"(V—(t)—) bZ, (t)h, (t —1) ,'bZ, (t) p, —' —. . (8)For a graphical illustration see Ref. [18]. Inserting (8)into (6), we arrive atbH(t) & ——,' g J,, (D —1)bZ, (t)bZ, (t)i j =1N——'gP 'b,Z(t)'A sufficient condition for b,H(t) & 0 is therefore given byI, ;„[J(D—1)]~ —P (10)where A, ;„[J(D —1)] denotes the smallest eigenvalue ofthe matrix J(D —1) with components J;j(D —1) andf3=max; [P, I. As long as AZ;(t)WO for some i, H has todecrease if (10) is met. Since H is bounded from belowfor sigmoid g, , bH(t) and b,Z,.(t) vanish asymptoticallyfor large t.We have thus demonstrated that the retrieval dynam-ics of synchronized analog networks with sigmoid ele-ments and retarded feedback is governed by a Lyapunovfunctional if the synaptic weights satisfy both (4) and (10).Furthermore, we have shown that the network relaxes toa solution with V;(t) = V, (t D), i.e., to a static state. or—alimit cycle with a period equal to D or an integer fractionof D.The model of Ref. [16] uses two-state threshold unitsV; =+1 with a parallel stochastic time evolution,Prob[ V;(t +1)=+1]= —,' [ I+tanh[Ph;(t)]]. Here, Pdenotes the noise level and h;(t) is given by (2) as in thecontinuous case. The deterministic limit,V~(t +1)=sgn[h;(r)], may also be regarded as a high-gain limit of the analog network if we consider input-output characteristics g; that saturate at +1 ( —1) forlarge positive (negative) arguments, e.g., g, (h, )=tanh(P;h;). The relationship between the discrete andthe continuous system is therefore similar to that betweenthe original Hopfield model [17] with random sequentialstochastic updates and its graded-response counterpart[20], operating in continuous time. This resemblance car-ries over to the Lyapunov functionals as well, where forP, —+ ~, the contribution of 6, to (5) can be neglected,leaving the functional for the Ising-spin model with dis-tributed delays.Returning to analog systems, we now discuss some re-sults for two special synaptic configurations. First, wechoose delay-independent symmetric couplings andr,„=D—1,J,J.(r) =J,J =JJ; for 0 & r & D —1 .The temporal sums in (5) decouple and we obtainBRIEF REPORTS 1417NH (r)= ——,' g J,,Z, (t)Z, (t)i j =1N D —1+ g g [G, (V, (t —~))i =1 ~=0(12)I, (—t —1 r)—V, (t —r)] .This expression is equivalent to the first Lyapunov func-tional of Ref. [18] where it is shown that if (10) and (ll)hold, there are fixed-point attractors only. With properlynormalized couplings, J; ~ D ', these solutions are iden-tical to the attractors of the graded-response model [20],however, regions of attraction and convergence times arechanged [18].Second, we consider a slight modification of Eq. (11),0 for ~=D —1Jj Jj for 0 & «D —1J,J.(r) =(14)where p labels the P sequences I V, (t„); 1 ~i ~N,1 ~t„~D]. The parameters e(r) may be used to definevarious network architectures [10]. The couplingstrengths will, in general, be asymmetric, J; (~)WJJ;(~),but for all networks with E(r)=e[D —(2+v)modD], the"extended synaptic symmetry" (4) still holds. For simpli-city, we set e(D —1)=0 to satisfy (10) as well.As a result of the modulo operation in Eq. (14), arbi-trary sequences are memorized as if they were D-periodiccycles. They will be retrieved accordingly. If duringlearning, a sequence is already presented ~,„ time stepsEquation (10) is now satisfied for all sigmoid g;. We havethus proven the conjecture of Marcus and Westervelt[18]: For synchronous analog networks with arbitrarysigmoid input-output functions, symmetric couplings andmaximum delay D —2, all attractors are restricted to belimit cycles with periods of D and its divisors.In passing I would like to note that if D is replaced byD + 1 in (13), both (11) and (13) describe the same net-work. However, the corresponding Lyapunov function-als differ even for constant external inputs, and it is onlyfor the second case that one may draw the above con-clusions. Furthermore, it is worthwhile to mention thatthe second Lyapunov functional of Ref. [18) followswithout guesswork as a special case from (5) and (13) forD =3.For the remaining part of the paper, we turn to moregeneral coupling matrices and discuss the storage of tem-poral associations with common duration D. Since weknow that the retrieval dynamics has to converge to D-periodic limit cycles once the conditions (4) and (10) aremet, the main task is to construct or learn synaptic cou-plings which guarantee that these limit cycles indeedresemble the desired spatiotemporal attractors for ap-propriate initial conditions. Similar to the static case,various approaches may be pursued. I will focus on oneunsupervised learning rule and briefly comment on someteacher-forced scenarios.Generalizing from networks with binary elements [16],let us takebefore synaptic strengths are altered, the modulo opera-tion may be dropped, leaving a learning rule that is Heb-bian [22] in the sense that it correlates presynaptic actiuitywith subsequent postsynaptic response —in our case theresponse is delayed by ~+ 1 time steps due to both thesignal lag and the additional step of the discrete-time dy-namics (1).For sequences of uncorrelated, binary random patternsV, (t ) =+1, quantitative analytical and numerical resultsabout the network performance have been obtained in thehigh-gain limit P;~ oo [16]. They show that the storagecapacity for temporal associations is comparable to thatfor static memories in the Hopfield model. Arbitrary sig-moid input-output functions may be treated in a similarfashion if one combines the concept of equivalent staticsystems [16] with a statistical mechanical analysis ofgraded-response neurons [23].The learning rule (14) may be utilized to store cycles ofcorrelated real-valued patterns as well. First numericalstudies have been performed for low-dimensional trajec-tories (small N) with high numbers of data points (largeD) For m. any examples, good retrieval could be obtainedthis way [24]—without any need for highly time-consuming supervised learning schemes [25]—however,teacher-forced algorithms might eventually be necessaryto solve more sophisticated tasks. Here, once again, theexistence of a Lyapunov functional is of great help sinceit allows for an application of mean-field techniques [26]to a wide class of supervised learning strategies like spa-tiotemporal extensions of the "Boltzmann Machine" con-cept [27] or, more general, contrastive-learning schemes[28,29].Concerning hardware applications, let us note that thedynamical system (1),(2) can readily be implemented withcharge-coupled devices. There, the vector-matrix multi-plication in Eq. (2) can be accomplished in a fully parallelfashion at a clock rate in the megahertz regime [30], al-lowing for fast systematic experiments of adaptive learn-ing schemes. Our theoretical analysis might help toguide and complement such work.In summary, we have studied the global dynamics ofparallel networks with analog elements and retardedfeedback. Using a novel Lyapunov functional it could beproven that for a wide class of system architectures andtransfer functions, the only attractors are fixed points andcertain periodic limit cycles. The results can be regardedas design criteria for artificial networks as well —theyshow how to extend well-known learning schemes to thespatiotemporal domain and guarantee the absence ofspurious chaotic trajectories.The present study has been triggered by earlier colla-borations with Leo van Hemmen, Reimer Kuhn, Zhao-ping Li, and Bernhard Sulzer. The author also benefitedfrom various activities within their Computation andNeural Systems program at the California Institute ofTechnology. It is a great pleasure to thank Pierre Baldi,John Hopfield, Marcus Mitchell, and Chuck Neugebauerfor valuable discussions. This work has been supportedby the Ofhce of Naval Research Contract No. N00014-87-K-0377 and by a grant from the Studienstiftung desDeutschen Volkes.1418 BRIEF REPORTS[1]See, for example, S. W. Kufiler, J. G. Nicholls, and A. R.Martin, From Neuron to Brain (Sinauer, Sunderland,1984), p. 99ff.[2] For an overview see D. J. Amit, Modeling Brain Function:The World of Attractor Neural Networks (Cambridge Uni-versity Press, Cambridge, 1989); J. Hertz, A. Krogh, andR. G. Palmer, Introduction to the Theory of Neural Computation (Addison-Wesley, Redwood City, 1991); Physicsof Neural Networks, edited by E. Domany, J. L. van Hem-men, and K. Schulten (Springer, Heidelberg, 1991).[3] E. Caianiello, J. Theor. Biol. 1, 204 (1961).[4] K. Aikara, T. Takabe, and M. Toyoda, Phys. Lett. A 144,333 (1990).[5] M. Y. Choi and B. A. Huberman, Phys. Rev. B 31, 2862(1985).[6] P. A. Getting, J. Neurophysiol. 49, 1036 (1983).[7] D. Kleinfeld, Proc. Natl. Acad. Sci. U.S.A. 83, 9469(1986).[8] H. Sompolinsky and I. Kanter, Phys. Rev. Lett. 57, 2861(1986).[9] A. C. C. Coolen and C. C. A. M. Gielen, Europhys. Lett.7, 281 (1988)~[10]A. V. M. Herz, B. Sulzer, R. Kiihn, and J. L. van Hem-men, Europhys. Lett. 7, 663 (1988); Biol. Cybern. 60, 457(1989).[11]M. Kerzberg and A. Zippelius, Phys. Scr. T 33, 54 (1990).[12] K. Bauer and U. Krey, Z. Phys. B Condens. Matter 79,461(1990).[13]U. Riedel, R. Kiihn, and J. L. van Hemmen, Phys. Rev. A38, 1105 (1988).[14]T. Fukai and M. Shiino, Phys. Rev. Lett. 64, 1465 (1990).[15] C. M. Marcus and R. M. Westervelt, Phys. Rev. A 39, 347(1989).[16]Z. Li and A. V. M. Herz, in Proceedings of the XI SitgesConference on Neural Networks, edited by L. Garrido,Lecture Notes in Physics Vol. 368 (Springer-Verlag, Ber-lin, 1990), pp. 287—302; A. V. M. Herz, Z. Li, and J. L.Van Hemmen, Phys. Rev. Lett. 66, 1370 (1991).[17]J. J. Hopfield, Proc. Natl. Acad. Sci. U.S.A. 79, 2554(1982).[18]C. M. Marcus and R. M. Westervelt, Phys. Rev. A 42,2410 (1990).[19]M. A. Cohen and S. Grossberg, IEEE Trans. SMC 13, 815(1983).[20] J. J. Hopfield, Proc. Natl. Acad. Sci. U.S.A. 81, 3088(1984).[21] C. M. Marcus and R. M Westervelt, Phys. Rev. A 40, 501(1989).[22] D. O. Hebb, The Organization of Behavior (Wiley, NewYork, 1949), p. 62.[23] R. Kiihn, in Proceedings of the XI Sitges Conference onNeural Networks (Ref. 15), pp. 19—32; R. Kuhn, S. Bos,and J. L. van Hemmen, Phys. Rev. A 43, 2084 (1991).[24] A. V. M. Herz and M. Q. Mitchell (unpublished).[25] B.A. Pearlmutter, Neural Comput. 1, 263 (1989).[26] C. Peterson and J. R. Anderson, Complex Systems 1, 995(1987).[27] D. H. Ackley, G. E. Hinton and T. J. Sejnowski, CognitiveSci. 9, 147 (1985).[28] J. Movellan, Proceedings of the 1990 Carnegie MellonSummer School (Morgan Kaufmann, San Mateo, 1991).[29] F. Pineda and P. Baldi (unpublished).[30] A. Agranat, C. F. Neugebauer, and A. Yariv, Appl. Opt.27, 4354 (1988); C. F. Neugebauer and A. Yariv (unpub-lished).

http://epub.ub.uni-muenchen.de/14830/1/p1415_1.pdf

Global analysis of parallel analog networks with retarded feedback

Abstract

Similar works

Full text

Available Versions

Crossref

Open Access LMU