It is shown that if the neuronal gains are small compared with the synaptic connection weights, then a bidirectional associative memory network with axonal signal transmission delays converges to the equilibria associated with exogenous inputs to the network. Both discrete and continuously distributed delays are considered; the asymptotic stability is global in the state space of neuronal activations and also is independent of the delays

Gopalsamy, Kondalsamy

He, Xue-Zhong

IEEE Transactions on Neural Networks

English

Flinders Academic Commons

998  IEEE TRANSACTIONS ON  NEURAL NETWORKS,  VOL.  5, NO.  6, NOVEMBER  1994 Delay-Independent Stability in Bidirectional Associative Memory Networks K.  Gopalsamy and Xue-Zhong He Abstract-It  is shown that if the neuronal gains are small com- pared with the synaptic connection weights, then a bidirectional associative  memory  network  with  axonal  signal  transmission delays  converges  to  the  equilibria  associated  with  exogenous inputs to the network; both discrete and continuously distributed delays are considered; the asymptotic stability is global in  the state space of neuronal activations and also is independent of the delays. I.  INTRODUCTION HE  stability characteristics of  equilibria of  continuous  T  bidirectional associative memory networks of  the type n  i = 1,2,...,n ;=1 --  dYi(t) -  -Yi(t) + Cmijs(xj(t))  + Ji dt j=1 and some of  their generalizations have been investigated by Kosko  [9],  [lo]. Networks  of  the  form  (1)  generalize the continuous Hopfield circuit model [8] and can be obtained as a special case from the model of Cohen and Grossberg [3]. If one assumes that the exogenous inputs Ii, Ji (i = 1,2,  . .  . ,  n)  and the connection weights mij (i,  j  = 1,2,  .  . . ,  n)  are constants while the neuronal output signal function S is a differentiable, monotonic nondecreasing real valued function on (-m,  m), then it is possible to introduce an energy function (or Lyapunov function) E  such that n  CT.  nn n  n  n i=l  j=1  j=1 (2) where S’(z)  = F.  It  has been shown in  [9] that n  n kl  j=1 One can show from (3) that as t i  m,  ii(t) i  0,  yi(t) +  0, i = 1,2,  . . . ,  n implying that the network (1) converges to an equilibrium corresponding to the constant extemal inputs Ii,  Ji (i = 1,2,  . . .  ,  n).  The equilibria are sometimes called pattems or memories associated with the extemal inputs I  and J. Manuscript  received June 25, 1992; revised June 15, 1993. The  authors are  with  the  School  of  Information  Science  & Technology, IEEE Log Number 921 1596. Flinders University,  Adelaide,  SA 5001, Australia. It is possible to simplify bidirectional networks of the type in (1) to a single system of  a network of  the type for  suitably defined nonlinear functions  fi,  i  = 1,2,  . .  . ,  n. In fact, a referee suggested that we  do such a simplification. The authors have to  retain the model  (1) as it stands since such a simplification will  alter the bidirectional interplay of the input-output nature of  the two layers of  the system and will  reduce the system to that of  a single layer system. For a detailed investigation of  single layer systems we  refer to a recent article of  Gopalsamy and He  [7]. The purpose of  this brief article is to investigate the exis- tence and stability characteristics of the equilibria of  networks of  the form n dUi(t)  - dt  -  -ui(t) + j=1  UijS(AjWj(t -  CrZj)) + I; i= 1,2,*..,n  (4) in which Xj pj,  rij ,  aij (2, j  = 1,2,  .  . . ,  n)  are nonnegative constantsand&, Ji, ai;, bij(i,j= 1,2,.‘.,n)  arerealnum- bers; for convenience of exposition in the following we choose the signal response function as follows S(z) = tanh(x),  z E (-m, 00).  (5) The time delays T~~ and ~7%~  correspond to the finite speed of the axonal transmission of signals; for example T~,  corresponds to the time lag from the time the 2-th  neuron in the I-layer emits a signal and the moment this signal becomes available for  the  j-th neuron  in  the  J-layer of  (4) (see for  instance Domany et al. [4]). The constants A,,  pJ correspond to the neuronal gains associated with the  neuronal activations. We refer to Babcock and Westervelt [l], Marcus and Westervelt [12], [13] and Marcus et al. [14] for linear analyses of  single layer networks with  delays. One of the problems in the analysis of the dynamics of the delay differential system (4) is the existence of  solutions of (4). The initial conditions associated with (4) are assumed to be  of  the  form u,(s)  = h(s),  s E  [-~*,0],  T*  = maxlsz,23snTz, u,(s)  = &(s),  s E [-0*,0],  (T* = maxl<z,,<n  gZI  1 i = 1,2,...,n  (6) 1045-9227/94$04.000  1994 IEEE IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 5, NO. 6,  NOVEMBER 1994  999 in which  q5i,  $i  are continuous real valued functions defined on their respective domains. One can use the method of  steps and continuation (see Elsgolt's and Norkin [SI) and show that solutions of  (4)-(6)  exist for all t 2 0. The following result provides sufficient conditions for the existence of  equilibria associated to each pair of  inputs I and J in (4). where UljS(XjYj) + I1 cj=1  @jS(XjYj) +  12 anjS(Xjyj)  +  In bljqpjzj)  + J1 Ej=1  bZjS(PjZj) +  Jz F~(x,y)  = r='  ...  1, i)  the connection weights uij, bij (i,  j = 1,2,.  . . ,  n)  are real  FZhY)  = I  ... Theorem 1:  Assume the following: constants; ii)  the  exogenous  inputs  Ii,  Ji (i =  1,2,.  ..  ,n) are  real constants;  We  note from (12) and (13) that if  (x,  y) and (X,  Y) are any two points of  R, then  iii)  the gain parameters pj, Xj  (j  = 1,2,  . . . ,  n)  and the time- delays  rij  and  '~ij  (i,j = 1,2,-..,n)  are  nonnegative constants:  llF(x9 Y)-F(X,Y)II nn = I  [.ij{S(XjYj/j) -  sj(xjY,)l iv)  there exists a number  c  E  (0,l) such that the neuronal i=l j=1  gains and the connection weights satisfy +bij{S(Pjxj) -  S(PjXj))l Then corresponding to each exogenous input pair of  vectors I = (11,12,-.-,In)  and J = (J1,Jz,..-,Jn)  there exists an unique equilibrium (U*,  'U*) of  (4) satisfying n n  i = 1,2,.  . .  ,  n.  (8) Uf  =  u;js(xj'U;) +  Ii j=1 W:  =  bijS(pju5)  + Ji j=1 Pro08  We have from (4) and (5) that an arbitrary solution of  (4)-(6)  satisfies the following differential inequalities i=1,2,...,n (9) -Ui(t) -  a; 5 -wi(t) -  pi  5 *  I  -ui(t) + a2 q  5 -wi(t)  +  pi where i = 1,2,-..,n.  (10) Pi =  lbijl + IJil j=1 is  invariant with  respect to  the delay  differential equations (4). Thus if  the system (4) has an equilibrium, then such an equilibrium is a fixed point of  the mapping F : R + RZn defined by nn n in deriving (15) and subsequent inequalities we have used the facts that 8;  lies between  Xjyj and  Xjq, 6'7  lies between pjxj and pjXj as well as S'(8) = 1 -  Sz(6) 5 1  for  8 E (0,~). The mapping F is continuous and F(R)  C R; it follows from (16) and  c < 1 that F is a contraction on 0. By  the well known contraction mapping principle, we conclude that there exists a unique point say (U*,  'U*) such that (18)  F(u*,  w*)  = (U*,  'U*) and this completes the proof. Thus there exists a unique pattem or memory (or equilib- rium) associated with each set of  the external inputs I and J when the connection weights uij and bij are fixed. In the next section we derive conditions for the global asymptotic stability of  the unique equilibrium (U*,  'U*) of  (4). loo0  IEEE  TRANSACTIONS  ON NEURAL NETWORKS, VOL. 5, NO.  6, NOVEMBER  1994 11.  RECALL  OF  ASSOCIATIVE  MEMORY The equilibrium or the pattem  (u*,w*) of  (4) associated with  a  given  (I,  J)  is  said  to  be  globally  asymptotically stable  independent  of  the  delays  if  every  solution  of  (4) corresponding to an arbitrarily given set of  initial values (6) satisfy solutions of  (21) laijlXjlYj(t -  UiJI i=l  j=1 i=l -  IYi(t)l  +  IbijIPMt -  %)I lim  U;@)  =U:  and  lim  vi(t) =  wz*  (i = 1,2,.-.,n). t-oo  t-co (19) Weremark that when aij # bij, Xj # 1, pj # 1, rij 0, the method of  Lyapunov functions exploited by  Hopfield [8], Cohen and  Grossberg [3], Kosko [9] is not  applicable; the  existence of  a  Lyapunov function is  dependent on  the symmetry of  the synaptic connection weights in [8], [9] and Theorem 2  Suppose the assumptions of  Theorem  1 hold. Then the equilibrium (U*,  w*) of  (4) is globally asymptotically stable. 0, U a..  -  b.. i  z3 - z3  n the case of  Kosko [9]. Proof:  Define the variables xi and y; by the following j=1 n One can derivative that the deviations xi,  y; are governed by  it is consequence of  (25) that It follows from (26) and (23) that V is bounded on (0,m); the boundedness of  V on (0,~)  implies that xi,  yi are bounded (0,  CO)  and this implies that xi,  y;  are uniformly continuous on  (0,  CO).  We  note from  (22) that  ki,  $i  are  bounded  on on (0,~).  A consequence of  (26) is that 9  = -yi(t)  + E  b;jpjSi(BG(t))xj(t -  ~ij) j=1 i = 1,2,  ".,n  (21) in which erj(t)  lies between Xjvg and Xjvj(t-gij)  and 19:~(t) lies between pju;  and pjuj(t -  T~~)  for j  = 1,2,.  . . ,  n. We note  that i=  1 S'(s) = 1 -  S2(s) 5 1,  s E R The uniform continuity of  [Ixi(t)l +  Iyi(t)l]  on (0,~) together with (27) implies by Barbalat's Lemma (see Barbalat  and  hence we  have from (21) that n  [2] or Gopalsamy [6]) that n n  [Ixi(t)l+  ~yi(t)ll  -+ o  as  t +  00.  (28) i=l Thus  it  follows  xi(t)  --+  O,yi(t)  -+  0  as t  -+  CO,  i  = 1,2,  . .  . ,  n and  hence (22) 9  5 -xi(t) + E  lUijIXjlYj(t -  %j)( 3=1 5 -yi(t) +  Ibij(pjlzj(t  -  ~j)l j=1 i = 1,2,...,n. Consider a Lyapunov functional V(t)  = V(z,  y)(t) defined by L 111.  DISTRIBUTED  DELAYS The  use  of  constant  fixed  delays  in  models  of  delayed feedback provides of a good approximation in simple circuits consisting of a small number of cells. Neural networks usually have a spatial extent due to the presence of  a multitude of (23)  I + 2  IbijlPj J'  IZj(S)I ds  * 3=l  t-rq Calculating the  upper right derivative D+V  of  V along the IEEE TRANSACTIONS ON NEURAL NETWORKS, VOL. 5, NO. 6,  NOVEMBER 1994  1001 parallel pathways with  a variety of  axon sizes and lengths. Thus there will be a distribution of conduction velocities along these pathways and a distribution of  propagation delays. In these circumstances the signal propagation is not instantaneous and  cannot  be  modeled  with  discrete  delays  and  a  more appropriate way  is  to  incorporate  continuously  distributed delays. The extent to which the values of the state variable in the past affect its present dynamics is determined by a delay kemel; the case of  constant discrete delays corresponds to a choice of the delay kemel to be a Dirac delta function. Kernels of  the form p+l m!  K(t)  = -tme--at  ,  t 2 0,  a,m  E [O,m) have been  found  mathematically tractable in  many  models of  dynamic  systems in  mathematical ecology especially in population dynamics. We  shall  now  consider  a  class  of  bidirectional  associa- tive memory  networks with  continuously distributed delays described by dt  7=1  03  I in which the extemal inputs li,  Ji and the connection weights aij ,  bij are constants; the neuronal gains Xj  and pj are positive constants; the delay kemels  IC::),  k$) are nonnegative valued continuous functions defined on [0, m) satisfying The initial values associated with (29) are of  the foq Ui(S) = di(S),  Yi(S) =  $i(S),  s E (-,OI, i=l,2,...,n  (31) where di,  $;  are assumed to be bounded continuous functions defined on (-w,O].  For an extensive discussion of the stability and asymptotic behavior of  integro-differential equations such as (29) and their applications, we refer to the recent monograph by  Gopalsamy  [6].  For  applications  of  integro-differential equations with continuously distributed delays such as those in (29), we refer to Tank and Hopfield [15]. One can see that the equilibrium (U*,  w*)  defined by (15) is again an equilibrium of  (29) due to (30). Theorem3:  Suppose the hypotheses of  Theorem  1 hold: suppose further (30) holds. Then the equilibrium (U*,  U*) of (29)  is  globally asymptotically stable in  the  sense that  all solutions of  (29)-(31)  satisfy Proof:  Details of proof are similar to those of Theorem 2 and hence we shall to brief. As before we define xi,  yi by (22) and derive from (29)-(30) that zi and yi  satisfy the system which as before leads to i = 1,2,...,n.  (34) We  consider a Lyapunov type functional V(t)  = V(x,y)(t) defined by i=l (35) By  our assumption on the initial values on (-m,O]  and the hypotheses on  the  delay kemels,  one  can  verify  that  V  is defined on (0,m) and V is bounded on (0, m).  Also the upper right derivative D+V  of V along the solutions of  (33) can be calculated so  that n D+V(t)  I  -(I -  c)  [IZi(t)l + IYZ(t)ll.  (36) i=l The  remaining  details  of  proof  are  identical  to  these  of Theorem 2 and hence are omitted. The proof  is complete. IV.  REMARKS Recall of  memories is one of  the processes by  which the brain retums in some sense from a current state to another state in which it has been before. In neural network models, memory corresponds to a temporally stationary or nonstationary equi- librium and recall is modeled by the convergence of  neuronal activations in the neuronal activation space to the equilibrium. The trigger provided for the system to recall the memory may come from outside as extemal inputs. Thus the patterns or equilibria associated with extemal inputs are recalled by  the convergence of  system dynamics; global asymptotic stability of  an  equilibrium means that  the  recall  is  “perfect” in  the sense no hints or guesses are needed as in the case of  local stability analyses; that is when the extemal inputs are provided to  the  system, irrespective of  the  initial values, the  system converges to the equilibrium associated with the inputs. Recall 1002  IEEE  TRANSACTIONS  ON  NEURAL NETWORKS,  VOL.  5, NO. 6, NOVEMBER  1994 with the help of hints and guesses correspond to local stability of  equilibria; since the initial values have to be  in a suitable neighborhood of  the corresponding equilibrium. It  is  known  (see  [12]) that  time  delays  in  response  or transmission can induce sustained oscillations and “chaos”. We have discussed the stability characteristics of an already trained bidirectional not  necessarily symmetric associative network with transmission delays and obtained sufficient conditions for [5]  L.  E.  El’sgolt’s  and  S. B.  Norkin, Introduction  to The Theory and Application of Differential Equations with Deviating Arguments.  New York  Academic Press,  1973. [6]  K.  Gopalsamy, Stabilify and  Oscillations  in Delay  DifSerential Equa- tions of  Population  Dynamics.  Dordrecht, The Netherlands: Kluwer Academic,  1992. 171  K.  Gopalsamy and  X. Z.  He, “Stability in asymmetric Hopfield nets with transmission delays,” Physica D (to appear). tional properties like those of  two state neurons,” P  roc. Nut. Acad. Sci. 181  J.  Hopfield, “Neurons with graded response have collective computa- the absence of delay induced persistent oscillations or chaos.  USA, vol. 81, pp.  3088-3092,  1984. In a forthcoming article  we discuss the dynamics of associative recall in  a bidirectional adaptive network with delays in  the learning dynamics. [9]  B. Kosko, “Bidirectional associative memories,” IEEE. Trans. Syst. Man and Cybernetics, vol. SMC-18, pp. 49-60, 1988. vol.  26 pp. 49474960,  1987. vol.  1,  pp.  44-57,  1990. 1101 -  , “Adaptive bidirectional associative memo,.ies,” ~~~1.  Oprjcs., [  111  -,  “Unsupervised learning in  noise,” IEEE. Trans. Neural  Net., 1121  C. M.  Marcus  and  R.  M.  Westervelt, “Dvnamics  of  analoe.  neural  REFERENCES [l] K. L. Babcock and R.  M.  Westervelt, “Dynamics of  simple electronic neural networks,” Physica,, vol. 28D, pp.  305-316,  1987. [2]  I. Barbalat, “Systemes d’equations differentielle d’oscillations nonlin- eaires,” Rev. Roumaine Math. Pures Appl., vol. 4, pp. 267-270,  1959. [3]  M.  A.  Cohen and S. Grossberg, “Absolute stability and global pattern formation and parallel memory storage by competitive neural networks,” IEEE. Trans. Syst. Man. and Cybernetics, vol. SMC-13 ,  pp. 815-821, 1983. [4]  E.  Domany, J. L. van Hemmen, and K. Schulten, ed., Models of  Neural Networks.  Berlin, Germany: Springer-Verlag, 1991. ~~ networks with time delay,” in Advances in Neural Information Proc., San Menlo, CA, 1989, pp.  568-576. 1131  -,  “Stability of  analog neural networks with  time delay,” Phys. Rev. A, vol. 39, pp. 347-359,  1989. 1141  C. M. Marcus, F.  R.  Waugh, and  R.  M.  Westervelt, “Nonlinear dy- namics and stability of analog neural networks,” Physica, vol. 51D, pp. 234-247,  1991. [  151  D. W. Tank and J. J. Hopfield, “Neural computation by  concentrating information in time,” Proc. Nut. Acad. Sci. USA,  vol. 84, pp.  1896-1991, 1987. I. 

Absolute stability and global pattern formation and parallel memory storage by competitive neural networks,”

Dynamics of simple electronic neural networks,”

Models of Neural Networks.

Systemes d’equations differentielle d’oscillations nonlineaires,” Rev.

Delay-independent stability in bidirectional associative memory networks

http://dspace.flinders.edu.au/xmlui/bitstream/handle/2328/25804/Gopalsamy%20Delay-independent.pdf?sequence=1

Delay-independent stability in bidirectional associative memory networks

Abstract

Similar works

Full text

Available Versions

Flinders Academic Commons