Tatar, Nasser-eddine

Repository of the Academy's Library

Miskolc Mathematical Notes HU e-ISSN 1787-2413Vol. 19 (2018), No. 1, pp. 631–640 DOI: 10.18514/MMN.2018.1175NEURAL NETWORKS WITH DISTRIBUTED DELAYS ANDHO¨LDER CONTINUOUS ACTIVATION FUNCTIONSNASSER-EDDINE TATARReceived 21 March, 2014Abstract. We consider a system which arises in Neural Network Theory with distributed delaysinvolving Ho¨lder continuous activation functions. We prove some results on global exponentialstability of the system. This extends the previous works where activation functions were assumedto be Lipschitz continuous.2010 Mathematics Subject Classification: 24G20; 34C11; 92B20Keywords: neural network, distributed delay, exponential stability, Ho¨lder continuity, non-Lipschitzcontinuity1. INTRODUCTIONIn this work we investigate the following systemx0i .t/D ai .t/xi .t/CmXjD1bijZ t 1Kij .t   s/fj xj .s/dsC ci .t/; i D 1; :::;m(1.1)for t > 0; with xj .t/D x0j .t/, t 2 . 1;0 where x0j .t/ are given continuous func-tions and bij are real constants. The functions ai .t/ (nonnegative); ci .t/; i D 1; :::;mare continuous functions and fj are the activation functions. The functions Kij .t/are assumed to be continuous and integrable.This system arises in (Artificial) Neural Network theory [7, 8] in which there isa growing interest these last two decades. Unlike the conventional machines (basedon von Neumann architecture) which use a single processor, the Neural Networkconsists of a large number of processors (arranged in layers) and in general a lar-ger number of interconnections between them. A processor receives inputs from thepreceding layer, process it and then pass it to the subsequent layer. There are variousapplications in: engineering, science, biology, finance, medicine and geology. NeuralNetworks are used to understand (complex) phenomena in these fields. They are ableThe author is grateful for the financial support and the facilities provided by King Fahd Universityof Petroleum and Minerals through project No. IN121044.c 2018 Miskolc University Press632 NASSER-EDDINE TATARto classify and recognize patterns. They are also used to solve mathematical pro-gramming problems. Their advantages over the traditional computers are the abilityto perform huge parallel computations, to classify certain products and materials andto predict certain phenomena.There are many papers in the literature dealing with the local or global asymptoticstability of the system. In particular, we are witnessing a lot of interest in exponentialstability of the (unique) equilibrium of the system for this problem and other similarones [2, 3, 15–18, 21, 23, 25, 26, 28] to cite but a few. A lot of efforts are devoted inimproving the set of conditions on the different coefficients involved in the system aswell as the class of activation functions. For the coefficients the main conditions turnaround a kind of dominance of the ”dissipation” coefficients ai on the other coeffi-cients. As for the activation functions the first researchers have dealt with specificexplicit ones. Then they moved to the assumptions: monotonicity, differentiabilityand boundedness. These conditions were later weakened to a Lipschitz condition.After that not much has been done for continuous but not Lipschitz continuous activ-ation functions compared to the Lipschitz case or even the discontinuous case. Non-Lipschitz continuous activation functions arise in many fields [10], see also [6] for anapplication involving Ho¨lder continuous functions. For a slightly weaker conditionor partial Lipschitz condition we refer the reader to [3, 4, 23, 25].Ho¨lder continuous activation functions were studied by Forti et al. in [4] for theproblemx0 D BxCTg.x/CI:The authors assumed that each component of g is bounded and is a non-decreasingpiecewise continuous function. The matrix  T is assumed Lyapunov DiagonallyStable. An exponential stability result and a finite time convergence result is provedthere and extended to a larger class of non-Lipschitz continuous activation functionsunder a stronger condition. In addition to that the authors investigated discontinuousactivation functions using the theory of Filippov. In fact discontinuous activationfunctions have been treated in several other papers (see [1,5,9,12–14,19,20,22,24])first under some boundedness and monotonicity conditions. Later these conditionwere dropped under some other conditions.Here we consider variable coefficients and activation functions fj that are Ho¨ldercontinuous i.e. ˇˇfj .x/ fj .y/ˇˇ Lj jx yj j˛ ; 0 < j˛ < 1 (1.2)for some positive constants Lj ; j D 1; :::;m: Clearly, Ho¨lder continuous functionsare not necessarily Lipschitz continuous. We prove global exponential stability ofthe system. This is achieved with the help of a Gronwall-type inequality due to E. H.Yang [27], some appropriate estimates and some Lyapunov-type functionals.The local existence is standard whereas the global existence may be derived using(2.2) and the Gronwall-type Lemma 1 below.In the next section we state and prove our results.NEURAL NETWORKS WITH DISTRIBUTED DELAYS 6332. EXPONENTIAL STABILITYIn this section it is proved that the system is globally asymptotically stable in anexponential manner when the activation functions (all or some of them) are onlyHo¨lder continuous.Definition 1. We say that the system (1.1) is globally asymptotically stable if forany two solutions xi .t/ and Nxi .t/ we havelimt!1XmiD1 jxi .t/  Nxi .t/j D 0:It is said to be exponentially asymptotically stable if there exist two positive constantsM and  such that XmiD1 jxi .t/  Nxi .t/j Me t ; t > 0:In case the coefficients in problem (1.1) are constant and there exists a uniqueequilibrium xi ; i D 1; :::;m, that is a solution of0D aixi CmXjD1bijfjxjZ 10Kij .s/dsC ci ; i D 1; :::;m: (2.1)then we obtain exponential stability of this equilibrium.We denote by yi .t/D xi .t/  Nxi .t/, y.t/DmPiD1jyi .t/j, a.t/ WDmin1im fai .t/g,1.t/ WD a.t/ mXi;jD1Ljˇˇbijˇˇpjj .t/Z 10ˇˇKij .s/ˇˇeR tCst a./dds; t  0and1.t/ WDmXi;jD1LjˇˇbijˇˇqjZ t 1ˇˇKij .t   s/ˇˇ qj =pjj .s/ds; t  0for some positive continuous functions j .t/; pj D 1= j˛ and qj D 1=.1  j˛ /; j D1; :::;m:Theorem 1. Assume that fj , j D 1; :::;m are Ho¨lder continuous and Kij .t/,i;j D 1; :::;m are continuous and integrable functions. Assume further that ai .t/;bij .t/, Lj ; i;j D 1; :::;m and the continuous functions i .t/ > 0 are such that1.t/ 0 (not identically zero),R t0 1./d !1 as t !1 andZ t01.s/eR s0 1./ddsis at most of polynomial growth P.t/. Then, the system (1.1) globally asymptoticallystable at the rate P.t/e R t0 1.s/ds; that isy.t/ P.t/e R t0 1.s/ds; t > 0:634 NASSER-EDDINE TATARProof. From the system (1.1), the assumptions (1.2), (2.2) and the adopted nota-tion we getDC jyi .t/j   ai .t/ jyi .t/jCmXjD1Ljˇˇbijˇˇ Z t 1ˇˇKij .t   s/ˇˇ ˇˇyj .s/ˇˇj˛ ds; t > 0;for i D 1; :::;m; where DC denotes the right Dini derivative. Therefore,DCy.t/  min1imfai .t/gy.t/CmXi;jD1Ljˇˇbijˇˇ Z t 1ˇˇKij .t   s/ˇˇ ˇˇyj .s/ˇˇj˛ ds; t > 0Hence, as we denote a.t/ WD min1im fai .t/g ; we getDCy.t/  a.t/y.t/CmXi;jD1Ljˇˇbijˇˇ Z t 1ˇˇKij .t   s/ˇˇ ˇˇyj .s/ˇˇj˛ ds; t > 0: (2.2)We estimate jy.t/j˛i ; 0 < ˛i < 1; i D 1; :::;m using Young’s inequality as followsˇˇyj .t/ˇˇj˛  j .t/pjˇˇyj .t/ˇˇC qj =pjj .t/qj; j .t/ > 0;1pjC 1qjD 1;j D 1; :::;m; t  0(2.3)with pj D 1= j˛ and qj D 1=.1  j˛ /; j D 1; :::;m: Plugging these relations (2.3) in(2.2) we findDCy.t/  a.t/y.t/CmXi;jD1Ljˇˇbijˇˇ Z t 1ˇˇKij .t   s/ˇˇ24j .s/pjˇˇyj .s/ˇˇC  qj =pjj .s/qi35ds  a.t/y.t/CmXi;jD1LjˇˇbijˇˇpjZ t 1ˇˇKij .t   s/ˇˇj .s/ˇˇyj .s/ˇˇdsCmXi;jD1LjˇˇbijˇˇqjZ t 1ˇˇKij .t   s/ˇˇ qj =pjj .s/ds; t > 0: (2.4)Now we introduce the following functionalV1.t/ WD y.t/C˚1.t/; t  0 (2.5)where˚1.t/ WDmXi;jD1Ljˇˇbijˇˇpje R t0 a.s/dsZ 10ˇˇKij .s/ˇˇ Z tt seR Cs0 a./dj ./ˇˇyj ./ˇˇdds:NEURAL NETWORKS WITH DISTRIBUTED DELAYS 635This functional ˚1.t/ will help us get rid of the delayed terms. Indeed, by differen-tiation it is easy to see that˚ 01.t/ WD  a.t/˚1.t/CmXi;jD1Ljˇˇbijˇˇpje R t0 a.s/dsZ 10ˇˇKij .s/ˇˇ heR tCs0 a./dj .t/ˇˇyj .t/ˇˇ  eR t0 a./dj .t   s/ ˇˇyj .t   s/ˇˇidsD a.t/˚1.t/CmXjD1"mXiD1LjˇˇbijˇˇpjZ 10ˇˇKij .s/ˇˇeR tCst a./dds#j .t/ˇˇyj .t/ˇˇ mXi;jD1LjˇˇbijˇˇpjZ 10ˇˇKij .s/ˇˇj .t   s/ˇˇyj .t   s/ˇˇds; t > 0 (2.6)where the last term is equal to the second term in the right hand side of (2.4) with anopposite sign. So summing up (2.4) and (2.6) we getDCV1.t/  a.t/y.t/ a.t/˚1.t/CmXi;jD1LjˇˇbijˇˇpjZ t 1ˇˇKij .t   s/ˇˇj .s/ˇˇyj .s/ˇˇdsCmXi;jD1LjˇˇbijˇˇqjZ t 1ˇˇKij .t   s/ˇˇ qj =pjj .s/dsCmXjD1"mXiD1LjˇˇbijˇˇpjZ 10ˇˇKij .s/ˇˇeR tCst a./dds#j .t/ˇˇyj .t/ˇˇ mXi;jD1LjˇˇbijˇˇpjZ 10ˇˇKij .s/ˇˇj .t   s/ˇˇyj .t   s/ˇˇds  24a.t/  mXi;jD1Ljˇˇbijˇˇpjj .t/Z 10ˇˇKij .s/ˇˇeR tCst a./dds35V1.t/CmXi;jD1LjˇˇbijˇˇqjZ t 1ˇˇKij .t   s/ˇˇ qj =pjj .s/ds; t > 0: (2.7)With our notation we can rewrite (2.7) asDCV1.t/  1.t/V1.t/C1.t/; t > 0:We derive thatDCnV1.t/eR t0 1.s/dso 1.t/eR t0 1.s/ds; t > 0:636 NASSER-EDDINE TATARA standard comparison theorem (see Theorem 1.4.1 in [11]) implies thatV1.t/eR t0 1.s/ds  V1.0/CZ t01.s/eR s0 1./dds; t > 0orV1.t/ V1.0/e R t0 1.s/dsC e R t0 1.s/dsZ t01.s/eR s0 1./dds; t > 0:The assumption that Z t01.s/eR s0 1./ddsis at most of polynomial growth implies that V1.t/ that and therefore y.t/ is decayingexponentially to zero. The following lemma due to E. H. Yang [27] will be needed in our next result.Lemma 1. Let u.t/ and fi .t/ i D 1; :::;n be non-negative continuous functionson an interval J D Œ0;T /; 0 < T  0 and a.t/ a non-decreasing function such thata.t/ 1 for t 2 J: Ifu.t/ a.t/CnXiD1Z t0fi .s/.u.s//ri ds; t 2 J; 0 < ri < 1thenu.t/ a.t/YniD1Gi .t/; t 2 JwhereGi .t/ WD1C .1  ri /Yi 1kD1Gk.t/Z t0fi .s/ds 12 ri; i D 1; :::;nandQ0iD1Gi .t/D 1; t 2 J:In our present situation we will useGi .t/ WD1C .1 ˛i /Yi 1kD1Gk.t/Z t0Qbi .s/ds 12 ˛i; i D 1; :::;mwhereQbi .t/ WD"mXiD1Ljˇˇbijˇˇ Z 10ˇˇKij .s/ˇˇeR sCt0 a./dds#e  j˛R t0 a./d :Theorem 2. Assume that fj are Ho¨lder continuous i.e. satisfy the relations (1.2),j D 1; :::;m, Kij .t/, i;j D 1; :::;m are continuous and integrable functions,NEURAL NETWORKS WITH DISTRIBUTED DELAYS 637QmiD1Gi .t/ grows at most as a polynomial andR t0 a.s/ds !1 as t !1: Then,we havey.t/ Œ1Cy.0/YmiD1Gi .t/exp Z t0a.s/ds; t > 0:Proof. We start from the inequality (see (2.2))DCy.t/  a.t/y.t/CmXi;jD1Ljˇˇbijˇˇ Z t 1ˇˇKij .t   s/ˇˇ ˇˇyj .s/ˇˇj˛ ds; t > 0: (2.8)Consider the new functionalV2.t/ WD y.t/C˚2.t/; t  0 (2.9)where˚2.t/ WDmXi;jD1Ljˇˇbijˇˇe R t0 a.s/dsZ 10ˇˇKij .s/ˇˇ Z tt seR sC0 a./dˇˇyj ./ˇˇj˛ d:(2.10)A differentiation of ˚2.t/ in (2.10) gives˚ 02.t/D a.t/˚2.t/CmXi;jD1Ljˇˇbijˇˇe R t0 a.s/dsZ 10ˇˇKij .s/ˇˇeR sCt0 a./ddsˇˇyj .t/ˇˇj˛ mXi;jD1Ljˇˇbijˇˇ Z 10ˇˇKij .s/ˇˇ ˇˇyj .t   s/ˇˇj˛ ds; t  0 (2.11)and from (2.8)-(2.11) we see thatDCV2.t/  a.t/y.t/ a.t/˚2.t/CmXi;jD1Ljˇˇbijˇˇ Z t 1ˇˇKij .t   s/ˇˇ ˇˇyj .s/ˇˇj˛ dsCmXi;jD1Ljˇˇbijˇˇe R t0 a.s/dsZ 10ˇˇKij .s/ˇˇeR sCt0 a./ddsˇˇyj .t/ˇˇj˛ mXi;jD1Ljˇˇbijˇˇ Z 10ˇˇKij .s/ˇˇ ˇˇyj .t   s/ˇˇj˛ ds  a.t/V2.t/CmXjD1"mXiD1Ljˇˇbijˇˇe R t0 a.s/dsZ 10ˇˇKij .s/ˇˇeR sCt0 a./dds#V j˛2 .t/; t > 0:ThusDCV2.t/expZ t0a.s/ds638 NASSER-EDDINE TATARmXjD1"mXiD1Ljˇˇbijˇˇ Z 10ˇˇKij .s/ˇˇeR sCt0 a./dds#V j˛2 .t/mXjD1"mXiD1Ljˇˇbijˇˇ Z 10ˇˇKij .s/ˇˇeR sCt0 a./dds#exp  j˛Z t0a.s/dsV2.t/expZ t0a.s/dsj˛; t > 0and by comparison and integrationV2.t/expZ t0a.s/ds  V2.0/CZ t08<:mXjD1"mXiD1Ljˇˇbijˇˇ Z 10ˇˇKij ./ˇˇeR Cs0 a./dd#e  j˛R s0 a./dV2.s/expZ s0a./dj˛ds; t > 0:In shortQV2.t/ V2.0/CmXjD1Z t0Qbj .s/ QV2.s/ j˛ ds; t > 0where QV2.t/ WD V2.t/expR t0 a.s/ds andQbi .t/ WD"mXiD1Ljˇˇbijˇˇ Z 10ˇˇKij ./ˇˇeR sCt0 a./dd#e  j˛R t0 a./d ; i D 1; :::;m:Therefore we can apply Lemma 1 to obtainQV2.t/ Œ1CV2.0/YmjD1Gj .t/whereGj .t/ WD1C .1  j˛ /Yj 1kD1Gk.t/Z t0Qbj .s/ds 12  j˛; j D 1; :::;mandQ0iD1Gi .t/D 1; t  0: The proof is complete. Remark 1. The coefficients bij being assumed constants is only a technical as-sumption to avoid duplication of an undesirable term in the derivative of ˚2.t/.REFERENCES[1] G. Bao and Z. Zeng, “Analysis and design of associative memories based on recurrent neuralnetwork with discontinuous activation functions,” Neurocomputing, vol. 77, no. 1, pp. 101–107,2012, doi: 10.1016/j.neucom.2011.08.026.[2] J. Cao, K. Yuan, and H.-X. Li, “Global asymptotical stability of recurrent neural networks withmultiple discrete delays and distributed delays,” IEEE Trans. Neural Netw., vol. 17, no. 6, pp.1646–1651, 2006, doi: 10.1109/tnn.2006.881488.NEURAL NETWORKS WITH DISTRIBUTED DELAYS 639[3] C. Feng and R. Plamondon, “On the stability analysis of delayed neural networks systems,” NeuralNetworks, vol. 14, no. 9, pp. 1181–1188, 2001, doi: 10.1016/s0893-6080(01)00088-0.[4] M. Forti, M. Grazzini, P. Nistri, and L. Pancioni, “Generalized Lyapunov approach for conver-gence of neural networks with discontinuous or non-Lipschitz activations,” Physica D: NonlinearPhenomena, vol. 214, no. 1, pp. 88–99, 2006, doi: 10.1016/j.physd.2005.12.006.[5] M. Forti and P. Nistri, “Global convergence of neural networks with discontinuous neuron activ-ations,” IEEE Trans. Circuits Syst.-I: Fund. Theory Appl., vol. 50, no. 11, pp. 1421–1435, 2003,doi: 10.1109/tcsi.2003.818614.[6] R. Gavalda` and H. T. Siegelmann, “Discontinuities in recurrent neural networks,” Neural Comput.,vol. 11, no. 3, pp. 715–745, 1999, doi: 10.1162/089976699300016638.[7] J. J. Hopfield, “Neural networks and physical systems with emergent collective compu-tational abilities,” Proc. Nat. Acad. Sci., vol. 79, no. 8, pp. 2554–2558, 1982, doi:10.1142/9789812799371-0043.[8] J. J. Hopfield, D. W. Tank et al., “Computing with neural circuits- a model,” Science, vol. 233, no.4764, pp. 625–633, 1986, doi: 10.1126/science.3755256.[9] Y. Huang, H. Zhang, and Z. Wang, “Dynamical stability analysis of multiple equilibrium points intime-varying delayed recurrent neural networks with discontinuous activation functions,” Neuro-computing, vol. 91, pp. 21–28, 2012, doi: 10.1016/j.neucom.2012.02.016.[10] B. Kosko, Neural Network and Fuzzy System - A Dynamical System Approach to Machine Intelli-gence. Prentice-Hall, New-Delhi,, 1991.[11] V. Lakshmikantham and S. Leela, Differential and integral inequalities. Academic press, 1969.[12] L. Li and L. Huang, “Dynamical behaviors of a class of recurrent neural networks with dis-continuous neuron activations,” Appl. Math Model., vol. 33, no. 12, pp. 4326–4336, 2009, doi:10.1016/j.apm.2009.03.014.[13] J. Liu, X. Liu, and W.-C. Xie, “Global convergence of neural networks with mixed time-varyingdelays and discontinuous neuron activations,” Information Sciences, vol. 183, no. 1, pp. 92–105,2012, doi: 10.1016/j.ins.2011.08.021.[14] X. Liu and J. Cao, “Robust state estimation for neural networks with discontinuous activations,”IEEE Trans. Syst., Man, Cyb.-Part B: Cybernetics, vol. 40, no. 6, pp. 1425–1437, 2010, doi:10.1109/tsmcb.2009.2039478.[15] X. Liu and N. Jiang, “Robust stability analysis of generalized neural networks with multiple dis-crete delays and multiple distributed delays,” Neurocomputing, vol. 72, no. 7, pp. 1789–1796,2009, doi: 10.1016/j.neucom.2008.06.005.[16] S. Mohamad, K. Gopalsamy, and H. Akca, “Exponential stability of artificial neural networkswith distributed delays and large impulses,” Nonlinear Anal., Real World Appl., vol. 9, no. 3, pp.872–888, 2008, doi: 10.1016/j.nonrwa.2007.01.011.[17] J. H. Park, “On global stability criterion of neural networks with continuously distributed delays,”Chaos, Solitons & Fractals Chaos Solitons & Fractals, vol. 37, no. 2, pp. 444–449, 2008, doi:10.1016/j.chaos.2006.09.021.[18] Z. Qiang, M. Run-Nian, and X. Jin, “Global exponential convergence analysis of Hopfield neuralnetworks with continuously distributed delays,” Commun. Theor. Phys., vol. 39, no. 3, pp. 381–384, 2003, doi: 10.1088/0253-6102/39/3/381.[19] S. Qin and X. Xue, “Global exponential stability and global convergence in finite time of neuralnetworks with discontinuous activations,” Neural Process Lett., vol. 29, no. 3, pp. 189–204, 2009,doi: 10.1007/s11063-009-9103-7.[20] J. Wang, L. Huang, and Z. Guo, “Global asymptotic stability of neural networks withdiscontinuous activations,” Neural Networks, vol. 22, no. 7, pp. 931–937, 2009, doi:10.1016/j.neunet.2009.04.004.640 NASSER-EDDINE TATAR[21] Y. Wang, W. Xiong, Q. Zhou, B. Xiao, and Y. Yu, “Global exponential stability of cellular neuralnetworks with continuously distributed delays and impulses,” Physics Letters A, vol. 350, no. 1,pp. 89–95, 2006, doi: 10.1016/j.physleta.2005.10.084.[22] Z. Wang, L. Huang, Y. Zuo, and L. Zhang, “Global robust stability of time-delay systems withdiscontinuous activation functions under polytopic parameter uncertainties,” Bull. Korean Math.Soc., vol. 47, no. 1, pp. 89–102, 2010, doi: 10.4134/bkms.2010.47.1.089.[23] H. Wu, “Global exponential stability of Hopfield neural networks with delays and inverseLipschitz neuron activations,” Nonlinear Anal., Real World Appl., vol. 10, no. 4, pp. 2297–2306,2009, doi: 10.1016/j.nonrwa.2008.04.016.[24] H. Wu, “Global stability analysis of a general class of discontinuous neural networks with lineargrowth activation functions,” Information Sciences, vol. 179, no. 19, pp. 3432–3441, 2009, doi:10.1016/j.ins.2009.06.006.[25] H. Wu, F. Tao, L. Qin, R. Shi, and L. He, “Robust exponential stability for interval neural networkswith delays and non-Lipschitz activation functions,” Nonlinear Dynamics, vol. 66, no. 4, pp. 479–487, 2011, doi: 10.1007/s11071-010-9926-9.[26] H.-F. Yanai and S.-I. Amari, “Auto-associative memory with two-stage dynamics of nonmonotonicneurons,” IEEE Trans. Neural Netw., vol. 7, no. 4, pp. 803–815, 1996, doi: 10.1109/72.508925.[27] E.-H. Yang, “Perturbations of nonlinear systems of ordinary differential equations,” J. Math. Anal.Appl., vol. 103, no. 1, pp. 1–15, 1984, doi: 10.1016/0022-247x(84)90151-3.[28] J. Zhou, S. Li, and Z. Yang, “Global exponential stability of Hopfield neural networkswith distributed delays,” Appl. Math. Model., vol. 33, no. 3, pp. 1513–1520, 2009, doi:10.1016/j.apm.2008.02.006.Author’s addressNasser-eddine TatarKing Fahd University of Petroleum and Minerals, Department of Mathematics and Statistics, Dhahran,31261 City, Saudi ArabiaE-mail address: tatarn@kfupm.edu.sa

Neural networks with delayed H\"{o}lder continuous activation functions

Abstract

Similar works

Full text

Available Versions

Repository of the Academy's Library