We study the asymptotic law of a network of interacting neurons when the
number of neurons becomes infinite. The dynamics of the neurons is described by
a set of stochastic differential equations in discrete time. The neurons
interact through the synaptic weights which are Gaussian correlated random
variables. We describe the asymptotic law of the network when the number of
neurons goes to infinity. Unlike previous works which made the biologically
unrealistic assumption that the weights were i.i.d. random variables, we assume
that they are correlated. We introduce the process-level empirical measure of
the trajectories of the solutions to the equations of the finite network of
neurons and the averaged law (with respect to the synaptic weights) of the
trajectories of the solutions to the equations of the network of neurons. The
result is that the image law through the empirical measure satisfies a large
deviation principle with a good rate function. We provide an analytical
expression of this rate function in terms of the spectral representation of
certain Gaussian processes

Faugeras, Olivier

MacLaurin, James

English

arXiv

International audienceWe study the asymptotic law of a network of interacting neurons when the number ofneurons becomes infinite. The dynamics of the neurons is described by a set of stochasticdifferential equations in discrete time. The neurons interact through the synaptic weightsthat are Gaussian correlated random variables. We describe the asymptotic law of thenetwork when the number of neurons goes to infinity. Unlike previous works which madethe biologically unrealistic assumption that the weights were i.i.d. random variables, weassume that they are correlated. We introduce the process-level empirical measure of thetrajectories of the solutions into the equations of the finite network of neurons and theaveraged law (with respect to the synaptic weights) of the trajectories of the solutions intothe equations of the network of neurons. The result (Theorem 3.1 below) is that the imagelaw through the empirical measure satisfies a large deviation principle with a good ratefunction. We provide an analytical expression of this rate function in terms of the spectralrepresentation of certain Gaussian processes.Nous considérons un réseau de neurones décrit par un système d’équations différentiellesstochastiques en temps discret. Les neurones interagissent au travers de poids synaptiquesqui sont des variables aléatoires gaussiennes corrélées. Nous caractérisons la loi asymptotiquede ce réseau lorsque le nombre de neurones tend vers l’infini. Tous les travauxprécédents faisaient l’hypothèse, irréaliste du point de vue de la biologie, de poidsindépendants. Nous introduisons la mesure empirique sur l’espace des trajectoires solutionsdes équations du réseau de neurones de taille finie et la loi moyennée (par rapport auxpoids synaptiques) des trajectoires de ces solutions. Le résultat (théorème 3.1 ci-dessous)est que l’image de cette loi par la mesure empirique satisfait un principe de grandesdéviations avec une bonne fonction de taux, dont nous donnons une expression analytiqueen fonction de la représentation spectrale de certains processus gaussiens

Maclaurin, James

INRIA a CCSD electronic archive server

HAL Id: hal-01074827https://hal.inria.fr/hal-01074827Submitted on 15 Oct 2014HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Asymptotic description of stochastic neural networks. I.Existence of a large deviation principleOlivier Faugeras, James MaclaurinTo cite this version:Olivier Faugeras, James Maclaurin. Asymptotic description of stochastic neural networks. I. Existenceof a large deviation principle. Comptes rendus de l’Académie des sciences. Série I, Mathématique,Elsevier, 2014, 352, pp.841 - 846. ￿10.1016/j.crma.2014.08.018￿. ￿hal-01074827￿Asymptotic description of stochastic neuralnetworks. I - existence of a Large DeviationPrincipleOlivier Faugeras a, James Maclaurin aaInria Sophia-Antipolis MéditerranéeAbstractWe study the asymptotic law of a network of interacting neurons when the numberof neurons becomes infinite. Given a completely connected network of neurons inwhich the synaptic weights are Gaussian correlated random variables, we describethe asymptotic law of the network when the number of neurons goes to infinity.Unlike previous works which made the biologically unplausible assumption thatthe weights were i.i.d. random variables, we assume that they are correlated. Weintroduce the process-level empirical measure of the trajectories of the solutions tothe equations of the finite network of neurons and the averaged law (with respectto the synaptic weights) of the trajectories of the solutions to the equations ofthe network of neurons. The result is that the image law through the empiricalmeasure satisfies a large deviation principle with a good rate function. We providean analytical expression of this rate function.RésuméDescription asymptotique de réseaux de neurones stochastiques. I -existence d’un principe de grandes déviationÉtant donné un réseau complètement connecté de neurones dans lequel les poidssynaptiques sont des variables aléatoires gaussiennes corrélées, nous caractérisonsla loi asymptotique de ce réseau lorsque le nombre de neurones tend vers l’infini.Tous les travaux précédents faisaient l’hypothèse, irréaliste du point de vue de labiologie, de poids indépendants. Nous introduisons la mesure empirique sur l’espacedes trajectoires solutions des équations du réseau de neurones de taille finie et laloi moyennée (par rapport aux poids synaptiques) des trajectoires de ces solutions.Le résultat est que la loi image de cette loi par la mesure empirique satisfait unprincipe de grandes déviations avec une bonne fonction de taux dont nous donnonsune expression analytique.Email address: firstname.name@inria.fr (Olivier Faugeras).Preprint submitted to the Académie des sciences 17 avril 2014Version française abrégéeNous considérons le problème de décrire la dynamique asymptotique d’un en-semble de 2n + 1 neurones lorsque ce nombre tend vers l’infini. Ce problèmeest motivé par un désir de parcimonie, par celui de rendre compte de l’appari-tion de phénomènes émergents, ainsi que par celui de comprendre les effets detaille finie. Nous considérons donc un réseau de 2n+1 neurones interconnectésdont la dynamique commune (en temps discret) obéit aux équations stochas-tiques (2). Dans celles-ci apparaissent les poids synaptiques ou coefficients decouplage notés Jnij qui sont des variables aléatoires gaussiennes corrélées. Pourrépondre à la question posée nous considérons la loi notée QVn de la solution à(2) moyennée par rapport aux poids synaptiques ou plus précisément l’imageΠn de cette loi par la mesure empirique (1). Nous montrons que cette loi satis-fait un principe de grande déviations avec une bonne fonction de taux H dontnous donnons une expression analytique dans la définition 3.1 et les équations(9) et (12). Ce travail généralise au cas des poids synaptiques corrélés ce-lui d’auteurs comme Sompolinsky [10] et Moynot et Samuelides [8] qui ontconsidéré le cas de poids synaptiques indépendants. Dans ce cas, plus simpled’un point de vue mathématique, mais beaucoup moins plausible d’un pointde vue biologique, on observe le phénomène de propagation du chaos. Nousmontrons dans un second article [4] que la bonne fonction de taux a un mini-mum unique que nous caractérisons complètement. La propagation du chaosn’a pas lieu mais la représentation est parcimonieuse dans un sens défini dans[4].1 Introduction1.1 Neural networksOur goal is to study the asymptotic behaviour and large deviations of a net-work of interacting neurons when the number of neurons becomes infinite.Sompolinsky also succesfully explored this particular topic [10] for fully con-nected networks of neurons. In his study of the continuous time dynamics ofnetworks of rate neurons, Sompolinsky and his colleagues assumed that thesynaptic weights in neuroscience, were random variables i.i.d. with zero meanGaussian laws. The main result obtained by Sompolinsky and his colleagues(using the local chaos hypothesis) under the previous hypotheses is that theaveraged law of the neurons dynamics is chaotic in the sense that the aver-aged law of a finite number of neurons converges to a product measure as thesystem gets very large.2The next efforts in the direction of understanding the averaged law of neu-rons are those of Cessac, Moynot and Samuelides [1,7,8,2,9]. From the tech-nical viewpoint, the study of the collective dynamics is done in discrete time.Moynot and Samuelides obtained a large deviation principle and were able todescribe in detail the limit averaged law that had been obtained by Cessacusing the local chaos hypothesis and to prove rigorously the propagation ofchaos property.One of the next challenges is to incorporate in the network model the fact thatthe synaptic weights are not independent and in effect often highly correlated.The problem we solve in this paper is the following. Given a completelyconnected network of neurons in which the synaptic weights are Gaussiancorrelated random variables, we describe the asymptotic law of the networkwhen the number of neurons goes to infinity. Like in [7,8] we study a discretetime dynamics but unlike these authors we cope with more complex intrinsicdynamics of the neurons.1.2 Mathematical frameworkFor some topological space Ω equipped with its Borelian σ-algebra B(Ω), wedenote the set of all probability measures by M(Ω). We equip M(Ω) withthe topology of weak convergence. For some positive integer n > 0, we letVn = {j ∈ ❩ : |j| ≤ n}. Let T = ❘T+1, for some positive integer T . Weequip T with the Euclidean topology, T ❩ with the cylindrical topology, anddenote the Borelian σ-algebra generated by this topology by B(T ❩). For someµ ∈ M(T ❩) governing a process (Xj)j∈❩, we let µVn ∈ M(T Vn) denote themarginal governing (Xj)j∈Vn . For some j ∈ ❩, let the shift operator Sj : T ❩ →T ❩ be S(ω)k = ωj+k. We let MS(T❩) be the set of all stationary probabilitymeasures µ on (T ❩,B(T ❩)) such that for all j ∈ ❩d, µ ◦ (Sj)−1 = µ. Let pn :T Vn → T ❩ be such that pn(ω)k = ωk mod Vn . Here, and throughout the paper,we take k mod Vn to be the element l ∈ Vn such that l = k mod (2n + 1).Define the process-level empirical measure µ̂n : TVn → MS(T ❩)asµ̂n(ω) =12n+ 1∑k∈VnδSkpn(ω). (1)Let (Y j) be a stationary Process on T such that the Y js are independent.Each Y j is governed by a law P , and we write the governing law in MS(T❩)as P❩. It is clear that the governing law over Vn may be written as P⊗Vn (thatis the product measure of P , indexed over Vn).The equation describing the time variation of the membrane potential U j of3the jth neuron writesU jt = γUjt−1 +∑i∈VnJnjif(Uit−1) + θj +Bjt−1, j ∈ Vn t = 1, . . . , T. (2)f : ❘ →]0, 1[ is a monotonically increasing Lipschitz continuous bijection. γis in [0, 1) and determines the time scale of the intrinsic dynamics of the neu-rons. The Bjt s are i.i.d. Gaussian random variables distributed as N1(0, σ2) 1 .They represent the fluctuations of the neurons’ membrane potentials. The θjsare i.i.d. as N1(θ̄, θ2). The are independent of the Bits and represent the cur-rent injected in the neurons. The U j0 s are assumed to be independent randomvariables with law µI .The Jnijs are the synaptic weights. Jnij represents the strength with which the‘presynaptic’ neuron j influences the ‘postsynaptic’ neuron i. They arise from astationary Gaussian random field specified by its mean and covariance functionE[Jnij] =J̄2n+ 1cov(JnijJnkl) =12n+ 1Λ ((k − i) mod Vn, (l − j) mod Vn) ,Λ is positive definite, let Λ̃ be the corresponding (positive) Fourier transform.We make the technical assumption that the series (Λ(i, j))i,j∈❩ is absolutelyconvergent to Λasum > 0 and convergent to Λsum > 0.We note Jn the (2n + 1) × (2n + 1) matrix of the synaptic weights, Jn =(Jnij)i,j∈Vn .The process (Y j) defined byY jt = γYjt−1 + θ̄ +Bjt−1, j ∈ Vn, t = 1, · · ·T, Yj0 = Uj0is stationary and independent. The law of each Y j is easily found to be givenbyP = (NT (0T , σ2IdT )⊗ µI) ◦Ψ,where Ψ : T → T is the following affine bijection. Writing v = Ψ(u), we definev0 = Ψ0(u) = u0vs = Ψs(u) = us − γus−1 − θ̄ s = 1, · · · , T.(3)We extend Ψ to a mapping T ❩ → T ❩ componentwise.The application Ψ defined in (3) plays a central role in the sequel we introducethe following definition.1. We note Np(m,Σ) the law of the p-dimensional Gaussian variable with meanm and covariance matrix Σ.4Définition 1.1 For each measure µ ∈ M(T Vn) or MS(T❩) we define µ to beµ ◦Ψ−1.We next introduce the following definitions.Définition 1.2 Let E2 be the subset of Ms(T❩) defined byE2 = {µ ∈ MS(T❩) |Eµ1,T [‖v0‖2] < ∞}.We define the process-level entropy to be, for µ ∈ MS(T❩)I(3)(µ, P❩) = limn→∞1(2n+ 1)I(2)(µVn , P⊗Vn).If µ /∈ E2, then I(3)(µ, P❩) = ∞. For further discussion of this rate function,and a proof that I(3) is well-defined, see [3].We note QVn(Jn) the element of M(T Vn) which is the law of the solutionto (2) conditioned on Jn. We let QVn = EJ [QVn(Jn)] be the law averagedwith respect to the weights. The reason for this is that we want to study theempirical measure µ̂n on path space. There is no reason for this to be a simpleproblem since for a fixed interaction Jn, the variables (U−n, · · · , Un) are notexchangeable. So we first study the law of µ̂n averaged over the interactions.Finally we introduce the image laws in terms of which the principal results ofthis paper are formulated.Définition 1.3 Let Πn and Rn in M(MS(T❩)) be the image laws of QVn andP⊗Vn through the function µ̂n : TVn → MS(T❩) defined by (1):Πn = QVn ◦ µ̂n Rn = P⊗Vn ◦ µ̂n2 The good rate functionWe obtain an LDP for the process with correlations (Πn) via the (simpler)process without correlations (Rn). To do this we obtain an expression for theRadon-Nikodym derivative of Πn with respect to Rn. This is done in propo-sitions 2.4 and 2.5. In equation (13) there appear certain Gaussian randomvariables defined from the right handside of the equations of the neuronaldynamics (2). Applying the Gaussian calculus to this expression we obtainequation (14) which expresses the Radon-Nikodym derivative as a function(depending on n) of the empirical measure (1). Using the fact that this func-tion is measurable we obtain equation (15). This equation is essential in a)finding the expression for the rate function H of definition 3.1, b) proving thelower-bound for Πn on the open sets, c) proving that the sequence (Πn) isexponentially tight.5The key idea is to associate to every stationary measure µ a certain stationaryGaussian process Gµ, or equivalently a certain Gaussian measure defined byits mean cµ and its covariance operator Kµ.Given µ inMS(T❩) we define a stationary Gaussian process Gµ, i.e. a measureQµ ∈ MS(T❩1,T ). For all i the mean of Gµ,it is given by cµt , wherecµt = J̄∫T ❩f(uit−1)dµ(u), t = 1, · · · , T , i ∈ ❩, (4)The covariance between the Gaussian vectors Gµ,i and Gµ,i+k is defined to beKµ,k = θ2δk1T†1T +∞∑l=−∞Λ(k, l)Mµ,l, (5)whereMµ,kst =∫T ❩f(u0s−1)f(ukt−1)dµ(u), (6)The above integrals are well-defined because of the definition of f and that theseries in (5) is convergent since the series (Λ(k, l))k, l∈❩ is absolutely convergentand the elements of Mµ,l are bounded by 1 for all l ∈ ❩.These definitions imply the existence of a Hermitian-valued spectral represen-tation for the sequence Mµ,k (resp. Kµ,k) noted M̃µ (resp. K̃µ) which satisfiesK̃µ(θ) = θ21Tt1T +12π∫ π−πΛ̃(θ,−ϕ)M̃µ(dϕ).We also use the partial sums, noted Kµ,k[n] , k ∈ Vn, in (5), to define anothersequence Aµ,k[n] which is used to define in the limit n → ∞Ãµ(θ) = K̃µ(θ)(σ2IdT + K̃µ(θ))−1. (7)We next define a functional Γ[n] = Γ[n],1 + Γ[n],2, which we use to characterisethe Radon-Nikodym derivative of Πn with respect to Rn. Let µ ∈ MS(T❩)andΓ[n],1(µ) = −12(2n+ 1)log(det(Id(2n+1)T +1σ2Kµ[n])), (8)where Kµ[n] the ((2n + 1)T × (2n + 1)T ) covariance matrix of the law Qµ,VnBecause of previous remarks the above expression has a sense. Taking thelimit when N → ∞ does not pose any problem and we can define Γ1(µ) =limn→∞ Γ[n],1(µ). The following lemma whose proof is again straightforwardindicates that this is well-defined.Lemma 2.1 When N goes to infinity the limit of (8) is given byΓ1(µ) = −14π∫ π−πlog(det(IdT +1σ2K̃µ(θ)))dθ (9)6for all µ ∈ M+1,S(T❩).It also follows easily from previous remarks thatProposition 2.1 Γ[n],1 and Γ1 are bounded below and continuous on MS(T❩).The definition of Γ[n],2(µ) is slightly more technical but follows directly fromproposition 2.4. For µ ∈ MS(T❩) letΓ[n],2(µ) =∫TVn1,Tφn(µ, v)µVn1,T(dv) (10)where φn : MS(T❩)× T Vn1,T → ❘ is defined byφn(µ, v) =12σ212n+ 1n∑j,k=−n†(vj−cµ)Aµ, k[n] (vk+j−cµ)+22n+ 1n∑j=−n〈cµ, vj〉−‖cµ‖2.(11)Γ[n],2(µ) is finite in the subset E2 ofMS(T❩) defined in definition 1.2. If µ /∈ E2,then we set Γ[n],2(µ) = ∞.We define Γ2(µ) = limN→∞ Γ[n],2(µ). The following proposition indicates thatΓ2(µ) is well-defined.Proposition 2.2 If the measure µ is in E2, i.e. if Eµ1,T [‖v0‖2] < ∞, thenΓ2(µ) is finite and writesΓ2(µ) =12σ2(12π∫ π−πÃµ(−θ) : ṽµ(dθ) + †cµ(Ãµ(0)− IdT )cµ+2Eµ1,T[tv0(IdT − Ãµ(0))cµ]). (12)The “:” symbol indicates the double contraction on the indexes.It is shown in [5] that φn(µ, v) defined by (11) is a continuous function of µwhich satisfiesφn(µ, v) ≥ −β2, β2 =T J̄22σ2Λsum(σ2 + θ2 + Λasum)By a standard argument we otain the following proposition.Proposition 2.3 Γ[n],2(µ) is lower-semicontinuous.We define Γ[n](µ) = Γ[n],1(µ) + Γ[n],2(µ). We may conclude from propositions2.1 and 2.3 that Γ[n] is lower-semicontinuous hence measurable.From these definitions it is relatively easy, and proved in [5], to show thatthe measure QVn is absolutely continuous with respect to P⊗Vn with a Radon-7Nikodym derivative which can be expressed as a function of the functionalΓ.Proposition 2.4 The Radon-Nikodym derivative of QVn with respect to P⊗Vnis given by the following expression.dQVndP⊗Vn(u) = Eexp1σ2∑j∈Vn〈Ψ1,T (uj), Gj〉 −12‖Gj‖2 , (13)the expectation being taken against the N T -dimensional Gaussian processes(Gi), i ∈ Vn given byGit =∑j∈VnJNij f(ujt−1), t = 1, · · · , T,and the function Ψ being defined by (3).Using standard Gaussian calculus we obtain the following proposition.Proposition 2.5 The Radon-Nikodym derivatives write asdQVndP⊗Vn(u) = exp(NΓ[n](µ̂n(u)), (14)dΠndRn(µ) = exp((2n+ 1)Γ[n](µ)). (15)Here µ ∈ MS(T❩), Γ[n](µ) = Γ[n],1(µ)+Γ[n],2(µ) and the expressions for Γ[n],1and Γ[n],2 have been defined in equations (8) and (10).3 The large deviation principleWe define the function H : (T ❩) → [0,+∞) as follows.Définition 3.1 Let H be the function M+1,S(T❩) → ❘ ∪ {+∞} defined byH(µ) =+∞ if I(3)(µ, P❩) = ∞I(3)(µ, P❩)− Γ(µ) otherwise,where Γ = Γ1 + Γ2.We state the following theorem.Theorem 3.1 Πn is governed by a large deviation principle with a good ratefunction H.8The proof is too long to be reproduced here, see [5]. We only give the generalstrategy. First we prove the lower bound on the open sets. For the upper boundon the closed sets, we simply avoid it by a) proving that (Πn) is exponentiallytight which allows us to b) restrict the proof of the upper bound to compactsets. The proof of b) is long and technical. It is built upon ideas found in [6].Note that we have found an analytical form for H through equations (9) and(12)References[1] B. Cessac, Increase in complexity in random neural networks, Journal dePhysique I (France), 5 (1995), pp. 409–432.[2] B. Cessac and M. Samuelides, From neuron to neural networks dynamics.,EPJ Special topics: Topics in Dynamical Neural Networks, 142 (2007), pp. 7–88.[3] J. Deuschel, D. Stroock, and H. Zessin, Microcanonical distributions forlattice gases, Communications in Mathematical Physics, 139 (1991).[4] O. Faugeras and J. Maclaurin, Asymptotic description of stochastic neuralnetworks. ii - characterization of the limit law, C. R. Acad. Sci. Paris, Ser. I,(2014).[5] O. Faugeras and J. MacLaurin, Large deviations of a spatially-ergodicneural network with learning, arxiv depot, INRIA Sophia Antipolis, 2014.[6] A. Guionnet, Dynamique de Langevin d’un verre de spins, PhD thesis,Université de Paris Sud, 1995.[7] O. Moynot, Etude mathématique de la dynamique des réseaux neuronauxaléatoires récurrents, PhD thesis, Université Paul Sabatier, Toulouse, 1999.[8] O. Moynot and M. Samuelides, Large deviations and mean-field theory forasymmetric random recurrent neural networks, Probability Theory and RelatedFields, 123 (2002), pp. 41–75.[9] M. Samuelides and B. Cessac, Random recurrent neural networks,European Physical Journal - Special Topics, 142 (2007), pp. 7–88.[10] H. Sompolinsky, A. Crisanti, and H. Sommers, Chaos in Random NeuralNetworks, Physical Review Letters, 61 (1988), pp. 259–262.9

Asymptotic description of stochastic neural networks. I. Existence of a large deviation principle

Numérisation de Documents Anciens Mathématiques

Olivier Faugeras

James Maclaurin

Comptes Rendus Mathématique

http://www.numdam.org/item/10.1016/j.crma.2014.08.018.pdf

Asymptotic description of stochastic neural networks. I - existence of a Large Deviation Principle

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server

Numérisation de Documents Anciens Mathématiques

Comptes Rendus Mathématique