Full article available at Springerlink: 

http://link.springer.com/chapter/10.1007%2F978-3-642-29347-4_17



DOI:

10.1007/978-3-642-29347-4_17In this work we study a simplified model of a neural activity flow in networks, whose connectivity is based on geometrical embedding, rather than being lattices or fully connected graphs. We present numerical results showing that as the spectrum (set of eigenvalues of adjacency matrix) of the resulting activity-based network develops a scale-free dependency. Moreover it strengthens and becomes valid for a wider segment along with the simulation progress, which implies a highly organised structure of the analysed graph

Piersa, Jarosław

Schreiber, Tomasz

English

Repository of Nicolaus Copernicus University

draftSpectra of the Spike-Flow Graphsin Geometrically Embedded Neural NetworksJaroslaw Piersa and Tomasz Schreiber ?piersaj[at]mat.umk.plFaculty of Mathematics and Computer Science, Nicolaus Copernicus University,PolandAbstract. In this work we study a simplified model of a neural activityflow in networks, whose connectivity is based on geometrical embedding,rather than being lattices or fully connected graphs. We present numeri-cal results showing that as the spectrum (set of eigenvalues of adjacencymatrix) of the resulting activity-based network develops a scale-free de-pendency. Moreover it strengthens and becomes valid for a wider segmentalong with the simulation progress, which implies a highly organisedstructure of the analysed graph.Keywords: geometric neural networks, graph spectrum, scale-freeness1 IntroductionThe spectrum of the graph is considered as an important characteristic of thegraph. A set of graph features can be easily obtained from the sole spectrumanalysis, for instance bi-partitioning, connectivity, a clustering coefficient [6] etc.While the spectrum does not provide a unique description of the graph up tothe isomorphism [6] it is graph-invariant.Throughout this paper, by graph spectrum we understand a set of eigenvaluesof the graph adjacency matrix A, namely set of λ ∈ C such that A · x = λ · x forsome vector x . Since A is symmetric, all eigenvalues are strictly real [6].In this work we set out to analyse activity graphs of the geometrically em-bedded neural networks. Clearly, one can distinguish between the structural (i.e.underlying) and the functional or spike-flow graphs (evolved during the dynam-ics) of the network [3]. Since the structural graph is strictly dependent on se-lected network model, we shall focus on the latter case, namely a spontaneouslyemerged subgraph. As it was presented in [10, 11], the resulting graph has a sig-nificantly different input degree distribution. Here we provide more insight intodifferences between predefined and resulting networks.? Deceased author, 1975-2010, Associate Professor at Faculty of Mathematics andComputer Science, NCU, researcher in the fields of probability, artificial intelligenceand physics.draftThe rest paper is organized as follows: we present the simplified activity flowmodel and the formal definition of the spike-flow graph in Sec. 2. Then we dis-cuss the results concerning an emergence of power-law or scale-free dependency(scaling as P(X = x) ∝ 1xk , where X is measured value and k > 1 is fixedparameter) in the spectrum of the obtained network in Sec. 3. Finally the workis concluded and potential aims of a future work are pointed in Sec. 4.2 Simulation ModelWe adopt the simplified model of neural activity coined in [10]. We argue, that itis valid for a modelling of activity in neurons and groups of neurons on corticallevel. The model might seem to have an abundance of degrees of freedom, butthey are mandatory, if the model is to exhibit an energy-driven self-organisationwhich is a feature of complex systems such as brain, see [5].Given a two-dimensional sphere S2 and expected density ρ  1 of neuronsin a square unit of the surface, we pick neurons from Poisson process on thesphere with intensity λ = |S2|ρ. Each neuron is given its Euclidean coordinates(xv, yv, zv) ∈ R3 accordingly to the process as well as an initial activity or chargeσv, which is stored in the unit. It should be interpreted as an abstract activitylevel of the neuron. In the model only non-negative integer values for charge areallowed σv ∈ N≥0. Starting configuration can be seen in Fig. 1.For every pair of neurons {u, v} a synaptic connection e = {u, v} is addedto the set of network’s edges E independently with probability P(e ∈ E) = g(e),where g() is a connectivity functiong({u, v}) ={d(u, v)−α d(u, v) ≥ 11 otherwise, (1)where d() is euclidean distance and α is the decay exponent approximately equalto the dimension of the embedding space (2 in our case), the formula of theconnectivity function was put forward in [7]. The synapse denotes a possibilityof direct interaction between connected units. Note, that formula (1) admits self-loops. Since the expected density of neurons ρ is large, with huge probability weobtain a connected network. The geometrical embedding of the network alongwith formula of g() result in varying lengths of the synapses. The number of shortor local ones is much greater, than of the long ones, which provide a connectionbetween the distant areas of the network. On the other hand a scale-free formulaof Eq. (1) ensures, that long synapses do not vanish too fast when the radius ofthe sphere increases, which would result in lattice-like structural network. Suchsystem seems to be more feasible to modelling of a real neural networks, thanbuilt on all-to-all connected graphs or regular grids.Each synapse, which was added to the graph, receives a gaussian weight wuv,which indicates its excitatory or inhibitory nature. The weight can be read as anaveraging over a number of factors, thus a gaussian distribution seems adequate.Positive weight indicates a tendency favour equal activity levels for connecteddraftneurons, while inhibitory synapse results in large differences (activity in firstunit keeps the second silent).Given a network activity configuration σ̄ = [σ1, ..., σN ] we define an energyof the system as follows: E(σ̄) = ∑(u,v)∈E wu,v|σv − σu| (2)The formula (2) bears similarity to the stochastic Boltzmann machine [1], al-though has been adjusted in order to account for any nonnegative value of theσv. Nonetheless, it still has the same interpretation of summarized interactioncost in the network and it still can be negative.The network undergoes its evolution according to the following dynamics:1. iterate many times:(a) randomly pick a pair of units u, v ∈ V, such that– {u, v} ∈ E ,– σu ≥ 1,(b) try to transfer one unit of charge from u to v (i.e. σu := σu − 1; σv :=σv + 1;),(c) if this reduces network energy, then accept the transfer,(d) otherwise accept it with probabilityP(u→ v) = exp(−β∆E), (3)where ∆E is an energy increase, which would be caused by the transfer,β > 0 is an inverse temperature and is assumed to be high (β  1),which results in rejecting most of jumps towards higher energy states.The transfer from u to v can be interpreted as spending some activity by u inorder to cause an excitation in v or (in the case of inhibitory synapse) inhibitingu by increasing activity in v. Note, that this activity-conserving dynamics mimicsa criticality state of the dynamics, i.e. the total activity in the network neithervanishes nor explodes. As Chialvo suggests, the emergence of scale-freeness istypical for such critical systems [5].The stop condition can be either fixed number of iterations or the moment,when the network reaches its steady state i.e. when most of charge is storedin small number of neurons and the dynamics freezes, such situation will bereferred as a ground state and the neurons with remaining charge as an elite.Both starting and ending phases are presented in Fig. 1.During the evolution the amount of activity flowing through an edge e =(u, v) (in both ways) is recorded, we will adopt the notation de for this value. Wedefine a spike-flow graph G′ = (V ′, E ′) as a multi-graph, consisting of all neuronsV ′ = V and all the synapses, which are present in E and the flow through themexceeded the threshold value E ′ = {e ∈ E : de ≥ θ}, with their multiplicitiesequal to the charge flowed (M(e) = de if de ≥ θ and M(e) = 0 otherwise), seeFig. 2.Unless stated otherwise in our simulations θ is assumed to be one. In suchcase every accepted transfer adds an edge into E ′.draftcharge 0-1 ( . 10 3)1-5 ( . 10 3)(a) Early phase of the simulation.charge 0-1 ( . 10 3)1-5 ( . 10 3)5-15 ( . 10 3)>15 ( . 10 3)(b) Ending phase.Fig. 1. A plot of the network of 5k = 5 · 103 neurons at early and late states of thesimulation. Left — starting, roughly uniform setup. Right — ending phase, all thecharge has stuck in the small number of elite units.Fig. 2. A plot of resulting spike flow sub-graph obtained at the end phase of thesimulation. The resulting graph was limited to around 700 vertices out of over 12000and was remapped onto a plane.draft3 ResultsThe simulations were carried on networks counting up to 35000 neurons. Figure3 presents obtained plot of i-th eigenvalue vs index i. The eigenvalues of the re-sulting spike-flow graph were sorted decreasingly and for log-plot issues negativeones were removed. An interesting feature is the middle part, where locally theplot behaves like a straight line for quite a long segment.050001000015000200000 5000 10000 15000 20000 25000ieigi(a) A plot of the graph spectrum.all eigenvaluesin linear segmentrange = [1000 .. 23000] = 68.53 %eigii(b) A log-log plot.Fig. 3. A plot of the spectrum of the spike-flow graph obtained throughout the dynam-ics of around 30k neurons. The plots present the i-th eigenvalue vs i, eigenvalues aresorted decreasingly. Middle part of the plot covers around 60% of the whole dataset.For reference — a line segment has a slope −2 (a function x−2 in log-log plot ).Note, that such behaviour in fully connected networks with winner-takes-allrule dynamics was theoretically predicted in [12] and confirmed in [9]. Despitethe facts, that in our model WTA dynamics is only approximated by an inversetemperature β and the structural network is no longer fully connected (actuallyit is quite sparse), the model produces strikingly similar feature.The plot failed to indicate clear scale-free dependence for top eigenvalues, see[9]. Instead this dependency arises in its mid-part and covers about 60% of thewhole data. We argue, that this feature might be rooted in lack of direct synapticconnections between the elite neurons, so it is not always possible to transferlarge amounts of charge directly between them. Somehow reiterated from [10]is an observable exponential truncation of this scale-free dependency, howeverwithout rigorous analysis one cannot state whether it is due to finite simulationsample.Additionally, there is no sign of first outlying eigenvalue, which is typical forErdős-Réyni random graph [2], recall that Erdős-Réyni random graph with nvertices and p ∈ (0, 1) is generated by including every edge of the full graph in-dependently with uniform probability p [8]. For comparison, a plot of eigenvaluesof the ER model is presented in Fig. 4.While Fig. 3 presents a spectrum obtained once the simulation was termi-nated, it turns out even more interesting to see it in various steps of the dynamics.draftiieigFig. 4. A log-log plot of the spectrum (the i-th eigenvalue vs i) of the Erdős-Réyni,random graph model with 10k neurons and its average connectivity is the same as inthe obtained spike-flow network for the same size. Note the first outlying eigenvalue.Fig. 5 presents a log-plot of the eigenvalues after computing 2, 10, 30, 50, 70 and100% of the iterations. It clearly indicates, that the spike-flow graph only beginsits self-organization process as a ER random graph. This seems quite natural,as in the dynamics we pick a synaptic edge randomly with uniform distributionand at start its probable that the connected neurons have not been drained fromtheir charge yet. Then, as the dynamics continues, a linear segment emerges andincreases in length.We have adopted following way to estimate the linear segment on the plot.A pair of indices (i1, i2) is referred as its borders if:– i1 < i2,– |ā− a1| < E1,– 1(i2−i1)∑i2i=i1(ei − a1 · i− a0)2 < E2,– i2 − i1 is maximal of all satisfying above points,where y = a1x + a0 stands for a formula of line approximating the data set{(log i1, log ei1)...(log i2, log ei2)}, the formula can be obtained using linear re-gression for instance; ej is a j-th largest eigenvalue; thresholds E1 and E2 werepicked arbitrarily depending on the number of neurons in the simulation anddesired accuracy and the ā is an expected slope. If no such pair exists, we con-clude that the spectrum does not have any linear segment (such situation didnot occurred, until the threshold values were ridiculously strict).Evolution of both indices i1, i2 defined as above and length of the segment(i2−i1) is presented in Fig. 6. Somehow unsettling, the growth depends mainly onreducing the beginning index i1, we also observe fluctuations of the upper boundi2. This seems be related to the vulnerability linear regression on non-uniformlydistributed data. Nevertheless, the i2−i1 clearly grows as the network undergoesits dynamics suggesting that i-th eigenvalue of the final spike-flow graph decaysas ci2 , even if the structural network was not fully connected.draft7016 neuronsrange = [1711 .. 3192] = 1481simulation step = 2%(a) After 2% of the iterations.7016 neuronsrange = [1061 .. 3512] = 2451simulation step = 10%(b) After 10 % of the iterations.7016 neuronsrange = [601 .. 3872] = 3271simulation step = 30%(c) After 30 % of the iterations.7016 neuronsrange = [421 .. 4052] = 3631simulation step = 50%(d) After 50 % of the iterations.7016 neuronsrange = [251 .. 3952] = 3701simulation step = 70%(e) After 70% of the iterations.7016 neuronsrange = [111 .. 3722] = 3611simulation step = 100%(f) After 100 % of the iterations.Fig. 5. Log-log plots of the spectrum of the spike-flow graph obtained in various stepsof the dynamics. Dotted line denotes whole spectrum, solid line — the estimated linearsegment. Simulation was carried on about 7 · 103 neurons and 2 · 109 iterations.draft0100020003000400050000 20 40 60 80 100lower bound i1upper bound i2segment length i2-i1indexsimulation progress [%]Fig. 6. Estimated length of the linear part of the spectrum throughout the simulation,measured every 2% of the iterations.4 Conclusion and Future WorkTo summarize, we have presented spectra of the spike-flow graphs of the ge-ometrical model of neural network throughout its evolution. As the networkapproaches its steady state a linear dependency emerges and grows in the spec-trum plot, which suggests a (exponentially truncated) power law dependencyin graph’s eigenvalues. At the early stages the spectrum resembles those of theErdős-Réyni random graph and only later it evolves towards more sophisticatedmodel.One of interesting directions, in which this work can be extended, is applyingthe same spectral analysis to medical data from fMRI scans, as it was discussedconcerning a graph degree distribution in [10] or graph diameter [11]. However,to our knowledge, no functional networks obtained from fMRI data have beenanalysed with spectral methods so far. As a second aim of ongoing research, wepoint out an analysis of the network resiliency and fault tolerance [2].AcknowledgementsThe author would also like thank to dr Filip Piękniewski for hints and remarksconcerning this work.This research was supported in part by PL-Grid Infrastructure. The authoris grateful to PL-Grid Project1 staff and help-line.The work has been partially supported by Ministry of Science and HigherEducation and grant DEC-2011/01/N/ST6/01931.1 http://www.plgrid.pldraftReferences1. Ackley D. H., Hinton G. E., Sejnowski T. J.: A Learning Algorithm for BoltzmannMachines, Cognitive Science, 9(1):147-169, 1985.2. Albert R., Barabasi A. L.: Statistical mechanics of complex networks, Reviews ofmodern physics, Vol 74, January 2002.3. Bullmore E., Sporns O.: Complex brain networks: graph theoretical analysis of struc-tural and functional systems, Nature Reviews, Neuroscience, vol 10, March 2009.4. Chung F., Lu L.: Complex graphs and networks, American Mathematical Society,2006.5. Chialvo D.: Critical brain networks, Physica A: Statistical Mechanics and its Appli-cations, Vol. 340, Issue 4, September 2004.6. Cvetković D., Rowlingson P., Simić S.: Eigenspaces of graphs, Cambridge UniversityPress, 19977. Eguiluz V., Chialvo D., Cecchi G., Baliki M., Apkarian V.: Scale-Free Brain Func-tional Networks, Physical Review Letters, PRL 94 018102, JAN 2005.8. Erdős P., Réyni A.: On random graphs I, Publ. Math. Debrecen 6, 290–297, 1959.9. Piekniewski F.: Spectra of the Spike Flow Graphs of Recurrent Neural Networks,Artificial Neural Networks - ICANN 2009, 19th International Conference, 2009,Proceedings, Part II.10. Piersa J., Piekniewski F., Schreiber T.: Theoretical model for mesoscopic-levelscale-free self-organization of functional brain networks, IEEE Transactions on Neu-ral Networks, vol. 21, no. 11, November 2010.11. Piersa J.: Diameter of the spike-flow graphs of geometrical neural networks, post-proceedings of 9th International Conference on Parallel Processing and AppliedMathematics, serie Lecture Notes on Computer Science, September 2011 (in print).12. Schreiber T.: Spectra of winner-take-all stochastic neural networks, arXiv [availableon-line], http://arxiv.org/PS cache/arxiv/pdf/0810/0810.3193v2.pdf, 3193(0810),October 2008, pp. 1–21.

Spectra of the Spike-Flow Graphs in Geometrically Embedded Neural Networks

https://repozytorium.umk.pl/bitstream/handle/item/1674/Piersa_2012_ICAISC__draft.pdf?sequence=1

Spectra of the Spike-Flow Graphs in Geometrically Embedded Neural Networks

Abstract

Similar works

Full text

Available Versions

Repository of Nicolaus Copernicus University