Full article is available  at Springerlink:

http://link.springer.com/chapter/10.1007%2F978-3-642-31464-3_52



DOI: 10.1007/978-3-642-31464-3_52Average path length is recognised as one of the vital characteristics of random graphs and complex networks. Despite a rather sparse structure, some cases were reported to have a relatively short lengths between every pair of nodes, making the whole network available in just several hops. This small-worldliness was reported in metabolic, social or linguistic networks and recently in the Internet. In this paper we present results concerning path length distribution and the diameter of the spike-flow graph obtained from dynamics of geometrically embedded neural networks. Numerical results confirm both short diameter and average path length of resulting activity graph. In addition to numerical results, we also discuss means of running simulations in a concurrent environment

Piersa, Jarosław

English

Repository of Nicolaus Copernicus University

draftDiameter of the spike-flow graphs of geometricalneural networksJaroslaw Piersapiersaj[at]mat.umk.plFaculty of Mathematics and Computer Science, Nicolaus Copernicus University,PolandAbstract. Average path length is recognised as one of the vital char-acteristics of random graphs and complex networks. Despite a rathersparse structure, some cases were reported to have a relatively shortlengths between every pair of nodes, making the whole network availablein just several hops. This small-worldliness was reported in metabolic,social or linguistic networks and recently in the Internet. In this paperwe present results concerning path length distribution and the diameterof the spike-flow graph obtained from dynamics of geometrically embed-ded neural networks. Numerical results confirm both short diameter andaverage path length of resulting activity graph. In addition to numeri-cal results, we also discuss means of running simulations in a concurrentenvironment.Keywords: geometrical neural networks, path length distribution, small-worldliness, graph diameter1 IntroductionNeural networks, while already turned out a worthy tool in machine learning,still constitute a potent source of knowledge about the nature of biological brain.Recently, a growing number of mathematical models were developed and studiedin order to shed some light [9,10].One of the most striking facts about brain networks is its sparsity. Recall, thatnumber of neurons in human brain is put at 1011 while the number of synapses1015, clearly this places brain somewhere in-between regular lattice and a fullyconnected graph.In [11] a flexible and mathematically feasible model of neural activity wasput forward. It was mathematically proven and numerically confirmed, that itsdegree distribution obeys a power law, moreover the exponent value is close toresults obtained from fMRI scans of human brain [7]. Continuing the research,we have decided to look closer on other commonly discussed features of themodel. The aim of this work is to present strictly numerical findings (although,supported by random graph theory) about the average path length of spike-flowgraphs, which tend emerge as a result of an self-organization process accompa-nying the dynamics of the system. Despite its sparsity, the graph turns out todrafthave relatively short diameter, which again bears similarity to features reportedin real brain networks [3]. However complex the structure of the network mightbe, it still admits passing of information between any areas within just a fewedge traverses. This striking feature is believed to be one of the foundations ofbrain’s resilience to noise and random failures [2,3]. In addition to presentingobtained results, we also provide a brief discussion about numerical details ofthe simulation, focusing on paralleisation possibilities.The article is organised as follows: we present the network model and thesimulation dynamics in Section 2. Than we present obtained results of the pathlength distribution in Section 3. Numerical, technical and implementation detailsare discussed in Section 4 and the paper is concluded with Section 5.2 Simulation modelWe start with description of the underlying structural network G = (V, E),where V is a set of neurons and E stands for synaptic connections. Given aradius R > 1 of the two-dimensional sphere S2 ⊂ R3 and the expected densityof neurons ρ  1 we pick number of neurons N in the system randomly fromPoissonian distribution P(ρ ·4πR2), namely with parameter λ scaling as densitytimes surface of the sphere. Each of N neurons v is than independently pickedfrom the uniform distribution on the sphere surface. Additionally, it receives itsinitial charge σv ≥ 0. The charge will be a subject to a dynamics, though weadmit only non-negative integer values for σv.A set of synapses is constructed in as follows: for each pair of neurons u, v asymmetric synaptic connection {u, v} is added to E independently with proba-bilityP({u, v} ∈ E) ={d(u, v)α d(u, v) ≥ 11 otherwise, (1)where d() stands for euclidean distance between neurons u and v. The exponentα is fixed at the value −2.5. If successfully included, the synapse e = {u, v}receives its weight wuv independently generated from the gaussian distributionwuv ∼ N(0, s2). The weight indicates an excitatory or inhibitory (when negative)type of the synapse.A network energy function depending on the charge stored in each of neuronsis defined asE(σ̄) =∑(u,v)∈Ewu,v|σv − σu|. (2)Having presented the structural network, we describe the dynamics. We adopta variation of the celebrated Ising spin glass model, though extended for multiplevalues of σ.– At each step we pick a random pair of neurons u, v ∈ V, such that they areconnected with a synapse {u, v} ∈ E and the charge stored in u is non-zeroσu ≥ 1,draft– We try to transfer a single unit of charge from u to v through the synapse,in other words σu := σu − 1 and σv := σv + 1,– if this transfer reduces energy of the system, it is accepted, and the dynamicsproceeds to next iteration,– if this transfer increases the energy by ∆E, then it is accepted with proba-bility P(u→ v) = exp(−β∆E) and rejected otherwise.The parameter β  1 stands for an inverse temperature and is assumed to belarge. Though the dynamics can run arbitrarily long, it is terminated after hittinggiven number of iterations or reaching the state, where no further transfers areaccepted.Fig. 1. A subgraph of resulting network, plot includes 719 neurons and 2500 synapses.Spatial coordinates were remapped from a sphere to a surface for better visibility.Despite a small sample, hubs (node with large degree) are clearly noticeable.3 Path length distribution resultsAs pointed out in [11] the the dynamics leads to concentration of the charge insmall number of nodes. For each synapse e = {u, v} a total amount of chargewhich flew either from u to v or from v to u is recorded and denoted as de.Reaching a predefined threshold value θ qualifies an edge as a vital in the evo-lution process. A graph built from the vital edges will be referred as a spikeflow graph, namely G1 = (V, E1 = {e ∈ E : de ≥ θ}). A value θ = 1 is assumedthroughout rest of the work, which results in spike flow graph consisting of alledges participating in the dynamics. A small subgraph of the resulting spike flownetwork is depicted on Fig. 1.draftAs a minimal path length between neurons u and v we understand classicaldefinition, i.e. l(u, v) is a minimum number of edges (e1 = {u1, v1}, .., en ={un, vn}), such that– they start at u and terminate at v, u1 = u, vn = v,– they are incident, ∀i=1..n−1vi = ui+1,– they are included in resulting spike flow graph ∀iei ∈ E1.For the sake of simplicity we assume l(u, u) = 0.00.20.40.60.810 1 2 3 4 5 6freqlengthPath Length Distribution 5k12k19k28k(a) α = −2.500.20.40.60.810 1 2 3 4 5 6freqlengthPath Length Distribution 7k12k19k28kalpha = -3.5(b) α = −3.5Fig. 2. A plot of path length distribution of the spike-flow graph obtained from asimulation of networks counting 5 to 28k neurons. The distribution slowly increases itsmode as the number of nodes grows.Obtained results of the path lengths are presented on Fig. 2. The resultswere collected from simulations of networks counting between 300 and up to5.5 · 104 neurons for exponent of the connectivity function (Eq. 1) α = −2.5and α = −3.5. As shown in [12] a resulting spike-flow graph obeys a power lawdegree distribution, this type of graphs also frequently exhibits a small worldphenomenon [2].Fig. 3 presents the dependency between the size of the simulation sampleand corresponding average path value. It presents a slow growth, but in allof our cases is still bounded by 4. A slow increase of the average path lengthis observable, which again is in agreement with theoretical results of classicalrandom graphs theory. Also note, that for our small samples the path lengthdistribution is focused on 3 values. The the rest (number of paths of length upto the graph diameter — see below) is non zero, though negligible.The diameter of the graph is defined as a maximum value of shortest pathlength l(u, v) between every pair of nodes (u, v) in the graph. Its plot is providedon Fig 3. Recall that for Erdős-Réyni random graph model the diameter growslogarithmically as the network size increases, see [5] Ch. 5 and 7 for rigorousproof. Clearly the diameter can be arbitrarily larger than an average path length,draft10-510-410-310-210-1100-1 0 1 2 3 4 5 6 7 8freqpath lengthPath Length Distributionneurons = 43166Fig. 3. A semi-log plot of the path length distribution of the spike-flow graph. Thenumber of paths of length 0 (selfloops) and 1, though negligible, is non zero. Numberof neurons N ' 40k, α = −3.5.however in our cases it is bounded by 6 for our samples, which means thatstarting in any neuron, the whole network is available in no more than 6 hops.While our model is not a ER model, this feature make it strikingly similar toWWW [1] or social graphs [6]. A few reports suggest small-worldliness to bepresent also in brain activity networks [3].Small discrepancy between l for different values of α seems to originate fromdifference in underlying structural graphs. Lower α yields less synapses in thestructural network G, while the spike-flow graph consists exclusively of synapsespresent in G. The formula of g was suggested in [7].While the clustering coefficient is an aim of ongoing research we can providea brief preliminary results concerning this value. After classical theory [5] wedefine a clustering coefficient of the node v as a ratio of existing edges in theneighbourhood of v to all possible edges.C(u) =|{e = (w, v) : e ∈ E ∧ (w, u) ∈ E ∧ (v, u) ∈ E}||{(w, v) : (w, u) ∈ E ∧ (v, u) ∈ E}|(3)The clustering coefficient of the graph C is defined as an average of clusteringcoefficients of its nodes.C =1|V|∑u∈VC(u) (4)Recall, that for ER graph model the clustering coefficient is relatively small (itis equal to the average connectivity of the graph). As it can be see in Table 1for spike-flow graph in our model the actual clustering coefficient is 2-3 ordersof magnitude higher then its average degree, which again is in agreement withfMRI data [3].draft0.511.522.533.544.550 10000 20000 30000 40000 50000 60000aplneuronsalpha = -2.5alpha = -3.5Fig. 4. A plot of the average path length of the spike-flow graph vs sample size.012345670 10000 20000 30000 40000 50000 60000diameterneuronsalpha = -2.5alpha = -3.5Fig. 5. A plot of the diameter (maximum path length connecting a pair of nodes) ofthe spike-flow graph vs sample size. Note, that slope segments indicate values, whichare not known (diameter must be an integer number).draftTable 1. Calculated clustering coefficient and average connectivity for obtained spike-flow graphs, α = −2.5. Note that the average degree is also a theoretical clusteringcoefficient for equivalent ER random graph, while by clustering coefficient we mean anactual, numerically obtained value for our model.Neurons Synapses Average degree Clustering coefficient3155 0.18M 0.038 0.24312568 0.8M 0.010 0.20719531 1.3M 0.006 0.19824630 1.7M 0.005 0.19428501 2.0M 0.004 0.19138322 2.6M 0.0035 0.18750241 3.5M 0.0027 0.18358244 4.1M 0.0023 0.1824 Numerical detailsThe simulations were carried on spheres with radii varying up to R = 30 withconstant density ρ = 10, see Sec. 2. This yields samples counting up to 55k neu-rons.Additionally, we expect an average 500 units of charge amount per neuronto be present in the network. In most cases a strict limit of iterations turned outa sufficient terminating condition.The simulation itself was paralleled to take advantage of multiprocessingenvironment. However, due to highly unpredictable dynamics and the usageof control-flow instructions one finds particularly demanding to put graphicalcomputing units into a good use [12]. We decided to implement a task division,i.e. splitting the total number of iterations among parallel threads. Sparse thoughthe resulting network might be, it still requires visiting all of the neighbourneurons before deciding whether (or not) to accept the transfer. Along withgrowing number of threads it increases frequency of waiting for the lock to bereleased and thus affects the speedup. The overall efficiency was about 66% forthree threads.For calculation of the path length distribution a classical Dijkstra algorithmhas been adapted with a domain-division paralleisation. The results of calcu-lation times are presented on Fig. 4. The computation time seems reasonable,when take into account sequential collecting data from threads. Obtained effi-ciency is about .78 for 4 threads, which is the number of computing cores. Theresults were obtained on Core Quad 2GHz CPU + 4GB RAM + Fedora 11–14(32bit, PAE) system.Some of the simulations were also run of the infrastructure of the PG-Grid,who kindly provided their computing resources. The timing and speedup resultsfrom those simulations, while still being collected and analysed, are beyond thescope of this work.Additionally a Monte Carlo was also implemented, however we found thetime, required to yield a results with a satisfactory precision, comparative to thedraft050010001500200025003000 0 1 2 3 4 5 6time (mins) threads12k Neurons18k Neurons(a) Computation time0200004000060000800001000000 1 2 3 4 5 6 7iterations per secondthreads12k Neurons28k Neurons(b) Iterations per secondFig. 6. Computation times obtained for the dynamics. Reprinted from [12].time required by the former algorithm. Although, for larger samples MC mayturn out indispensable.01002003004005006000 1 2 3 4 5 6 7 8time (mins)threads38k Neurons50k Neurons(a) Computation time.01e+062e+063e+064e+065e+066e+067e+060 1 2 3 4 5 6 7paths per secondthreads38k Neurons50k Neurons(b) Number of paths calculated per sec-ond.Fig. 7. Computation times obtained and number of path lengths calculated per secondfor sample 38k neurons. The results were obtained on 4 cores CPU.5 Conclusion and future workIn this work we have presented our findings concerning path length distributionin spike-flow graphs of neural network. The distribution turns out concentratedon few values and its mean value is relatively low, making the whole networkavailable to reach within just a few hops. Along with high clicqueliness deter-mined by the clustering coefficient, this suggests a complicated structure of thedraftresulting network, even despite not so complicated dynamics. Clearly, the net-work bears resemblances to social graphs or WWW networks, which also tendto exhibit a small-world phenomenon.It might turn out interesting to see how the network evolves to its finalshape throughout the simulation. This unfortunately requires either more com-putational power or reduction of the size of the network, in order to keep thesimulation time reasonable. As an additional aim of future work, we point atconsidering a directed version of the spike flow graph, which is natural feature ofWWW networks [1]. Since the dynamics in our network is directed, the distinc-tion between a input and output synapses seems to be a forgone conclusion. Onemore vital aspect frequently discussed when working with complex networks isclustering coefficient [2,3]. While it was bearly mentioned in in Sec. 3, a coherentcomparison between the results obtained from fMRI and our model is a focus ofongoing research.AcknowledgementsThe author would like to mention the unparalleled support from prof. TomaszSchreiber, Associate Professor at Faculty of Mathematics and Computer Science,NCU, a bright young scientist, who carried researches in mathematics, computerscience and physics, prematurely died in December 2010.The author acknowledges a cooperation and a dash of red ink from dr FilipPiękniewski.Additionally, the author is grateful to PL-Grid Project1 for providing a com-puting infrastructure for simulations.The work has been partially supported by Polish Ministry of Science andgrant project 2011/01/N/ST6/01931.The author would also like to acknowledge the support of the EuropeanSocial Fund and Government of Poland as a part of Integrated OperationalProgram for Regional Development, ZPORR, Activity 2.6 ”Regional InnovationStrategies and Knowledge Transfer” of Kuyavian-Pomeranian Voivodeship inPoland (2008-2009).References1. R. Albert, H. Jeong, A. L. Barabasi Diameter of the World-Wide Web, Nature, Vol401, 9 September 1999,2. R. Albert, A. L. Barabasi, Statistical mechanics of complex networks, Reviews ofmodern physics, Vol 74, January 20023. D. S. Bassett E. Bullmore Small-World Brain Networks, The Neuroscientist, Volume12, Number 6, 20064. E. Bullmore, O. Sporns, Complex brain networks: graph theoretical analysis of struc-tural and functional systems, Nature Reviews, Neuroscience, vol 10, March 20091 http://www.plgrid.pldraft5. F. Chung, L. Lu, Complex graphs and networks, American Mathematical Society,20066. P. Csermely, Weak links: the universal key to the stability of networks and complexsystems, Springer Verlag (Heidelberg, Germany) 20097. V. Eguiluz, D. Chialvo, G. Cecchi, M. Baliki, V. Apkarian Scale-free brain functionalnetworks, Physical Review Letters, PRL 94 018102, JAN 2005,8. F. Piekniewski, Spontaneous scale-free structures in spike flow graphs for recurrentneural networks, Ph.D. dissertation, Warsaw University, Warsaw, Poland, 2008,9. F. Piekniewski and T.Schreiber, Spontaneous scale-free structure of spike flow graphsin recurrent neural networks, Neural Networks vol. 21, no. 10, pp. 1530 - 1536, 200810. F. Piekniewski, Spectra of the spike flow graphs of recurrent neural networks, Ar-tificial Neural Networks - ICANN 2009, 19th International Conference, 2009, Pro-ceedings, Part II11. J. Piersa, F. Piekniewski, T. Schreiber, Theoretical model for mesoscopic-levelscale-free self-organization of functional brain networks, IEEE Transactions on Neu-ral Networks, vol. 21, no. 11, November 201012. J. Piersa, T. Schreiber Scale-free degree distribution in information-flow graphs ofgeometrical neural networks. Simulations in concurren environmnet (in Polish) ac-cepted for Mathematical Methods in Modeling and Analysis of Concurrent Systems2010-07 – Postproceedings,13. T. Schreiber Spectra of winner-take-all stochastic neural networks, arXiv[available on-line], http://arxiv.org/PS_cache/arxiv/pdf/0810/0810.3193v2.pdf, 3193(0810), October 2008, pp. 1–21.14. D. Watts, S. Strogatz Collective dynamics of ’small-world’ networks, Nature, vol393, 4 June 1998, pp. 440-442.

Diameter of the spike-flow graphs of geometrical neural networks

https://repozytorium.umk.pl/bitstream/handle/item/1673/Piersa_PPAM_draft.pdf?sequence=1

Diameter of the spike-flow graphs of geometrical neural networks

Abstract

Similar works

Full text

Available Versions

Repository of Nicolaus Copernicus University