Enabling Massive Deep Neural Networks with the GraphBLAS

Deep Neural Networks (DNNs) have emerged as a core tool for machine learning. The computations performed during DNN training and inference are dominated by operations on the weight matrices describing the DNN. As DNNs incorporate more stages and more nodes per stage, these weight matrices may be required to be sparse because of memory limitations. The GraphBLAS.org math library standard was developed to provide high performance manipulation of sparse weight matrices and input/output vectors. For sufficiently sparse matrices, a sparse matrix library requires significantly less memory than the corresponding dense matrix implementation. This paper provides a brief description of the mathematics underlying the GraphBLAS. In addition, the equations of a typical DNN are rewritten in a form designed to use the GraphBLAS. An implementation of the DNN is given using a preliminary GraphBLAS C library. The performance of the GraphBLAS implementation is measured relative to a standard dense linear algebra library implementation. For various sizes of DNN weight matrices, it is shown that the GraphBLAS sparse implementation outperforms a BLAS dense implementation as the weight matrix becomes sparser.Comment: 10 pages, 7 figures, to appear in the 2017 IEEE High Performance Extreme Computing (HPEC) conferenc

A geometric entropy detecting the Erd\"os-R\'enyi phase transition

We propose a method to associate a differentiable Riemannian manifold to a generic many degrees of freedom discrete system which is not described by a Hamiltonian function. Then, in analogy with classical Statistical Mechanics, we introduce an entropy as the logarithm of the volume of the manifold. The geometric entropy so defined is able to detect a paradigmatic phase transition occurring in random graphs theory: the appearance of the `giant component' according to the Erd\"os-R\'enyi theorem.Comment: 11 pages, 3 figures. arXiv admin note: substantial text overlap with arXiv:1410.545

Riemannian-geometric entropy for measuring network complexity

A central issue of the science of complex systems is the quantitative characterization of complexity. In the present work we address this issue by resorting to information geometry. Actually we propose a constructive way to associate to a - in principle any - network a differentiable object (a Riemannian manifold) whose volume is used to define an entropy. The effectiveness of the latter to measure networks complexity is successfully proved through its capability of detecting a classical phase transition occurring in both random graphs and scale--free networks, as well as of characterizing small Exponential random graphs, Configuration Models and real networks.Comment: 15 pages, 3 figure

Spectra of "Real-World" Graphs: Beyond the Semi-Circle Law

Many natural and social systems develop complex networks, that are usually modelled as random graphs. The eigenvalue spectrum of these graphs provides information about their structural properties. While the semi-circle law is known to describe the spectral density of uncorrelated random graphs, much less is known about the eigenvalues of real-world graphs, describing such complex systems as the Internet, metabolic pathways, networks of power stations, scientific collaborations or movie actors, which are inherently correlated and usually very sparse. An important limitation in addressing the spectra of these systems is that the numerical determination of the spectra for systems with more than a few thousand nodes is prohibitively time and memory consuming. Making use of recent advances in algorithms for spectral characterization, here we develop new methods to determine the eigenvalues of networks comparable in size to real systems, obtaining several surprising results on the spectra of adjacency matrices corresponding to models of real-world graphs. We find that when the number of links grows as the number of nodes, the spectral density of uncorrelated random graphs does not converge to the semi-circle law. Furthermore, the spectral densities of real-world graphs have specific features depending on the details of the corresponding models. In particular, scale-free graphs develop a triangle-like spectral density with a power law tail, while small-world graphs have a complex spectral density function consisting of several sharp peaks. These and further results indicate that the spectra of correlated graphs represent a practical tool for graph classification and can provide useful insight into the relevant structural properties of real networks.Comment: 14 pages, 9 figures (corrected typos, added references) accepted for Phys. Rev.