Deep Expander Networks: Efficient Deep Networks from Graph Theory

Efficient CNN designs like ResNets and DenseNet were proposed to improve accuracy vs efficiency trade-offs. They essentially increased the connectivity, allowing efficient information flow across layers. Inspired by these techniques, we propose to model connections between filters of a CNN using graphs which are simultaneously sparse and well connected. Sparsity results in efficiency while well connectedness can preserve the expressive power of the CNNs. We use a well-studied class of graphs from theoretical computer science that satisfies these properties known as Expander graphs. Expander graphs are used to model connections between filters in CNNs to design networks called X-Nets. We present two guarantees on the connectivity of X-Nets: Each node influences every node in a layer in logarithmic steps, and the number of paths between two sets of nodes is proportional to the product of their sizes. We also propose efficient training and inference algorithms, making it possible to train deeper and wider X-Nets effectively. Expander based models give a 4% improvement in accuracy on MobileNet over grouped convolutions, a popular technique, which has the same sparsity but worse connectivity. X-Nets give better performance trade-offs than the original ResNet and DenseNet-BC architectures. We achieve model sizes comparable to state-of-the-art pruning techniques using our simple architecture design, without any pruning. We hope that this work motivates other approaches to utilize results from graph theory to develop efficient network architectures.Comment: ECCV'1

Random Unitaries Give Quantum Expanders

We show that randomly choosing the matrices in a completely positive map from the unitary group gives a quantum expander. We consider Hermitian and non-Hermitian cases, and we provide asymptotically tight bounds in the Hermitian case on the typical value of the second largest eigenvalue. The key idea is the use of Schwinger-Dyson equations from lattice gauge theory to efficiently compute averages over the unitary group.Comment: 14 pages, 1 figur

Overlap properties of geometric expanders

The {\em overlap number} of a finite $(d+1)$-uniform hypergraph $H$ is defined as the largest constant $c(H)\in (0,1]$ such that no matter how we map the vertices of $H$ into $\R^d$, there is a point covered by at least a $c(H)$-fraction of the simplices induced by the images of its hyperedges. In~\cite{Gro2}, motivated by the search for an analogue of the notion of graph expansion for higher dimensional simplicial complexes, it was asked whether or not there exists a sequence $\{H_n\}_{n=1}^\infty$ of arbitrarily large $(d+1)$-uniform hypergraphs with bounded degree, for which $\inf_{n\ge 1} c(H_n)>0$. Using both random methods and explicit constructions, we answer this question positively by constructing infinite families of $(d+1)$-uniform hypergraphs with bounded degree such that their overlap numbers are bounded from below by a positive constant $c=c(d)$. We also show that, for every $d$, the best value of the constant $c=c(d)$ that can be achieved by such a construction is asymptotically equal to the limit of the overlap numbers of the complete $(d+1)$-uniform hypergraphs with $n$ vertices, as $n\rightarrow\infty$. For the proof of the latter statement, we establish the following geometric partitioning result of independent interest. For any $d$ and any $\epsilon>0$, there exists $K=K(\epsilon,d)\ge d+1$ satisfying the following condition. For any $k\ge K$, for any point $q \in \mathbb{R}^d$ and for any finite Borel measure $\mu$ on $\mathbb{R}^d$ with respect to which every hyperplane has measure $0$, there is a partition $\mathbb{R}^d=A_1 \cup \ldots \cup A_{k}$ into $k$ measurable parts of equal measure such that all but at most an $\epsilon$-fraction of the $(d+1)$-tuples $A_{i_1},\ldots,A_{i_{d+1}}$ have the property that either all simplices with one vertex in each $A_{i_j}$ contain $q$ or none of these simplices contain $q$

MV3: A new word based stream cipher using rapid mixing and revolving buffers

MV3 is a new word based stream cipher for encrypting long streams of data. A direct adaptation of a byte based cipher such as RC4 into a 32- or 64-bit word version will obviously need vast amounts of memory. This scaling issue necessitates a look for new components and principles, as well as mathematical analysis to justify their use. Our approach, like RC4's, is based on rapidly mixing random walks on directed graphs (that is, walks which reach a random state quickly, from any starting point). We begin with some well understood walks, and then introduce nonlinearity in their steps in order to improve security and show long term statistical correlations are negligible. To minimize the short term correlations, as well as to deter attacks using equations involving successive outputs, we provide a method for sequencing the outputs derived from the walk using three revolving buffers. The cipher is fast -- it runs at a speed of less than 5 cycles per byte on a Pentium IV processor. A word based cipher needs to output more bits per step, which exposes more correlations for attacks. Moreover we seek simplicity of construction and transparent analysis. To meet these requirements, we use a larger state and claim security corresponding to only a fraction of it. Our design is for an adequately secure word-based cipher; our very preliminary estimate puts the security close to exhaustive search for keys of size < 256 bits.Comment: 27 pages, shortened version will appear in "Topics in Cryptology - CT-RSA 2007

The spectrum of the random environment and localization of noise

We consider random walk on a mildly random environment on finite transitive d- regular graphs of increasing girth. After scaling and centering, the analytic spectrum of the transition matrix converges in distribution to a Gaussian noise. An interesting phenomenon occurs at d = 2: as the limit graph changes from a regular tree to the integers, the noise becomes localized.Comment: 18 pages, 1 figur

Comparing the reliability of networks by spectral analysis

We provide a method for the ranking of the reliability of two networks with the same connectance. Our method is based on the Cheeger constant linking the topological property of a network with its spectrum. We first analyze a set of twisted rings with the same connectance and degree distribution, and obtain the ranking of their reliability using their eigenvalue gaps. The results are generalized to general networks using the method of rewiring. The success of our ranking method is verified numerically for the IEEE57, the Erd\H{o}s-R\'enyi, and the Small-World networks.Comment: 7 pages, 3 figure

Linear-time list recovery of high-rate expander codes

We show that expander codes, when properly instantiated, are high-rate list recoverable codes with linear-time list recovery algorithms. List recoverable codes have been useful recently in constructing efficiently list-decodable codes, as well as explicit constructions of matrices for compressive sensing and group testing. Previous list recoverable codes with linear-time decoding algorithms have all had rate at most 1/2; in contrast, our codes can have rate $1 - \epsilon$ for any $\epsilon > 0$. We can plug our high-rate codes into a construction of Meir (2014) to obtain linear-time list recoverable codes of arbitrary rates, which approach the optimal trade-off between the number of non-trivial lists provided and the rate of the code. While list-recovery is interesting on its own, our primary motivation is applications to list-decoding. A slight strengthening of our result would implies linear-time and optimally list-decodable codes for all rates, and our work is a step in the direction of solving this important problem

Majority Dynamics and Aggregation of Information in Social Networks

Consider n individuals who, by popular vote, choose among q >= 2 alternatives, one of which is "better" than the others. Assume that each individual votes independently at random, and that the probability of voting for the better alternative is larger than the probability of voting for any other. It follows from the law of large numbers that a plurality vote among the n individuals would result in the correct outcome, with probability approaching one exponentially quickly as n tends to infinity. Our interest in this paper is in a variant of the process above where, after forming their initial opinions, the voters update their decisions based on some interaction with their neighbors in a social network. Our main example is "majority dynamics", in which each voter adopts the most popular opinion among its friends. The interaction repeats for some number of rounds and is then followed by a population-wide plurality vote. The question we tackle is that of "efficient aggregation of information": in which cases is the better alternative chosen with probability approaching one as n tends to infinity? Conversely, for which sequences of growing graphs does aggregation fail, so that the wrong alternative gets chosen with probability bounded away from zero? We construct a family of examples in which interaction prevents efficient aggregation of information, and give a condition on the social network which ensures that aggregation occurs. For the case of majority dynamics we also investigate the question of unanimity in the limit. In particular, if the voters' social network is an expander graph, we show that if the initial population is sufficiently biased towards a particular alternative then that alternative will eventually become the unanimous preference of the entire population.Comment: 22 page

Expansion in SL_d(Z/qZ), q arbitrary

Let S be a fixed finite symmetric subset of SL_d(Z), and assume that it generates a Zariski-dense subgroup G. We show that the Cayley graphs of pi_q(G) with respect to the generating set pi_q(S) form a family of expanders, where pi_q is the projection map Z->Z/qZ