Mean Field Limit of the Learning Dynamics of Multilayer Neural Networks

Can multilayer neural networks -- typically constructed as highly complex structures with many nonlinearly activated neurons across layers -- behave in a non-trivial way that yet simplifies away a major part of their complexities? In this work, we uncover a phenomenon in which the behavior of these complex networks -- under suitable scalings and stochastic gradient descent dynamics -- becomes independent of the number of neurons as this number grows sufficiently large. We develop a formalism in which this many-neurons limiting behavior is captured by a set of equations, thereby exposing a previously unknown operating regime of these networks. While the current pursuit is mathematically non-rigorous, it is complemented with several experiments that validate the existence of this behavior

Time-lagged Ordered Lasso for network inference

Accurate gene regulatory networks can be used to explain the emergence of different phenotypes, disease mechanisms, and other biological functions. Many methods have been proposed to infer networks from gene expression data but have been hampered by problems such as low sample size, inaccurate constraints, and incomplete characterizations of regulatory dynamics. Since expression regulation is dynamic, time-course data can be used to infer causality, but these datasets tend to be short or sparsely sampled. In addition, temporal methods typically assume that the expression of a gene at a time point depends on the expression of other genes at only the immediately preceding time point, while other methods include additional time points without any constraints to account for their temporal distance. These limitations can contribute to inaccurate networks with many missing and anomalous links. We adapted the time-lagged Ordered Lasso, a regularized regression method with temporal monotonicity constraints, for \textit{de novo} reconstruction. We also developed a semi-supervised method that embeds prior network information into the Ordered Lasso to discover novel regulatory dependencies in existing pathways. We evaluated these approaches on simulated data for a repressilator, time-course data from past DREAM challenges, and a HeLa cell cycle dataset to show that they can produce accurate networks subject to the dynamics and assumptions of the time-lagged Ordered Lasso regression

Semi-supervised network inference using simulated gene expression dynamics

Motivation: Inferring the structure of gene regulatory networks from high--throughput datasets remains an important and unsolved problem. Current methods are hampered by problems such as noise, low sample size, and incomplete characterizations of regulatory dynamics, leading to networks with missing and anomalous links. Integration of prior network information (e.g., from pathway databases) has the potential to improve reconstructions. Results: We developed a semi--supervised network reconstruction algorithm that enables the synthesis of information from partially known networks with time course gene expression data. We adapted PLS-VIP for time course data and used reference networks to simulate expression data from which null distributions of VIP scores are generated and used to estimate edge probabilities for input expression data. By using simulated dynamics to generate reference distributions, this approach incorporates previously known regulatory relationships and links the network to the dynamics to form a semi-supervised approach that discovers novel and anomalous connections. We applied this approach to data from a sleep deprivation study with KEGG pathways treated as prior networks, as well as to synthetic data from several DREAM challenges, and find that it is able to recover many of the true edges and identify errors in these networks, suggesting its ability to derive posterior networks that accurately reflect gene expression dynamics

Interior gradient estimates for quasilinear elliptic equations

We study quasilinear elliptic equations of the form $\text{div} \mathbf{A}(x,u,\nabla u) = \text{div}\mathbf{F}$ in bounded domains in $\mathbb{R}^n$, $n\geq 1$. The vector field $\mathbf{A}$ is allowed to be discontinuous in $x$, Lipschitz continuous in $u$ and its growth in the gradient variable is like the $p$-Laplace operator with $1<p<\infty$. We establish interior $W^{1,q}$-estimates for locally bounded weak solutions to the equations for every $q>p$, and we show that similar results also hold true in the setting of {\it Orlicz} spaces. Our regularity estimates extend results which are only known for the case $\mathbf{A}$ is independent of $u$ and they complement the well-known interior $C^{1,\alpha}$- estimates obtained by DiBenedetto \cite{D} and Tolksdorf \cite{T} for general quasilinear elliptic equations

Smooth (non)rigidity of cusp-decomposable manifolds

We define cusp-decomposable manifolds and prove smooth rigidity within this class of manifolds. These manifolds generally do not admit a nonpositively curved metric but can be decomposed into pieces that are diffeomorphic to finite volume, locally symmetric, negatively curved manifolds with cusps. We prove that the group of outer automorphisms of the fundamental group of such a manifold is an extension of an abelian group by a finite group. Elements of the abelian group are induced by diffeomorphisms that are analogous to Dehn twists in surface topology. We also prove that the outer automophism group can be realized by a group of diffeomorphisms of the manifold.Comment: 13 pages, 1 figure. Accepted for publication in Comment. Math. Helv. arXiv admin note: substantial overlap with arXiv:1105.520

Nil happens. What about Sol?

We construct complete, finite volume, 4-dimensional manifolds with sectional curvature $-1<K<0$ with cusp cross sections compact solvmanifolds.Comment: 2 page

Actions of higher rank, irreducible lattices on \CAT(0) cubical complexes

Let $\Gamma$ be an irreducible lattice of \Q-rank $\geq 2$ in a semisimple Lie group of noncompact type. We prove that any action of $\Gamma$ on a \CAT(0) cubical complex has a global fixed point.Comment: 5 pages, 1 figure. arXiv admin note: text overlap with arXiv:1108.412

Minimal orbifolds and (a)symmetry of piecewise locally symmetric manifolds

We show that if $g$ is a Riemannian metric on a closed piecewise locally symmetric manifold $M$, then the lift of $g$ to the universal cover $\widetilde{M}$ has a discrete isometry group. We also show that the index [\Isom(\widetilde{M}): \pi_1(M)] is bounded by a constant independent of $g$.Comment: 8 pages, 2 figure

On finite volume, negatively curved manifolds

We study noncompact, complete, finite volume, negatively curved manifolds $M$. We construct $M$ with infinitely generated fundamental groups in all dimensions $n \geq 2$. We construct $M$ whose cusp cross sections are compact hyperbolic manifolds in all dimension $n\geq 3$. In contrast we show that if sectional curvature $-1<K(M)<0$, then cusp cross sections have zero simplicial volume. We construct negatively curved lattices that do not contain any parabolic isometries. We show that there are $M$ such that $\widetilde{M}$ does not satisfy the visibility axiom. We give a condition on the curvature growth versus the volume decay that guarantees topological finiteness. We raise a few questions on finite volume, negatively curved manifolds.Comment: 18 page

Gluing locally symmetric manifolds: asphericity and rigidity

We use the reflection group trick to glue manifolds with corners that are Borel-Serre compactifications of locally symmetric spaces of noncompact type and obtain aspherical manifolds. We call these \emph{piecewise locally symmetric} manifolds. This class of spaces provide new examples of aspherical manifolds whose fundamental groups have the structure of a complex of groups. These manifolds typically do not admit a locally \CAT(0) metric. We prove that any self homotopy equivalence of such manifolds is homotopic to a homeomorphism. We compute the group of self homotopy equivalences of such a manifold and show that it can contain a normal free abelian subgroup, and thus can be infinite.Comment: 26 pages, 2 figure