A representer theorem for deep kernel learning

In this paper we provide a finite-sample and an infinite-sample representer theorem for the concatenation of (linear combinations of) kernel functions of reproducing kernel Hilbert spaces. These results serve as mathematical foundation for the analysis of machine learning algorithms based on compositions of functions. As a direct consequence in the finite-sample case, the corresponding infinite-dimensional minimization problems can be recast into (nonlinear) finite-dimensional minimization problems, which can be tackled with nonlinear optimization algorithms. Moreover, we show how concatenated machine learning problems can be reformulated as neural networks and how our representer theorem applies to a broad class of state-of-the-art deep learning methods

Vertex routing models

A class of models describing the flow of information within networks via routing processes is proposed and investigated, concentrating on the effects of memory traces on the global properties. The long-term flow of information is governed by cyclic attractors, allowing to define a measure for the information centrality of a vertex given by the number of attractors passing through this vertex. We find the number of vertices having a non-zero information centrality to be extensive/sub-extensive for models with/without a memory trace in the thermodynamic limit. We evaluate the distribution of the number of cycles, of the cycle length and of the maximal basins of attraction, finding a complete scaling collapse in the thermodynamic limit for the latter. Possible implications of our results on the information flow in social networks are discussed.Comment: 12 pages, 6 figure

Differentiable Programming Tensor Networks

Differentiable programming is a fresh programming paradigm which composes parameterized algorithmic components and trains them using automatic differentiation (AD). The concept emerges from deep learning but is not only limited to training neural networks. We present theory and practice of programming tensor network algorithms in a fully differentiable way. By formulating the tensor network algorithm as a computation graph, one can compute higher order derivatives of the program accurately and efficiently using AD. We present essential techniques to differentiate through the tensor networks contractions, including stable AD for tensor decomposition and efficient backpropagation through fixed point iterations. As a demonstration, we compute the specific heat of the Ising model directly by taking the second order derivative of the free energy obtained in the tensor renormalization group calculation. Next, we perform gradient based variational optimization of infinite projected entangled pair states for quantum antiferromagnetic Heisenberg model and obtain start-of-the-art variational energy and magnetization with moderate efforts. Differentiable programming removes laborious human efforts in deriving and implementing analytical gradients for tensor network programs, which opens the door to more innovations in tensor network algorithms and applications.Comment: Typos corrected, discussion and refs added; revised version accepted for publication in PRX. Source code available at https://github.com/wangleiphy/tensorgra

Chapter 14 – Evolutionary Algorithms Applied to Electronic-Structure Informatics: Accelerated Materials Design Using Data Discovery vs. Data Searching

We exemplify and propose extending the use of genetic programs (GPs) – a genetic algorithm (GA) that evolves computer programs via mechanisms similar to genetics and natural selection – to symbolically regress key functional relationships between materials data, especially from electronic structure. GPs can extract structure–property relations or enable simulations across multiple scales of time and/or length. Uniquely, GP-based regression permits “data discovery” – finding relevant data and/or extracting correlations (data reduction/data mining) – in contrast to searching for what you know, or you think you know (intuition). First, catalysis-related materials correlations are discussed, where simple electronic-structure-based rules are revealed using well-developed intuition, and then, after introducing the concepts, GP regression is used to obtain (i) a constitutive relation between flow stress and strain rate in aluminum, and (ii) multi-time-scale kinetics for surface alloys. We close with some outlook for a range of applications (materials discovery, excited-state chemistry, and multiscaling) that could rely primarily on density functional theory results

Training Two-Layer ReLU Networks with Gradient Descent is Inconsistent

We prove that two-layer (Leaky)ReLU networks initialized by e.g. the widely used method proposed by He et al. (2015) and trained using gradient descent on a least-squares loss are not universally consistent. Specifically, we describe a large class of one-dimensional data-generating distributions for which, with high probability, gradient descent only finds a bad local minimum of the optimization landscape. It turns out that in these cases, the found network essentially performs linear regression even if the target function is non-linear. We further provide numerical evidence that this happens in practical situations, for some multi-dimensional distributions and that stochastic gradient descent exhibits similar behavior.Comment: Changes in v2: Single-column layout, NTK discussion, new experiment, updated introduction, improved explanations. 20 pages + 33 pages appendix. Code available at https://github.com/dholzmueller/nn_inconsistenc

Toward Improving Understanding of the Structure and Biophysics of Glycosaminoglycans

Glycosaminoglycans (GAGs) are the linear carbohydrate components of proteoglycans (PGs) that mediate PG bioactivities, including signal transduction, tissue morphogenesis, and matrix assembly. To understand GAG function, it is important to understand GAG structure and biophysics at atomic resolution. This is a challenge for existing experimental and computational methods because GAGs are heterogeneous, conformationally complex, and polydisperse, containing up to 200 monosaccharides. Molecular dynamics (MD) simulations come close to overcoming this challenge but are only feasible for short GAG polymers. To address this problem, we developed an algorithm that applies conformations from unbiased all-atom explicit-solvent MD simulations of short GAG polymers to rapidly construct 3-D atomic-resolution models of GAGs of arbitrary length. MD simulations of GAG 10-mers (i.e., polymers containing 10 monosaccharides) and 20-mers were run and conformations of all monosaccharide rings and glycosidic linkages were analyzed and compared to existing experimental data. These analyses demonstrated that (1) MD-generated GAG conformations are in agreement with existing experimental data; (2) MD-generated GAG 10-mer ring and linkage conformations match those in corresponding GAG 20-mers, suggesting that these conformations are representative of those in longer GAG biopolymers; and (3) rings and linkages in GAG 10- and 20-mers behave randomly and independently in MD simulation. Together, these findings indicate that MD-generated GAG 20-mer ring and linkage conformations can be used to construct thermodynamically-correct models of GAG polymers. Indeed, our findings demonstrate that our algorithm constructs GAG 10- and 20-mer conformational ensembles that accurately represent the backbone flexibility seen in MD simulations. Furthermore, within a day, our algorithm constructs conformational ensembles of GAG 200-mers that we would reasonably expect from MD simulation, demonstrating the efficiency of the algorithm and reduction in its time and computational cost compared to simulation. While there are other programs that can quickly construct atomic-resolution models of GAGs, those programs use conformations from short GAG subunits in solid state. Our findings suggest that GAG 20-mers are more flexible than short GAG subunits, meaning our program constructs ensembles that more accurately represent GAG polymer backbone flexibility and provide valuable insights toward improving the understanding of the structure and biophysics of GAGs