Comparing Information-Theoretic Measures of Complexity in Boltzmann Machines

In the past three decades, many theoretical measures of complexity have been proposed to help understand complex systems. In this work, for the first time, we place these measures on a level playing field, to explore the qualitative similarities and differences between them, and their shortcomings. Specifically, using the Boltzmann machine architecture (a fully connected recurrent neural network) with uniformly distributed weights as our model of study, we numerically measure how complexity changes as a function of network dynamics and network parameters. We apply an extension of one such information-theoretic measure of complexity to understand incremental Hebbian learning in Hopfield networks, a fully recurrent architecture model of autoassociative memory. In the course of Hebbian learning, the total information flow reflects a natural upward trend in complexity as the network attempts to learn more and more patterns.Comment: 16 pages, 7 figures; Appears in Entropy, Special Issue "Information Geometry II

Synchronization of heterogeneous oscillators under network modifications: Perturbation and optimization of the synchrony alignment function

Synchronization is central to many complex systems in engineering physics (e.g., the power-grid, Josephson junction circuits, and electro-chemical oscillators) and biology (e.g., neuronal, circadian, and cardiac rhythms). Despite these widespread applications---for which proper functionality depends sensitively on the extent of synchronization---there remains a lack of understanding for how systems evolve and adapt to enhance or inhibit synchronization. We study how network modifications affect the synchronization properties of network-coupled dynamical systems that have heterogeneous node dynamics (e.g., phase oscillators with non-identical frequencies), which is often the case for real-world systems. Our approach relies on a synchrony alignment function (SAF) that quantifies the interplay between heterogeneity of the network and of the oscillators and provides an objective measure for a system's ability to synchronize. We conduct a spectral perturbation analysis of the SAF for structural network modifications including the addition and removal of edges, which subsequently ranks the edges according to their importance to synchronization. Based on this analysis, we develop gradient-descent algorithms to efficiently solve optimization problems that aim to maximize phase synchronization via network modifications. We support these and other results with numerical experiments.Comment: 25 pages, 6 figure