Natural Wake-Sleep Algorithm

The benefits of using the natural gradient are well known in a wide range of optimization problems. However, for the training of common neural networks the resulting increase in computational complexity sets a limitation to its practical application. Helmholtz Machines are a particular type of generative model composed of two Sigmoid Belief Networks (SBNs), acting as an encoder and a decoder, commonly trained using the Wake-Sleep (WS) algorithm and its reweighted version RWS. For SBNs, it has been shown how the locality of the connections in the graphical structure induces sparsity in the Fisher information matrix. The resulting block diagonal structure can be efficiently exploited to reduce the computational complexity of the Fisher matrix inversion and thus compute the natural gradient exactly, without the need of approximations. We present a geometric adaptation of well-known methods from the literature, introducing the Natural Wake-Sleep (NWS) and the Natural Reweighted Wake-Sleep (NRWS) algorithms. We present an experimental analysis of the novel geometrical algorithms based on the convergence speed and the value of the log-likelihood, both with respect to the number of iterations and the time complexity and demonstrating improvements on these aspects over their respective non-geometric baselines.Comment: 19 pages, 9 figure

Quantum Gravity: Has Spacetime Quantum Properties?

The incompatibility between GR and QM is generally seen as a sufficient motivation for the development of a theory of Quantum Gravity. If - so a typical argumentation - QM gives a universally valid basis for the description of all natural systems, then the gravitational field should have quantum properties. Together with the arguments against semi-classical theories of gravity, this leads to a strategy which takes a quantization of GR as the natural avenue to Quantum Gravity. And a quantization of the gravitational field would in some sense correspond to a quantization of geometry. Spacetime would have quantum properties. But, this strategy will only be successful, if gravity is a fundamental interaction. - What, if gravity is instead an intrinsically classical phenomenon? Then, if QM is nevertheless fundamentally valid, gravity can not be a fundamental interaction. An intrinsically classical gravity in a quantum world would have to be an emergent, induced or residual, macroscopic effect, caused by other interactions. The gravitational field (as well as spacetime) would not have any quantum properties. A quantization of GR would lead to artifacts without any relation to nature. The serious problems of all approaches to Quantum Gravity that start from a direct quantization of GR or try to capture the quantum properties of gravity in form of a 'graviton' dynamics - together with the, meanwhile, rich spectrum of approaches to an emergent gravity and/or spacetime - make this latter option more and more interesting for the development of a theory of Quantum Gravity. The most advanced emergent gravity (and spacetime) scenarios are of an information-theoretical, quantum-computational type.Comment: 31 page

Recovering the Graph Underlying Networked Dynamical Systems under Partial Observability: A Deep Learning Approach

We study the problem of graph structure identification, i.e., of recovering the graph of dependencies among time series. We model these time series data as components of the state of linear stochastic networked dynamical systems. We assume partial observability, where the state evolution of only a subset of nodes comprising the network is observed. We devise a new feature vector computed from the observed time series and prove that these features are linearly separable, i.e., there exists a hyperplane that separates the cluster of features associated with connected pairs of nodes from those associated with disconnected pairs. This renders the features amenable to train a variety of classifiers to perform causal inference. In particular, we use these features to train Convolutional Neural Networks (CNNs). The resulting causal inference mechanism outperforms state-of-the-art counterparts w.r.t. sample-complexity. The trained CNNs generalize well over structurally distinct networks (dense or sparse) and noise-level profiles. Remarkably, they also generalize well to real-world networks while trained over a synthetic network (realization of a random graph). Finally, the proposed method consistently reconstructs the graph in a pairwise manner, that is, by deciding if an edge or arrow is present or absent in each pair of nodes, from the corresponding time series of each pair. This fits the framework of large-scale systems, where observation or processing of all nodes in the network is prohibitive.Comment: Accepted at The 37th AAAI Conference on Artificial Intelligence (main track