Sampling from Gaussian graphical models using subgraph perturbations

The problem of efficiently drawing samples from a Gaussian graphical model or Gaussian Markov random field is studied. We introduce the subgraph perturbation sampling algorithm, which makes use of any pre-existing tractable inference algorithm for a subgraph by perturbing this algorithm so as to yield asymptotically exact samples for the intended distribution. The subgraph can have any structure for which efficient inference algorithms exist: for example, tree-structured, low tree-width, or having a small feedback vertex set. The experimental results demonstrate that this subgraph perturbation algorithm efficiently yields accurate samples for many graph topologies.United States. Air Force Office of Scientific Research (Grant FA9550-12-1-0287

Sampling decomposable graphs using a Markov chain on junction trees

Full Bayesian computational inference for model determination in undirected graphical models is currently restricted to decomposable graphs, except for problems of very small scale. In this paper we develop new, more efficient methodology for such inference, by making two contributions to the computational geometry of decomposable graphs. The first of these provides sufficient conditions under which it is possible to completely connect two disconnected complete subsets of vertices, or perform the reverse procedure, yet maintain decomposability of the graph. The second is a new Markov chain Monte Carlo sampler for arbitrary positive distributions on decomposable graphs, taking a junction tree representing the graph as its state variable. The resulting methodology is illustrated with numerical experiments on three specific models.Comment: 22 pages, 7 figures, 1 table. V2 as V1 except that Fig 1 was corrected. V3 has significant edits, dropping some figures and including additional examples and a discussion of the non-decomposable case. V4 is further edited following review, and includes additional reference

Characterizing the Shape of Activation Space in Deep Neural Networks

The representations learned by deep neural networks are difficult to interpret in part due to their large parameter space and the complexities introduced by their multi-layer structure. We introduce a method for computing persistent homology over the graphical activation structure of neural networks, which provides access to the task-relevant substructures activated throughout the network for a given input. This topological perspective provides unique insights into the distributed representations encoded by neural networks in terms of the shape of their activation structures. We demonstrate the value of this approach by showing an alternative explanation for the existence of adversarial examples. By studying the topology of network activations across multiple architectures and datasets, we find that adversarial perturbations do not add activations that target the semantic structure of the adversarial class as previously hypothesized. Rather, adversarial examples are explainable as alterations to the dominant activation structures induced by the original image, suggesting the class representations learned by deep networks are problematically sparse on the input space

Sequential sampling of junction trees for decomposable graphs

The junction-tree representation provides an attractive structural property for organizing a decomposable graph. In this study, we present a novel stochastic algorithm, which we call the junction-tree expander, for sequential sampling of junction trees for decomposable graphs. We show that recursive application of the junction-tree expander, expanding incrementally the underlying graph with one vertex at a time, has full support on the space of junction trees with any given number of underlying vertices. A direct application of our suggested algorithm is demonstrated in a sequential Monte Carlo setting designed for sampling from distributions on spaces of decomposable graphs, where the junction-tree expander can be effectively employed as proposal kernel; see the companion paper Olsson et al. 2019 [16]. A numerical study illustrates the utility of our approach by two examples: in the first one, how the junction-tree expander can be incorporated successfully into a particle Gibbs sampler for Bayesian structure learning in decomposable graphical models; in the second one, we provide an unbiased estimator of the number of decomposable graphs for a given number of vertices. All the methods proposed in the paper are implemented in the Python library trilearn.Comment: 31 pages, 7 figure

Parallel sampling of decomposable graphs using Markov chain on junction trees

Bayesian inference for undirected graphical models is mostly restricted to the class of decomposable graphs, as they enjoy a rich set of properties making them amenable to high-dimensional problems. While parameter inference is straightforward in this setup, inferring the underlying graph is a challenge driven by the computational difficultly in exploring the space of decomposable graphs. This work makes two contributions to address this problem. First, we provide sufficient and necessary conditions for when multi-edge perturbations maintain decomposability of the graph. Using these, we characterize a simple class of partitions that efficiently classify all edge perturbations by whether they maintain decomposability. Second, we propose a new parallel non-reversible Markov chain Monte Carlo sampler for distributions over junction tree representations of the graph, where at every step, all edge perturbations within a partition are executed simultaneously. Through simulations, we demonstrate the efficiency of our new edge perturbation conditions and class of partitions. We find that our parallel sampler yields improved mixing properties in comparison to the single-move variate, and outperforms current methods. The implementation of our work is available in a Python package.Comment: 20 pages, 10 figures, with appendix and supplementary materia

Next nearest neighbour Ising models on random graphs

This paper develops results for the next nearest neighbour Ising model on random graphs. Besides being an essential ingredient in classic models for frustrated systems, second neighbour interactions interactions arise naturally in several applications such as the colour diversity problem and graphical games. We demonstrate ensembles of random graphs, including regular connectivity graphs, that have a periodic variation of free energy, with either the ratio of nearest to next nearest couplings, or the mean number of nearest neighbours. When the coupling ratio is integer paramagnetic phases can be found at zero temperature. This is shown to be related to the locked or unlocked nature of the interactions. For anti-ferromagnetic couplings, spin glass phases are demonstrated at low temperature. The interaction structure is formulated as a factor graph, the solution on a tree is developed. The replica symmetric and energetic one-step replica symmetry breaking solution is developed using the cavity method. We calculate within these frameworks the phase diagram and demonstrate the existence of dynamical transitions at zero temperature for cases of anti-ferromagnetic coupling on regular and inhomogeneous random graphs.Comment: 55 pages, 15 figures, version 2 with minor revisions, to be published J. Stat. Mec

Searching for a Source of Difference: a Graphical Model Approach

In this work, we look at a two-sample problem within the framework of Gaussian graphical models. When the global hypothesis of equality of two distributions is rejected, the interest is usually in localizing the source of di erence. Motivated by the idea that diseases can be seen as system perturbations, and by the need to distinguish between the origin of perturbation and components aected by the perturbation, we introduce the concept of a minimal seed set, and its graphical counterpart a graphical seed set. They intuitively consist of variables driving the dierence between the two conditions. We propose a simple and fast testing procedure to estimate the graphical seed set from data, and study its nite sample behavior with a stimulation study. We illustrate our approach in the context of gene set analysis by means of a publicly available gene expression dataset

Modeling Network Populations via Graph Distances

This article introduces a new class of models for multiple networks. The core idea is to parametrize a distribution on labelled graphs in terms of a Fr\'{e}chet mean graph (which depends on a user-specified choice of metric or graph distance) and a parameter that controls the concentration of this distribution about its mean. Entropy is the natural parameter for such control, varying from a point mass concentrated on the Fr\'{e}chet mean itself to a uniform distribution over all graphs on a given vertex set. We provide a hierarchical Bayesian approach for exploiting this construction, along with straightforward strategies for sampling from the resultant posterior distribution. We conclude by demonstrating the efficacy of our approach via simulation studies and two multiple-network data analysis examples: one drawn from systems biology and the other from neuroscience.Comment: 33 pages, 8 figure