14,719 research outputs found
High-dimensional Sparse Inverse Covariance Estimation using Greedy Methods
In this paper we consider the task of estimating the non-zero pattern of the
sparse inverse covariance matrix of a zero-mean Gaussian random vector from a
set of iid samples. Note that this is also equivalent to recovering the
underlying graph structure of a sparse Gaussian Markov Random Field (GMRF). We
present two novel greedy approaches to solving this problem. The first
estimates the non-zero covariates of the overall inverse covariance matrix
using a series of global forward and backward greedy steps. The second
estimates the neighborhood of each node in the graph separately, again using
greedy forward and backward steps, and combines the intermediate neighborhoods
to form an overall estimate. The principal contribution of this paper is a
rigorous analysis of the sparsistency, or consistency in recovering the
sparsity pattern of the inverse covariance matrix. Surprisingly, we show that
both the local and global greedy methods learn the full structure of the model
with high probability given just samples, which is a
\emph{significant} improvement over state of the art -regularized
Gaussian MLE (Graphical Lasso) that requires samples. Moreover,
the restricted eigenvalue and smoothness conditions imposed by our greedy
methods are much weaker than the strong irrepresentable conditions required by
the -regularization based methods. We corroborate our results with
extensive simulations and examples, comparing our local and global greedy
methods to the -regularized Gaussian MLE as well as the Neighborhood
Greedy method to that of nodewise -regularized linear regression
(Neighborhood Lasso).Comment: Accepted to AI STAT 2012 for Oral Presentatio
Finding Non-overlapping Clusters for Generalized Inference Over Graphical Models
Graphical models use graphs to compactly capture stochastic dependencies
amongst a collection of random variables. Inference over graphical models
corresponds to finding marginal probability distributions given joint
probability distributions. In general, this is computationally intractable,
which has led to a quest for finding efficient approximate inference
algorithms. We propose a framework for generalized inference over graphical
models that can be used as a wrapper for improving the estimates of approximate
inference algorithms. Instead of applying an inference algorithm to the
original graph, we apply the inference algorithm to a block-graph, defined as a
graph in which the nodes are non-overlapping clusters of nodes from the
original graph. This results in marginal estimates of a cluster of nodes, which
we further marginalize to get the marginal estimates of each node. Our proposed
block-graph construction algorithm is simple, efficient, and motivated by the
observation that approximate inference is more accurate on graphs with longer
cycles. We present extensive numerical simulations that illustrate our
block-graph framework with a variety of inference algorithms (e.g., those in
the libDAI software package). These simulations show the improvements provided
by our framework.Comment: Extended the previous version to include extensive numerical
simulations. See http://www.ima.umn.edu/~dvats/GeneralizedInference.html for
code and dat
Partition MCMC for inference on acyclic digraphs
Acyclic digraphs are the underlying representation of Bayesian networks, a
widely used class of probabilistic graphical models. Learning the underlying
graph from data is a way of gaining insights about the structural properties of
a domain. Structure learning forms one of the inference challenges of
statistical graphical models.
MCMC methods, notably structure MCMC, to sample graphs from the posterior
distribution given the data are probably the only viable option for Bayesian
model averaging. Score modularity and restrictions on the number of parents of
each node allow the graphs to be grouped into larger collections, which can be
scored as a whole to improve the chain's convergence. Current examples of
algorithms taking advantage of grouping are the biased order MCMC, which acts
on the alternative space of permuted triangular matrices, and non ergodic edge
reversal moves.
Here we propose a novel algorithm, which employs the underlying combinatorial
structure of DAGs to define a new grouping. As a result convergence is improved
compared to structure MCMC, while still retaining the property of producing an
unbiased sample. Finally the method can be combined with edge reversal moves to
improve the sampler further.Comment: Revised version. 34 pages, 16 figures. R code available at
https://github.com/annlia/partitionMCM
- …