11 research outputs found
High-dimensional Sparse Inverse Covariance Estimation using Greedy Methods
In this paper we consider the task of estimating the non-zero pattern of the
sparse inverse covariance matrix of a zero-mean Gaussian random vector from a
set of iid samples. Note that this is also equivalent to recovering the
underlying graph structure of a sparse Gaussian Markov Random Field (GMRF). We
present two novel greedy approaches to solving this problem. The first
estimates the non-zero covariates of the overall inverse covariance matrix
using a series of global forward and backward greedy steps. The second
estimates the neighborhood of each node in the graph separately, again using
greedy forward and backward steps, and combines the intermediate neighborhoods
to form an overall estimate. The principal contribution of this paper is a
rigorous analysis of the sparsistency, or consistency in recovering the
sparsity pattern of the inverse covariance matrix. Surprisingly, we show that
both the local and global greedy methods learn the full structure of the model
with high probability given just samples, which is a
\emph{significant} improvement over state of the art -regularized
Gaussian MLE (Graphical Lasso) that requires samples. Moreover,
the restricted eigenvalue and smoothness conditions imposed by our greedy
methods are much weaker than the strong irrepresentable conditions required by
the -regularization based methods. We corroborate our results with
extensive simulations and examples, comparing our local and global greedy
methods to the -regularized Gaussian MLE as well as the Neighborhood
Greedy method to that of nodewise -regularized linear regression
(Neighborhood Lasso).Comment: Accepted to AI STAT 2012 for Oral Presentatio
Forward-Backward Greedy Algorithms for General Convex Smooth Functions over A Cardinality Constraint
We consider forward-backward greedy algorithms for solving sparse feature
selection problems with general convex smooth functions. A state-of-the-art
greedy method, the Forward-Backward greedy algorithm (FoBa-obj) requires to
solve a large number of optimization problems, thus it is not scalable for
large-size problems. The FoBa-gdt algorithm, which uses the gradient
information for feature selection at each forward iteration, significantly
improves the efficiency of FoBa-obj. In this paper, we systematically analyze
the theoretical properties of both forward-backward greedy algorithms. Our main
contributions are: 1) We derive better theoretical bounds than existing
analyses regarding FoBa-obj for general smooth convex functions; 2) We show
that FoBa-gdt achieves the same theoretical performance as FoBa-obj under the
same condition: restricted strong convexity condition. Our new bounds are
consistent with the bounds of a special case (least squares) and fills a
previously existing theoretical gap for general convex smooth functions; 3) We
show that the restricted strong convexity condition is satisfied if the number
of independent samples is more than where is the
sparsity number and is the dimension of the variable; 4) We apply FoBa-gdt
(with the conditional random field objective) to the sensor selection problem
for human indoor activity recognition and our results show that FoBa-gdt
outperforms other methods (including the ones based on forward greedy selection
and L1-regularization)
Graph Estimation From Multi-attribute Data
Many real world network problems often concern multivariate nodal attributes
such as image, textual, and multi-view feature vectors on nodes, rather than
simple univariate nodal attributes. The existing graph estimation methods built
on Gaussian graphical models and covariance selection algorithms can not handle
such data, neither can the theories developed around such methods be directly
applied. In this paper, we propose a new principled framework for estimating
graphs from multi-attribute data. Instead of estimating the partial correlation
as in current literature, our method estimates the partial canonical
correlations that naturally accommodate complex nodal features.
Computationally, we provide an efficient algorithm which utilizes the
multi-attribute structure. Theoretically, we provide sufficient conditions
which guarantee consistent graph recovery. Extensive simulation studies
demonstrate performance of our method under various conditions. Furthermore, we
provide illustrative applications to uncovering gene regulatory networks from
gene and protein profiles, and uncovering brain connectivity graph from
functional magnetic resonance imaging data.Comment: Extended simulation study. Added an application to a new data se