4,121 research outputs found
High-dimensional Sparse Inverse Covariance Estimation using Greedy Methods
In this paper we consider the task of estimating the non-zero pattern of the
sparse inverse covariance matrix of a zero-mean Gaussian random vector from a
set of iid samples. Note that this is also equivalent to recovering the
underlying graph structure of a sparse Gaussian Markov Random Field (GMRF). We
present two novel greedy approaches to solving this problem. The first
estimates the non-zero covariates of the overall inverse covariance matrix
using a series of global forward and backward greedy steps. The second
estimates the neighborhood of each node in the graph separately, again using
greedy forward and backward steps, and combines the intermediate neighborhoods
to form an overall estimate. The principal contribution of this paper is a
rigorous analysis of the sparsistency, or consistency in recovering the
sparsity pattern of the inverse covariance matrix. Surprisingly, we show that
both the local and global greedy methods learn the full structure of the model
with high probability given just samples, which is a
\emph{significant} improvement over state of the art -regularized
Gaussian MLE (Graphical Lasso) that requires samples. Moreover,
the restricted eigenvalue and smoothness conditions imposed by our greedy
methods are much weaker than the strong irrepresentable conditions required by
the -regularization based methods. We corroborate our results with
extensive simulations and examples, comparing our local and global greedy
methods to the -regularized Gaussian MLE as well as the Neighborhood
Greedy method to that of nodewise -regularized linear regression
(Neighborhood Lasso).Comment: Accepted to AI STAT 2012 for Oral Presentatio
Analysis of Testing-Based Forward Model Selection
This paper introduces and analyzes a procedure called Testing-based forward
model selection (TBFMS) in linear regression problems. This procedure
inductively selects covariates that add predictive power into a working
statistical model before estimating a final regression. The criterion for
deciding which covariate to include next and when to stop including covariates
is derived from a profile of traditional statistical hypothesis tests. This
paper proves probabilistic bounds, which depend on the quality of the tests,
for prediction error and the number of selected covariates. As an example, the
bounds are then specialized to a case with heteroskedastic data, with tests
constructed with the help of Huber-Eicker-White standard errors. Under the
assumed regularity conditions, these tests lead to estimation convergence rates
matching other common high-dimensional estimators including Lasso
High-Dimensional Boosting: Rate of Convergence
Boosting is one of the most significant developments in machine learning.
This paper studies the rate of convergence of Boosting, which is tailored
for regression, in a high-dimensional setting. Moreover, we introduce so-called
\textquotedblleft post-Boosting\textquotedblright. This is a post-selection
estimator which applies ordinary least squares to the variables selected in the
first stage by Boosting. Another variant is \textquotedblleft Orthogonal
Boosting\textquotedblright\ where after each step an orthogonal projection is
conducted. We show that both post-Boosting and the orthogonal boosting
achieve the same rate of convergence as LASSO in a sparse, high-dimensional
setting. We show that the rate of convergence of the classical Boosting
depends on the design matrix described by a sparse eigenvalue constant. To show
the latter results, we derive new approximation results for the pure greedy
algorithm, based on analyzing the revisiting behavior of Boosting. We also
introduce feasible rules for early stopping, which can be easily implemented
and used in applied work. Our results also allow a direct comparison between
LASSO and boosting which has been missing from the literature. Finally, we
present simulation studies and applications to illustrate the relevance of our
theoretical results and to provide insights into the practical aspects of
boosting. In these simulation studies, post-Boosting clearly outperforms
LASSO.Comment: 19 pages, 4 tables; AMS 2000 subject classifications: Primary 62J05,
62J07, 41A25; secondary 49M15, 68Q3
- β¦