2,660 research outputs found
Optimal computational and statistical rates of convergence for sparse nonconvex learning problems
We provide theoretical analysis of the statistical and computational
properties of penalized -estimators that can be formulated as the solution
to a possibly nonconvex optimization problem. Many important estimators fall in
this category, including least squares regression with nonconvex
regularization, generalized linear models with nonconvex regularization and
sparse elliptical random design regression. For these problems, it is
intractable to calculate the global solution due to the nonconvex formulation.
In this paper, we propose an approximate regularization path-following method
for solving a variety of learning problems with nonconvex objective functions.
Under a unified analytic framework, we simultaneously provide explicit
statistical and computational rates of convergence for any local solution
attained by the algorithm. Computationally, our algorithm attains a global
geometric rate of convergence for calculating the full regularization path,
which is optimal among all first-order algorithms. Unlike most existing methods
that only attain geometric rates of convergence for one single regularization
parameter, our algorithm calculates the full regularization path with the same
iteration complexity. In particular, we provide a refined iteration complexity
bound to sharply characterize the performance of each stage along the
regularization path. Statistically, we provide sharp sample complexity analysis
for all the approximate local solutions along the regularization path. In
particular, our analysis improves upon existing results by providing a more
refined sample complexity bound as well as an exact support recovery result for
the final estimator. These results show that the final estimator attains an
oracle statistical property due to the usage of nonconvex penalty.Comment: Published in at http://dx.doi.org/10.1214/14-AOS1238 the Annals of
Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Calibrating nonconvex penalized regression in ultra-high dimension
We investigate high-dimensional nonconvex penalized regression, where the
number of covariates may grow at an exponential rate. Although recent
asymptotic theory established that there exists a local minimum possessing the
oracle property under general conditions, it is still largely an open problem
how to identify the oracle estimator among potentially multiple local minima.
There are two main obstacles: (1) due to the presence of multiple minima, the
solution path is nonunique and is not guaranteed to contain the oracle
estimator; (2) even if a solution path is known to contain the oracle
estimator, the optimal tuning parameter depends on many unknown factors and is
hard to estimate. To address these two challenging issues, we first prove that
an easy-to-calculate calibrated CCCP algorithm produces a consistent solution
path which contains the oracle estimator with probability approaching one.
Furthermore, we propose a high-dimensional BIC criterion and show that it can
be applied to the solution path to select the optimal tuning parameter which
asymptotically identifies the oracle estimator. The theory for a general class
of nonconvex penalties in the ultra-high dimensional setup is established when
the random errors follow the sub-Gaussian distribution. Monte Carlo studies
confirm that the calibrated CCCP algorithm combined with the proposed
high-dimensional BIC has desirable performance in identifying the underlying
sparsity pattern for high-dimensional data analysis.Comment: Published in at http://dx.doi.org/10.1214/13-AOS1159 the Annals of
Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Analysis of Testing-Based Forward Model Selection
This paper introduces and analyzes a procedure called Testing-based forward
model selection (TBFMS) in linear regression problems. This procedure
inductively selects covariates that add predictive power into a working
statistical model before estimating a final regression. The criterion for
deciding which covariate to include next and when to stop including covariates
is derived from a profile of traditional statistical hypothesis tests. This
paper proves probabilistic bounds, which depend on the quality of the tests,
for prediction error and the number of selected covariates. As an example, the
bounds are then specialized to a case with heteroskedastic data, with tests
constructed with the help of Huber-Eicker-White standard errors. Under the
assumed regularity conditions, these tests lead to estimation convergence rates
matching other common high-dimensional estimators including Lasso
Inference in Additively Separable Models With a High-Dimensional Set of Conditioning Variables
This paper studies nonparametric series estimation and inference for the
effect of a single variable of interest x on an outcome y in the presence of
potentially high-dimensional conditioning variables z. The context is an
additively separable model E[y|x, z] = g0(x) + h0(z). The model is
high-dimensional in the sense that the series of approximating functions for
h0(z) can have more terms than the sample size, thereby allowing z to have
potentially very many measured characteristics. The model is required to be
approximately sparse: h0(z) can be approximated using only a small subset of
series terms whose identities are unknown. This paper proposes an estimation
and inference method for g0(x) called Post-Nonparametric Double Selection which
is a generalization of Post-Double Selection. Standard rates of convergence and
asymptotic normality for the estimator are shown to hold uniformly over a large
class of sparse data generating processes. A simulation study illustrates
finite sample estimation properties of the proposed estimator and coverage
properties of the corresponding confidence intervals. Finally, an empirical
application to college admissions policy demonstrates the practical
implementation of the proposed method
- …