167 research outputs found

### Optimal computational and statistical rates of convergence for sparse nonconvex learning problems

We provide theoretical analysis of the statistical and computational
properties of penalized $M$-estimators that can be formulated as the solution
to a possibly nonconvex optimization problem. Many important estimators fall in
this category, including least squares regression with nonconvex
regularization, generalized linear models with nonconvex regularization and
sparse elliptical random design regression. For these problems, it is
intractable to calculate the global solution due to the nonconvex formulation.
In this paper, we propose an approximate regularization path-following method
for solving a variety of learning problems with nonconvex objective functions.
Under a unified analytic framework, we simultaneously provide explicit
statistical and computational rates of convergence for any local solution
attained by the algorithm. Computationally, our algorithm attains a global
geometric rate of convergence for calculating the full regularization path,
which is optimal among all first-order algorithms. Unlike most existing methods
that only attain geometric rates of convergence for one single regularization
parameter, our algorithm calculates the full regularization path with the same
iteration complexity. In particular, we provide a refined iteration complexity
bound to sharply characterize the performance of each stage along the
regularization path. Statistically, we provide sharp sample complexity analysis
for all the approximate local solutions along the regularization path. In
particular, our analysis improves upon existing results by providing a more
refined sample complexity bound as well as an exact support recovery result for
the final estimator. These results show that the final estimator attains an
oracle statistical property due to the usage of nonconvex penalty.Comment: Published in at http://dx.doi.org/10.1214/14-AOS1238 the Annals of
Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical
Statistics (http://www.imstat.org

### Optimal linear estimation under unknown nonlinear transform

Linear regression studies the problem of estimating a model parameter
$\beta^* \in \mathbb{R}^p$, from $n$ observations
$\{(y_i,\mathbf{x}_i)\}_{i=1}^n$ from linear model $y_i = \langle
\mathbf{x}_i,\beta^* \rangle + \epsilon_i$. We consider a significant
generalization in which the relationship between $\langle \mathbf{x}_i,\beta^*
\rangle$ and $y_i$ is noisy, quantized to a single bit, potentially nonlinear,
noninvertible, as well as unknown. This model is known as the single-index
model in statistics, and, among other things, it represents a significant
generalization of one-bit compressed sensing. We propose a novel spectral-based
estimation procedure and show that we can recover $\beta^*$ in settings (i.e.,
classes of link function $f$) where previous algorithms fail. In general, our
algorithm requires only very mild restrictions on the (unknown) functional
relationship between $y_i$ and $\langle \mathbf{x}_i,\beta^* \rangle$. We also
consider the high dimensional setting where $\beta^*$ is sparse ,and introduce
a two-stage nonconvex framework that addresses estimation challenges in high
dimensional regimes where $p \gg n$. For a broad class of link functions
between $\langle \mathbf{x}_i,\beta^* \rangle$ and $y_i$, we establish minimax
lower bounds that demonstrate the optimality of our estimators in both the
classical and high dimensional regimes.Comment: 25 pages, 3 figure

- â€¦