16,981 research outputs found
An -Regularization Approach to High-Dimensional Errors-in-variables Models
Several new estimation methods have been recently proposed for the linear
regression model with observation error in the design. Different assumptions on
the data generating process have motivated different estimators and analysis.
In particular, the literature considered (1) observation errors in the design
uniformly bounded by some , and (2) zero mean independent
observation errors. Under the first assumption, the rates of convergence of the
proposed estimators depend explicitly on , while the second
assumption has been applied when an estimator for the second moment of the
observational error is available. This work proposes and studies two new
estimators which, compared to other procedures for regression models with
errors in the design, exploit an additional -norm regularization.
The first estimator is applicable when both (1) and (2) hold but does not
require an estimator for the second moment of the observational error. The
second estimator is applicable under (2) and requires an estimator for the
second moment of the observation error. Importantly, we impose no assumption on
the accuracy of this pilot estimator, in contrast to the previously known
procedures. As the recent proposals, we allow the number of covariates to be
much larger than the sample size. We establish the rates of convergence of the
estimators and compare them with the bounds obtained for related estimators in
the literature. These comparisons show interesting insights on the interplay of
the assumptions and the achievable rates of convergence
Sparse Regression Learning by Aggregation and Langevin Monte-Carlo
We consider the problem of regression learning for deterministic design and
independent random errors. We start by proving a sharp PAC-Bayesian type bound
for the exponentially weighted aggregate (EWA) under the expected squared
empirical loss. For a broad class of noise distributions the presented bound is
valid whenever the temperature parameter of the EWA is larger than or
equal to , where is the noise variance. A remarkable
feature of this result is that it is valid even for unbounded regression
functions and the choice of the temperature parameter depends exclusively on
the noise level. Next, we apply this general bound to the problem of
aggregating the elements of a finite-dimensional linear space spanned by a
dictionary of functions . We allow to be much larger
than the sample size but we assume that the true regression function can be
well approximated by a sparse linear combination of functions . Under
this sparsity scenario, we propose an EWA with a heavy tailed prior and we show
that it satisfies a sparsity oracle inequality with leading constant one.
Finally, we propose several Langevin Monte-Carlo algorithms to approximately
compute such an EWA when the number of aggregated functions can be large.
We discuss in some detail the convergence of these algorithms and present
numerical experiments that confirm our theoretical findings.Comment: Short version published in COLT 200
- …