419 research outputs found
Aggregation of Affine Estimators
We consider the problem of aggregating a general collection of affine
estimators for fixed design regression. Relevant examples include some commonly
used statistical estimators such as least squares, ridge and robust least
squares estimators. Dalalyan and Salmon (2012) have established that, for this
problem, exponentially weighted (EW) model selection aggregation leads to sharp
oracle inequalities in expectation, but similar bounds in deviation were not
previously known. While results indicate that the same aggregation scheme may
not satisfy sharp oracle inequalities with high probability, we prove that a
weaker notion of oracle inequality for EW that holds with high probability.
Moreover, using a generalization of the newly introduced -aggregation scheme
we also prove sharp oracle inequalities that hold with high probability.
Finally, we apply our results to universal aggregation and show that our
proposed estimator leads simultaneously to all the best known bounds for
aggregation, including -aggregation, , with high
probability
Sharp Oracle Inequalities for Aggregation of Affine Estimators
We consider the problem of combining a (possibly uncountably infinite) set of
affine estimators in non-parametric regression model with heteroscedastic
Gaussian noise. Focusing on the exponentially weighted aggregate, we prove a
PAC-Bayesian type inequality that leads to sharp oracle inequalities in
discrete but also in continuous settings. The framework is general enough to
cover the combinations of various procedures such as least square regression,
kernel ridge regression, shrinking estimators and many other estimators used in
the literature on statistical inverse problems. As a consequence, we show that
the proposed aggregate provides an adaptive estimator in the exact minimax
sense without neither discretizing the range of tuning parameters nor splitting
the set of observations. We also illustrate numerically the good performance
achieved by the exponentially weighted aggregate
Pac-bayesian bounds for sparse regression estimation with exponential weights
We consider the sparse regression model where the number of parameters is
larger than the sample size . The difficulty when considering
high-dimensional problems is to propose estimators achieving a good compromise
between statistical and computational performances. The BIC estimator for
instance performs well from the statistical point of view \cite{BTW07} but can
only be computed for values of of at most a few tens. The Lasso estimator
is solution of a convex minimization problem, hence computable for large value
of . However stringent conditions on the design are required to establish
fast rates of convergence for this estimator. Dalalyan and Tsybakov
\cite{arnak} propose a method achieving a good compromise between the
statistical and computational aspects of the problem. Their estimator can be
computed for reasonably large and satisfies nice statistical properties
under weak assumptions on the design. However, \cite{arnak} proposes sparsity
oracle inequalities in expectation for the empirical excess risk only. In this
paper, we propose an aggregation procedure similar to that of \cite{arnak} but
with improved statistical performances. Our main theoretical result is a
sparsity oracle inequality in probability for the true excess risk for a
version of exponential weight estimator. We also propose a MCMC method to
compute our estimator for reasonably large values of .Comment: 19 page
Kullback-Leibler aggregation and misspecified generalized linear models
In a regression setup with deterministic design, we study the pure
aggregation problem and introduce a natural extension from the Gaussian
distribution to distributions in the exponential family. While this extension
bears strong connections with generalized linear models, it does not require
identifiability of the parameter or even that the model on the systematic
component is true. It is shown that this problem can be solved by constrained
and/or penalized likelihood maximization and we derive sharp oracle
inequalities that hold both in expectation and with high probability. Finally
all the bounds are proved to be optimal in a minimax sense.Comment: Published in at http://dx.doi.org/10.1214/11-AOS961 the Annals of
Statistics (http://www.imstat.org/aos/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Sparse Estimation by Exponential Weighting
Consider a regression model with fixed design and Gaussian noise where the
regression function can potentially be well approximated by a function that
admits a sparse representation in a given dictionary. This paper resorts to
exponential weights to exploit this underlying sparsity by implementing the
principle of sparsity pattern aggregation. This model selection take on sparse
estimation allows us to derive sparsity oracle inequalities in several popular
frameworks, including ordinary sparsity, fused sparsity and group sparsity. One
striking aspect of these theoretical results is that they hold under no
condition in the dictionary. Moreover, we describe an efficient implementation
of the sparsity pattern aggregation principle that compares favorably to
state-of-the-art procedures on some basic numerical examples.Comment: Published in at http://dx.doi.org/10.1214/12-STS393 the Statistical
Science (http://www.imstat.org/sts/) by the Institute of Mathematical
Statistics (http://www.imstat.org
Solution of linear ill-posed problems by model selection and aggregation
We consider a general statistical linear inverse problem, where the solution
is represented via a known (possibly overcomplete) dictionary that allows its
sparse representation. We propose two different approaches. A model selection
estimator selects a single model by minimizing the penalized empirical risk
over all possible models. By contrast with direct problems, the penalty depends
on the model itself rather than on its size only as for complexity penalties. A
Q-aggregate estimator averages over the entire collection of estimators with
properly chosen weights. Under mild conditions on the dictionary, we establish
oracle inequalities both with high probability and in expectation for the two
estimators. Moreover, for the latter estimator these inequalities are sharp.
The proposed procedures are implemented numerically and their performance is
assessed by a simulation study.Comment: 20 pages, 2 figure
- …