5,310 research outputs found
On Lasso refitting strategies
A well-know drawback of l_1-penalized estimators is the systematic shrinkage
of the large coefficients towards zero. A simple remedy is to treat Lasso as a
model-selection procedure and to perform a second refitting step on the
selected support. In this work we formalize the notion of refitting and provide
oracle bounds for arbitrary refitting procedures of the Lasso solution. One of
the most widely used refitting techniques which is based on Least-Squares may
bring a problem of interpretability, since the signs of the refitted estimator
might be flipped with respect to the original estimator. This problem arises
from the fact that the Least-Squares refitting considers only the support of
the Lasso solution, avoiding any information about signs or amplitudes. To this
end we define a sign consistent refitting as an arbitrary refitting procedure,
preserving the signs of the first step Lasso solution and provide Oracle
inequalities for such estimators. Finally, we consider special refitting
strategies: Bregman Lasso and Boosted Lasso. Bregman Lasso has a fruitful
property to converge to the Sign-Least-Squares refitting (Least-Squares with
sign constraints), which provides with greater interpretability. We
additionally study the Bregman Lasso refitting in the case of orthogonal
design, providing with simple intuition behind the proposed method. Boosted
Lasso, in contrast, considers information about magnitudes of the first Lasso
step and allows to develop better oracle rates for prediction. Finally, we
conduct an extensive numerical study to show advantages of one approach over
others in different synthetic and semi-real scenarios.Comment: revised versio
Model Selection: Two Fundamental Measures of Coherence and Their Algorithmic Significance
The problem of model selection arises in a number of contexts, such as
compressed sensing, subset selection in linear regression, estimation of
structures in graphical models, and signal denoising. This paper generalizes
the notion of \emph{incoherence} in the existing literature on model selection
and introduces two fundamental measures of coherence---termed as the worst-case
coherence and the average coherence---among the columns of a design matrix. In
particular, it utilizes these two measures of coherence to provide an in-depth
analysis of a simple one-step thresholding (OST) algorithm for model selection.
One of the key insights offered by the ensuing analysis is that OST is feasible
for model selection as long as the design matrix obeys an easily verifiable
property. In addition, the paper also characterizes the model-selection
performance of OST in terms of the worst-case coherence, \mu, and establishes
that OST performs near-optimally in the low signal-to-noise ratio regime for N
x C design matrices with \mu = O(N^{-1/2}). Finally, in contrast to some of the
existing literature on model selection, the analysis in the paper is
nonasymptotic in nature, it does not require knowledge of the true model order,
it is applicable to generic (random or deterministic) design matrices, and it
neither requires submatrices of the design matrix to have full rank, nor does
it assume a statistical prior on the values of the nonzero entries of the data
vector.Comment: 5 pages; Accepted for Proc. 2010 IEEE International Symposium on
Information Theory (ISIT 2010
Nonparametric Independence Screening in Sparse Ultra-High Dimensional Additive Models
A variable screening procedure via correlation learning was proposed Fan and
Lv (2008) to reduce dimensionality in sparse ultra-high dimensional models.
Even when the true model is linear, the marginal regression can be highly
nonlinear. To address this issue, we further extend the correlation learning to
marginal nonparametric learning. Our nonparametric independence screening is
called NIS, a specific member of the sure independence screening. Several
closely related variable screening procedures are proposed. Under the
nonparametric additive models, it is shown that under some mild technical
conditions, the proposed independence screening methods enjoy a sure screening
property. The extent to which the dimensionality can be reduced by independence
screening is also explicitly quantified. As a methodological extension, an
iterative nonparametric independence screening (INIS) is also proposed to
enhance the finite sample performance for fitting sparse additive models. The
simulation results and a real data analysis demonstrate that the proposed
procedure works well with moderate sample size and large dimension and performs
better than competing methods.Comment: 48 page
- …