7,278 research outputs found
Transductive versions of the LASSO and the Dantzig Selector
We consider the linear regression problem, where the number of covariates
is possibly larger than the number of observations , under sparsity assumptions. On the one hand, several methods have
been successfully proposed to perform this task, for example the LASSO or the
Dantzig Selector. On the other hand, consider new values . If one wants to estimate the corresponding 's, one should
think of a specific estimator devoted to this task, referred by Vapnik as a
"transductive" estimator. This estimator may differ from an estimator designed
to the more general task "estimate on the whole domain". In this work, we
propose a generalized version both of the LASSO and the Dantzig Selector, based
on the geometrical remarks about the LASSO in pr\'evious works. The "usual"
LASSO and Dantzig Selector, as well as new estimators interpreted as
transductive versions of the LASSO, appear as special cases. These estimators
are interesting at least from a theoretical point of view: we can give
theoretical guarantees for these estimators under hypotheses that are relaxed
versions of the hypotheses required in the papers about the "usual" LASSO.
These estimators can also be efficiently computed, with results comparable to
the ones of the LASSO
On Lasso refitting strategies
A well-know drawback of l_1-penalized estimators is the systematic shrinkage
of the large coefficients towards zero. A simple remedy is to treat Lasso as a
model-selection procedure and to perform a second refitting step on the
selected support. In this work we formalize the notion of refitting and provide
oracle bounds for arbitrary refitting procedures of the Lasso solution. One of
the most widely used refitting techniques which is based on Least-Squares may
bring a problem of interpretability, since the signs of the refitted estimator
might be flipped with respect to the original estimator. This problem arises
from the fact that the Least-Squares refitting considers only the support of
the Lasso solution, avoiding any information about signs or amplitudes. To this
end we define a sign consistent refitting as an arbitrary refitting procedure,
preserving the signs of the first step Lasso solution and provide Oracle
inequalities for such estimators. Finally, we consider special refitting
strategies: Bregman Lasso and Boosted Lasso. Bregman Lasso has a fruitful
property to converge to the Sign-Least-Squares refitting (Least-Squares with
sign constraints), which provides with greater interpretability. We
additionally study the Bregman Lasso refitting in the case of orthogonal
design, providing with simple intuition behind the proposed method. Boosted
Lasso, in contrast, considers information about magnitudes of the first Lasso
step and allows to develop better oracle rates for prediction. Finally, we
conduct an extensive numerical study to show advantages of one approach over
others in different synthetic and semi-real scenarios.Comment: revised versio
Safe Screening With Variational Inequalities and Its Application to LASSO
Sparse learning techniques have been routinely used for feature selection as
the resulting model usually has a small number of non-zero entries. Safe
screening, which eliminates the features that are guaranteed to have zero
coefficients for a certain value of the regularization parameter, is a
technique for improving the computational efficiency. Safe screening is gaining
increasing attention since 1) solving sparse learning formulations usually has
a high computational cost especially when the number of features is large and
2) one needs to try several regularization parameters to select a suitable
model. In this paper, we propose an approach called "Sasvi" (Safe screening
with variational inequalities). Sasvi makes use of the variational inequality
that provides the sufficient and necessary optimality condition for the dual
problem. Several existing approaches for Lasso screening can be casted as
relaxed versions of the proposed Sasvi, thus Sasvi provides a stronger safe
screening rule. We further study the monotone properties of Sasvi for Lasso,
based on which a sure removal regularization parameter can be identified for
each feature. Experimental results on both synthetic and real data sets are
reported to demonstrate the effectiveness of the proposed Sasvi for Lasso
screening.Comment: Accepted by International Conference on Machine Learning 201
Improved variable selection with Forward-Lasso adaptive shrinkage
Recently, considerable interest has focused on variable selection methods in
regression situations where the number of predictors, , is large relative to
the number of observations, . Two commonly applied variable selection
approaches are the Lasso, which computes highly shrunk regression coefficients,
and Forward Selection, which uses no shrinkage. We propose a new approach,
"Forward-Lasso Adaptive SHrinkage" (FLASH), which includes the Lasso and
Forward Selection as special cases, and can be used in both the linear
regression and the Generalized Linear Model domains. As with the Lasso and
Forward Selection, FLASH iteratively adds one variable to the model in a
hierarchical fashion but, unlike these methods, at each step adjusts the level
of shrinkage so as to optimize the selection of the next variable. We first
present FLASH in the linear regression setting and show that it can be fitted
using a variant of the computationally efficient LARS algorithm. Then, we
extend FLASH to the GLM domain and demonstrate, through numerous simulations
and real world data sets, as well as some theoretical analysis, that FLASH
generally outperforms many competing approaches.Comment: Published in at http://dx.doi.org/10.1214/10-AOAS375 the Annals of
Applied Statistics (http://www.imstat.org/aoas/) by the Institute of
Mathematical Statistics (http://www.imstat.org
- …