3,785 research outputs found
Efficient Smoothed Concomitant Lasso Estimation for High Dimensional Regression
In high dimensional settings, sparse structures are crucial for efficiency,
both in term of memory, computation and performance. It is customary to
consider penalty to enforce sparsity in such scenarios. Sparsity
enforcing methods, the Lasso being a canonical example, are popular candidates
to address high dimension. For efficiency, they rely on tuning a parameter
trading data fitting versus sparsity. For the Lasso theory to hold this tuning
parameter should be proportional to the noise level, yet the latter is often
unknown in practice. A possible remedy is to jointly optimize over the
regression parameter as well as over the noise level. This has been considered
under several names in the literature: Scaled-Lasso, Square-root Lasso,
Concomitant Lasso estimation for instance, and could be of interest for
confidence sets or uncertainty quantification. In this work, after illustrating
numerical difficulties for the Smoothed Concomitant Lasso formulation, we
propose a modification we coined Smoothed Concomitant Lasso, aimed at
increasing numerical stability. We propose an efficient and accurate solver
leading to a computational cost no more expansive than the one for the Lasso.
We leverage on standard ingredients behind the success of fast Lasso solvers: a
coordinate descent algorithm, combined with safe screening rules to achieve
speed efficiency, by eliminating early irrelevant features
Two-Layer Feature Reduction for Sparse-Group Lasso via Decomposition of Convex Sets
Sparse-Group Lasso (SGL) has been shown to be a powerful regression technique
for simultaneously discovering group and within-group sparse patterns by using
a combination of the and norms. However, in large-scale
applications, the complexity of the regularizers entails great computational
challenges. In this paper, we propose a novel Two-Layer Feature REduction
method (TLFre) for SGL via a decomposition of its dual feasible set. The
two-layer reduction is able to quickly identify the inactive groups and the
inactive features, respectively, which are guaranteed to be absent from the
sparse representation and can be removed from the optimization. Existing
feature reduction methods are only applicable for sparse models with one
sparsity-inducing regularizer. To our best knowledge, TLFre is the first one
that is capable of dealing with multiple sparsity-inducing regularizers.
Moreover, TLFre has a very low computational cost and can be integrated with
any existing solvers. We also develop a screening method---called DPC
(DecomPosition of Convex set)---for the nonnegative Lasso problem. Experiments
on both synthetic and real data sets show that TLFre and DPC improve the
efficiency of SGL and nonnegative Lasso by several orders of magnitude
- …