158 research outputs found
A Screening Strategy for Structured Optimization Involving Nonconvex Regularization
In this paper, we develop a simple yet effective screening rule strategy to
improve the computational efficiency in solving structured optimization
involving nonconvex regularization. Based on an iteratively
reweighted (IRL1) framework, the proposed screening rule works like a
preprocessing module that potentially removes the inactive groups before
starting the subproblem solver, thereby reducing the computational time in
total. This is mainly achieved by heuristically exploiting the dual subproblem
information during each iteration.Moreover, we prove that our screening rule
can remove all inactive variables in a finite number of iterations of the IRL1
method. Numerical experiments illustrate the efficiency of our screening rule
strategy compared with several state-of-the-art algorithms
Screening for Sparse Online Learning
Sparsity promoting regularizers are widely used to impose low-complexity
structure (e.g. l1-norm for sparsity) to the regression coefficients of
supervised learning. In the realm of deterministic optimization, the sequence
generated by iterative algorithms (such as proximal gradient descent) exhibit
"finite activity identification", namely, they can identify the low-complexity
structure in a finite number of iterations. However, most online algorithms
(such as proximal stochastic gradient descent) do not have the property owing
to the vanishing step-size and non-vanishing variance. In this paper, by
combining with a screening rule, we show how to eliminate useless features of
the iterates generated by online algorithms, and thereby enforce finite
activity identification. One consequence is that when combined with any
convergent online algorithm, sparsity properties imposed by the regularizer can
be exploited for computational gains. Numerically, significant acceleration can
be obtained
Nonsmoothness in Machine Learning: specific structure, proximal identification, and applications
Nonsmoothness is often a curse for optimization; but it is sometimes a
blessing, in particular for applications in machine learning. In this paper, we
present the specific structure of nonsmooth optimization problems appearing in
machine learning and illustrate how to leverage this structure in practice, for
compression, acceleration, or dimension reduction. We pay a special attention
to the presentation to make it concise and easily accessible, with both simple
examples and general results
- …