1,139 research outputs found
Sparse Modeling for Image and Vision Processing
In recent years, a large amount of multi-disciplinary research has been
conducted on sparse models and their applications. In statistics and machine
learning, the sparsity principle is used to perform model selection---that is,
automatically selecting a simple model among a large collection of them. In
signal processing, sparse coding consists of representing data with linear
combinations of a few dictionary elements. Subsequently, the corresponding
tools have been widely adopted by several scientific communities such as
neuroscience, bioinformatics, or computer vision. The goal of this monograph is
to offer a self-contained view of sparse modeling for visual recognition and
image processing. More specifically, we focus on applications where the
dictionary is learned and adapted to data, yielding a compact representation
that has been successful in various contexts.Comment: 205 pages, to appear in Foundations and Trends in Computer Graphics
and Visio
Stochastic Training of Neural Networks via Successive Convex Approximations
This paper proposes a new family of algorithms for training neural networks
(NNs). These are based on recent developments in the field of non-convex
optimization, going under the general name of successive convex approximation
(SCA) techniques. The basic idea is to iteratively replace the original
(non-convex, highly dimensional) learning problem with a sequence of (strongly
convex) approximations, which are both accurate and simple to optimize.
Differently from similar ideas (e.g., quasi-Newton algorithms), the
approximations can be constructed using only first-order information of the
neural network function, in a stochastic fashion, while exploiting the overall
structure of the learning problem for a faster convergence. We discuss several
use cases, based on different choices for the loss function (e.g., squared loss
and cross-entropy loss), and for the regularization of the NN's weights. We
experiment on several medium-sized benchmark problems, and on a large-scale
dataset involving simulated physical data. The results show how the algorithm
outperforms state-of-the-art techniques, providing faster convergence to a
better minimum. Additionally, we show how the algorithm can be easily
parallelized over multiple computational units without hindering its
performance. In particular, each computational unit can optimize a tailored
surrogate function defined on a randomly assigned subset of the input
variables, whose dimension can be selected depending entirely on the available
computational power.Comment: Preprint submitted to IEEE Transactions on Neural Networks and
Learning System
Smoothing proximal gradient method for general structured sparse regression
We study the problem of estimating high-dimensional regression models
regularized by a structured sparsity-inducing penalty that encodes prior
structural information on either the input or output variables. We consider two
widely adopted types of penalties of this kind as motivating examples: (1) the
general overlapping-group-lasso penalty, generalized from the group-lasso
penalty; and (2) the graph-guided-fused-lasso penalty, generalized from the
fused-lasso penalty. For both types of penalties, due to their nonseparability
and nonsmoothness, developing an efficient optimization method remains a
challenging problem. In this paper we propose a general optimization approach,
the smoothing proximal gradient (SPG) method, which can solve structured sparse
regression problems with any smooth convex loss under a wide spectrum of
structured sparsity-inducing penalties. Our approach combines a smoothing
technique with an effective proximal gradient method. It achieves a convergence
rate significantly faster than the standard first-order methods, subgradient
methods, and is much more scalable than the most widely used interior-point
methods. The efficiency and scalability of our method are demonstrated on both
simulation experiments and real genetic data sets.Comment: Published in at http://dx.doi.org/10.1214/11-AOAS514 the Annals of
Applied Statistics (http://www.imstat.org/aoas/) by the Institute of
Mathematical Statistics (http://www.imstat.org
- …