38,733 research outputs found
Stochastic Parallel Block Coordinate Descent for Large-scale Saddle Point Problems
We consider convex-concave saddle point problems with a separable structure
and non-strongly convex functions. We propose an efficient stochastic block
coordinate descent method using adaptive primal-dual updates, which enables
flexible parallel optimization for large-scale problems. Our method shares the
efficiency and flexibility of block coordinate descent methods with the
simplicity of primal-dual methods and utilizing the structure of the separable
convex-concave saddle point problem. It is capable of solving a wide range of
machine learning applications, including robust principal component analysis,
Lasso, and feature selection by group Lasso, etc. Theoretically and
empirically, we demonstrate significantly better performance than
state-of-the-art methods in all these applications.Comment: Accepted by AAAI 201
CoCoA: A General Framework for Communication-Efficient Distributed Optimization
The scale of modern datasets necessitates the development of efficient
distributed optimization methods for machine learning. We present a
general-purpose framework for distributed computing environments, CoCoA, that
has an efficient communication scheme and is applicable to a wide variety of
problems in machine learning and signal processing. We extend the framework to
cover general non-strongly-convex regularizers, including L1-regularized
problems like lasso, sparse logistic regression, and elastic net
regularization, and show how earlier work can be derived as a special case. We
provide convergence guarantees for the class of convex regularized loss
minimization objectives, leveraging a novel approach in handling
non-strongly-convex regularizers and non-smooth loss functions. The resulting
framework has markedly improved performance over state-of-the-art methods, as
we illustrate with an extensive set of experiments on real distributed
datasets
Flexible Parallel Algorithms for Big Data Optimization
We propose a decomposition framework for the parallel optimization of the sum
of a differentiable function and a (block) separable nonsmooth, convex one. The
latter term is typically used to enforce structure in the solution as, for
example, in Lasso problems. Our framework is very flexible and includes both
fully parallel Jacobi schemes and Gauss-Seidel (Southwell-type) ones, as well
as virtually all possibilities in between (e.g., gradient- or Newton-type
methods) with only a subset of variables updated at each iteration. Our
theoretical convergence results improve on existing ones, and numerical results
show that the new method compares favorably to existing algorithms.Comment: submitted to IEEE ICASSP 201
Regularized Ordinal Regression and the ordinalNet R Package
Regularization techniques such as the lasso (Tibshirani 1996) and elastic net
(Zou and Hastie 2005) can be used to improve regression model coefficient
estimation and prediction accuracy, as well as to perform variable selection.
Ordinal regression models are widely used in applications where the use of
regularization could be beneficial; however, these models are not included in
many popular software packages for regularized regression. We propose a
coordinate descent algorithm to fit a broad class of ordinal regression models
with an elastic net penalty. Furthermore, we demonstrate that each model in
this class generalizes to a more flexible form, for instance to accommodate
unordered categorical data. We introduce an elastic net penalty class that
applies to both model forms. Additionally, this penalty can be used to shrink a
non-ordinal model toward its ordinal counterpart. Finally, we introduce the R
package ordinalNet, which implements the algorithm for this model class
Hybrid Random/Deterministic Parallel Algorithms for Nonconvex Big Data Optimization
We propose a decomposition framework for the parallel optimization of the sum
of a differentiable {(possibly nonconvex)} function and a nonsmooth (possibly
nonseparable), convex one. The latter term is usually employed to enforce
structure in the solution, typically sparsity. The main contribution of this
work is a novel \emph{parallel, hybrid random/deterministic} decomposition
scheme wherein, at each iteration, a subset of (block) variables is updated at
the same time by minimizing local convex approximations of the original
nonconvex function. To tackle with huge-scale problems, the (block) variables
to be updated are chosen according to a \emph{mixed random and deterministic}
procedure, which captures the advantages of both pure deterministic and random
update-based schemes. Almost sure convergence of the proposed scheme is
established. Numerical results show that on huge-scale problems the proposed
hybrid random/deterministic algorithm outperforms both random and deterministic
schemes.Comment: The order of the authors is alphabetica
- …