1,413 research outputs found
Regularized Jacobi iteration for decentralized convex optimization with separable constraints
We consider multi-agent, convex optimization programs subject to separable
constraints, where the constraint function of each agent involves only its
local decision vector, while the decision vectors of all agents are coupled via
a common objective function. We focus on a regularized variant of the so called
Jacobi algorithm for decentralized computation in such problems. We first
consider the case where the objective function is quadratic, and provide a
fixed-point theoretic analysis showing that the algorithm converges to a
minimizer of the centralized problem. Moreover, we quantify the potential
benefits of such an iterative scheme by comparing it against a scaled projected
gradient algorithm. We then consider the general case and show that all limit
points of the proposed iteration are optimal solutions of the centralized
problem. The efficacy of the proposed algorithm is illustrated by applying it
to the problem of optimal charging of electric vehicles, where, as opposed to
earlier approaches, we show convergence to an optimal charging scheme for a
finite, possibly large, number of vehicles
Flexible Parallel Algorithms for Big Data Optimization
We propose a decomposition framework for the parallel optimization of the sum
of a differentiable function and a (block) separable nonsmooth, convex one. The
latter term is typically used to enforce structure in the solution as, for
example, in Lasso problems. Our framework is very flexible and includes both
fully parallel Jacobi schemes and Gauss-Seidel (Southwell-type) ones, as well
as virtually all possibilities in between (e.g., gradient- or Newton-type
methods) with only a subset of variables updated at each iteration. Our
theoretical convergence results improve on existing ones, and numerical results
show that the new method compares favorably to existing algorithms.Comment: submitted to IEEE ICASSP 201
Parallel Selective Algorithms for Big Data Optimization
We propose a decomposition framework for the parallel optimization of the sum
of a differentiable (possibly nonconvex) function and a (block) separable
nonsmooth, convex one. The latter term is usually employed to enforce structure
in the solution, typically sparsity. Our framework is very flexible and
includes both fully parallel Jacobi schemes and Gauss- Seidel (i.e.,
sequential) ones, as well as virtually all possibilities "in between" with only
a subset of variables updated at each iteration. Our theoretical convergence
results improve on existing ones, and numerical results on LASSO, logistic
regression, and some nonconvex quadratic problems show that the new method
consistently outperforms existing algorithms.Comment: This work is an extended version of the conference paper that has
been presented at IEEE ICASSP'14. The first and the second author contributed
equally to the paper. This revised version contains new numerical results on
non convex quadratic problem
Fixed-point and coordinate descent algorithms for regularized kernel methods
In this paper, we study two general classes of optimization algorithms for
kernel methods with convex loss function and quadratic norm regularization, and
analyze their convergence. The first approach, based on fixed-point iterations,
is simple to implement and analyze, and can be easily parallelized. The second,
based on coordinate descent, exploits the structure of additively separable
loss functions to compute solutions of line searches in closed form. Instances
of these general classes of algorithms are already incorporated into state of
the art machine learning software for large scale problems. We start from a
solution characterization of the regularized problem, obtained using
sub-differential calculus and resolvents of monotone operators, that holds for
general convex loss functions regardless of differentiability. The two
methodologies described in the paper can be regarded as instances of non-linear
Jacobi and Gauss-Seidel algorithms, and are both well-suited to solve large
scale problems
CoCoA: A General Framework for Communication-Efficient Distributed Optimization
The scale of modern datasets necessitates the development of efficient
distributed optimization methods for machine learning. We present a
general-purpose framework for distributed computing environments, CoCoA, that
has an efficient communication scheme and is applicable to a wide variety of
problems in machine learning and signal processing. We extend the framework to
cover general non-strongly-convex regularizers, including L1-regularized
problems like lasso, sparse logistic regression, and elastic net
regularization, and show how earlier work can be derived as a special case. We
provide convergence guarantees for the class of convex regularized loss
minimization objectives, leveraging a novel approach in handling
non-strongly-convex regularizers and non-smooth loss functions. The resulting
framework has markedly improved performance over state-of-the-art methods, as
we illustrate with an extensive set of experiments on real distributed
datasets
A Second Order Non-Smooth Variational Model for Restoring Manifold-Valued Images
We introduce a new non-smooth variational model for the restoration of
manifold-valued data which includes second order differences in the
regularization term. While such models were successfully applied for
real-valued images, we introduce the second order difference and the
corresponding variational models for manifold data, which up to now only
existed for cyclic data. The approach requires a combination of techniques from
numerical analysis, convex optimization and differential geometry. First, we
establish a suitable definition of absolute second order differences for
signals and images with values in a manifold. Employing this definition, we
introduce a variational denoising model based on first and second order
differences in the manifold setup. In order to minimize the corresponding
functional, we develop an algorithm using an inexact cyclic proximal point
algorithm. We propose an efficient strategy for the computation of the
corresponding proximal mappings in symmetric spaces utilizing the machinery of
Jacobi fields. For the n-sphere and the manifold of symmetric positive definite
matrices, we demonstrate the performance of our algorithm in practice. We prove
the convergence of the proposed exact and inexact variant of the cyclic
proximal point algorithm in Hadamard spaces. These results which are of
interest on its own include, e.g., the manifold of symmetric positive definite
matrices
- …