18,471 research outputs found
CoCoA: A General Framework for Communication-Efficient Distributed Optimization
The scale of modern datasets necessitates the development of efficient
distributed optimization methods for machine learning. We present a
general-purpose framework for distributed computing environments, CoCoA, that
has an efficient communication scheme and is applicable to a wide variety of
problems in machine learning and signal processing. We extend the framework to
cover general non-strongly-convex regularizers, including L1-regularized
problems like lasso, sparse logistic regression, and elastic net
regularization, and show how earlier work can be derived as a special case. We
provide convergence guarantees for the class of convex regularized loss
minimization objectives, leveraging a novel approach in handling
non-strongly-convex regularizers and non-smooth loss functions. The resulting
framework has markedly improved performance over state-of-the-art methods, as
we illustrate with an extensive set of experiments on real distributed
datasets
A stochastic proximal alternating method for non-smooth non-convex optimization
We introduce SPRING, a novel stochastic proximal alternating linearized
minimization algorithm for solving a class of non-smooth and non-convex
optimization problems. Large-scale imaging problems are becoming increasingly
prevalent due to advances in data acquisition and computational capabilities.
Motivated by the success of stochastic optimization methods, we propose a
stochastic variant of proximal alternating linearized minimization (PALM)
algorithm \cite{bolte2014proximal}. We provide global convergence guarantees,
demonstrating that our proposed method with variance-reduced stochastic
gradient estimators, such as SAGA \cite{SAGA} and SARAH \cite{sarah}, achieves
state-of-the-art oracle complexities. We also demonstrate the efficacy of our
algorithm via several numerical examples including sparse non-negative matrix
factorization, sparse principal component analysis, and blind image
deconvolution.Comment: 28 pages, 11 page appendi
Convergence Analysis and Improvements for Projection Algorithms and Splitting Methods
Non-smooth convex optimization problems occur in all fields of engineering. A common approach to solving this class of problems is proximal algorithms, or splitting methods. These first-order optimization algorithms are often simple, well suited to solve large-scale problems and have a low computational cost per iteration. Essentially, they encode the solution to an optimization problem as a fixed point of some operator, and iterating this operator eventually results in convergence to an optimal point. However, as for other first order methods, the convergence rate is heavily dependent on the conditioning of the problem. Even though the per-iteration cost is usually low, the number of iterations can become prohibitively large for ill-conditioned problems, especially if a high accuracy solution is sought.In this thesis, a few methods for alleviating this slow convergence are studied, which can be divided into two main approaches. The first are heuristic methods that can be applied to a range of fixed-point algorithms. They are based on understanding typical behavior of these algorithms. While these methods are shown to converge, they come with no guarantees on improved convergence rates.The other approach studies the theoretical rates of a class of projection methods that are used to solve convex feasibility problems. These are problems where the goal is to find a point in the intersection of two, or possibly more, convex sets. A study of how the parameters in the algorithm affect the theoretical convergence rate is presented, as well as how they can be chosen to optimize this rate
- âŠ