165 research outputs found
Douglas-Rachford Splitting: Complexity Estimates and Accelerated Variants
We propose a new approach for analyzing convergence of the Douglas-Rachford
splitting method for solving convex composite optimization problems. The
approach is based on a continuously differentiable function, the
Douglas-Rachford Envelope (DRE), whose stationary points correspond to the
solutions of the original (possibly nonsmooth) problem. By proving the
equivalence between the Douglas-Rachford splitting method and a scaled gradient
method applied to the DRE, results from smooth unconstrained optimization are
employed to analyze convergence properties of DRS, to tune the method and to
derive an accelerated version of it
Distributed Solution of Large-Scale Linear Systems via Accelerated Projection-Based Consensus
Solving a large-scale system of linear equations is a key step at the heart
of many algorithms in machine learning, scientific computing, and beyond. When
the problem dimension is large, computational and/or memory constraints make it
desirable, or even necessary, to perform the task in a distributed fashion. In
this paper, we consider a common scenario in which a taskmaster intends to
solve a large-scale system of linear equations by distributing subsets of the
equations among a number of computing machines/cores. We propose an accelerated
distributed consensus algorithm, in which at each iteration every machine
updates its solution by adding a scaled version of the projection of an error
signal onto the nullspace of its system of equations, and where the taskmaster
conducts an averaging over the solutions with momentum. The convergence
behavior of the proposed algorithm is analyzed in detail and analytically shown
to compare favorably with the convergence rate of alternative distributed
methods, namely distributed gradient descent, distributed versions of
Nesterov's accelerated gradient descent and heavy-ball method, the block
Cimmino method, and ADMM. On randomly chosen linear systems, as well as on
real-world data sets, the proposed method offers significant speed-up relative
to all the aforementioned methods. Finally, our analysis suggests a novel
variation of the distributed heavy-ball method, which employs a particular
distributed preconditioning, and which achieves the same theoretical
convergence rate as the proposed consensus-based method
Nonconvex Sparse Spectral Clustering by Alternating Direction Method of Multipliers and Its Convergence Analysis
Spectral Clustering (SC) is a widely used data clustering method which first
learns a low-dimensional embedding of data by computing the eigenvectors of
the normalized Laplacian matrix, and then performs k-means on to get
the final clustering result. The Sparse Spectral Clustering (SSC) method
extends SC with a sparse regularization on by using the block
diagonal structure prior of in the ideal case. However, encouraging
to be sparse leads to a heavily nonconvex problem which is
challenging to solve and the work (Lu, Yan, and Lin 2016) proposes a convex
relaxation in the pursuit of this aim indirectly. However, the convex
relaxation generally leads to a loose approximation and the quality of the
solution is not clear. This work instead considers to solve the nonconvex
formulation of SSC which directly encourages to be sparse. We propose
an efficient Alternating Direction Method of Multipliers (ADMM) to solve the
nonconvex SSC and provide the convergence guarantee. In particular, we prove
that the sequences generated by ADMM always exist a limit point and any limit
point is a stationary point. Our analysis does not impose any assumptions on
the iterates and thus is practical. Our proposed ADMM for nonconvex problems
allows the stepsize to be increasing but upper bounded, and this makes it very
efficient in practice. Experimental analysis on several real data sets verifies
the effectiveness of our method.Comment: Proceedings of the AAAI Conference on Artificial Intelligence (AAAI).
201
- …