14 research outputs found
Accelerated Variance Reduced Stochastic ADMM
Recently, many variance reduced stochastic alternating direction method of
multipliers (ADMM) methods (e.g.\ SAG-ADMM, SDCA-ADMM and SVRG-ADMM) have made
exciting progress such as linear convergence rates for strongly convex
problems. However, the best known convergence rate for general convex problems
is O(1/T) as opposed to O(1/T^2) of accelerated batch algorithms, where is
the number of iterations. Thus, there still remains a gap in convergence rates
between existing stochastic ADMM and batch algorithms. To bridge this gap, we
introduce the momentum acceleration trick for batch optimization into the
stochastic variance reduced gradient based ADMM (SVRG-ADMM), which leads to an
accelerated (ASVRG-ADMM) method. Then we design two different momentum term
update rules for strongly convex and general convex cases. We prove that
ASVRG-ADMM converges linearly for strongly convex problems. Besides having a
low per-iteration complexity as existing stochastic ADMM methods, ASVRG-ADMM
improves the convergence rate on general convex problems from O(1/T) to
O(1/T^2). Our experimental results show the effectiveness of ASVRG-ADMM.Comment: 16 pages, 5 figures, Appears in Proceedings of the 31th AAAI
Conference on Artificial Intelligence (AAAI), San Francisco, California, USA,
pp. 2287--2293, 201
Scalable Peaceman-Rachford Splitting Method with Proximal Terms
Along with developing of Peaceman-Rachford Splittling Method (PRSM), many
batch algorithms based on it have been studied very deeply. But almost no
algorithm focused on the performance of stochastic version of PRSM. In this
paper, we propose a new stochastic algorithm based on PRSM, prove its
convergence rate in ergodic sense, and test its performance on both artificial
and real data. We show that our proposed algorithm, Stochastic Scalable PRSM
(SS-PRSM), enjoys the convergence rate, which is the same as those
newest stochastic algorithms that based on ADMM but faster than general
Stochastic ADMM (which is ). Our algorithm also owns wide
flexibility, outperforms many state-of-the-art stochastic algorithms coming
from ADMM, and has low memory cost in large-scale splitting optimization
problems