We study a stochastic and distributed algorithm for nonconvex problems whose
objective consists of a sum of N nonconvex Li​/N-smooth functions, plus a
nonsmooth regularizer. The proposed NonconvEx primal-dual SpliTTing (NESTT)
algorithm splits the problem into N subproblems, and utilizes an augmented
Lagrangian based primal-dual scheme to solve it in a distributed and stochastic
manner. With a special non-uniform sampling, a version of NESTT achieves
ϵ-stationary solution using
O((∑i=1N​Li​/N​)2/ϵ) gradient evaluations,
which can be up to O(N) times better than the (proximal) gradient
descent methods. It also achieves Q-linear convergence rate for nonconvex
ℓ1​ penalized quadratic problems with polyhedral constraints. Further, we
reveal a fundamental connection between primal-dual based methods and a few
primal only methods such as IAG/SAG/SAGA.Comment: 35 pages, 2 figure