28 research outputs found
On the convergence rate of the scaled proximal decomposition on the graph of a maximal monotone operator (SPDG) algorithm
Relying on fixed point techniques, Mahey, Oualibouch and Tao introduced the
scaled proximal decomposition on the graph of a maximal monotone operator
(SPDG) algorithm and analyzed its performance on inclusions for strongly
monotone and Lipschitz continuous operators. The SPDG algorithm generalizes the
Spingarn's partial inverse method by allowing scaling factors, a key strategy
to speed up the convergence of numerical algorithms. In this note, we show that
the SPDG algorithm can alternatively be analyzed by means of the original
Spingarn's partial inverse framework, tracing back to the 1983 Spingarn's
paper. We simply show that under the assumptions considered by Mahey,
Oualibouch and Tao, the Spingarn's partial inverse of the underlying maximal
monotone operator is strongly monotone, which allows one to employ recent
results on the convergence and iteration-complexity of proximal point type
methods for strongly monotone operators. By doing this, we additionally obtain
a potentially faster convergence for the SPDG algorithm and a more accurate
upper bound on the number of iterations needed to achieve prescribed
tolerances, specially on ill-conditioned problems
An efficient adaptive accelerated inexact proximal point method for solving linearly constrained nonconvex composite problems
This paper proposes an efficient adaptive variant of a quadratic penalty
accelerated inexact proximal point (QP-AIPP) method proposed earlier by the
authors. Both the QP-AIPP method and its variant solve linearly set constrained
nonconvex composite optimization problems using a quadratic penalty approach
where the generated penalized subproblems are solved by a variant of the
underlying AIPP method. The variant, in turn, solves a given penalized
subproblem by generating a sequence of proximal subproblems which are then
solved by an accelerated composite gradient algorithm. The main difference
between AIPP and its variant is that the proximal subproblems in the former are
always convex while the ones in the latter are not necessarily convex due to
the fact that their prox parameters are chosen as aggressively as possible so
as to improve efficiency. The possibly nonconvex proximal subproblems generated
by the AIPP variant are also tentatively solved by a novel adaptive accelerated
composite gradient algorithm based on the validity of some key convergence
inequalities. As a result, the variant generates a sequence of proximal
subproblems where the stepsizes are adaptively changed according to the
responses obtained from the calls to the accelerated composite gradient
algorithm. Finally, numerical results are given to demonstrate the efficiency
of the proposed AIPP and QP-AIPP variants
A FISTA-type accelerated gradient algorithm for solving smooth nonconvex composite optimization problems
In this paper, we describe and establish iteration-complexity of two
accelerated composite gradient (ACG) variants to solve a smooth nonconvex
composite optimization problem whose objective function is the sum of a
nonconvex differentiable function with a Lipschitz continuous gradient
and a simple nonsmooth closed convex function . When is convex, the
first ACG variant reduces to the well-known FISTA for a specific choice of the
input, and hence the first one can be viewed as a natural extension of the
latter one to the nonconvex setting. The first variant requires an input pair
such that is -weakly convex, is -Lipschitz
continuous, and (possibly ), which is usually hard to obtain or
poorly estimated. The second variant on the other hand can start from an
arbitrary input pair of positive scalars and its complexity is shown to
be not worse, and better in some cases, than that of the first variant for a
large range of the input pairs. Finally, numerical results are provided to
illustrate the efficiency of the two ACG variants.Comment: 28 page
Accelerated first-order primal-dual proximal methods for linearly constrained composite convex programming
Motivated by big data applications, first-order methods have been extremely
popular in recent years. However, naive gradient methods generally converge
slowly. Hence, much efforts have been made to accelerate various first-order
methods. This paper proposes two accelerated methods towards solving structured
linearly constrained convex programming, for which we assume composite convex
objective.
The first method is the accelerated linearized augmented Lagrangian method
(LALM). At each update to the primal variable, it allows linearization to the
differentiable function and also the augmented term, and thus it enables easy
subproblems. Assuming merely weak convexity, we show that LALM owns
convergence if parameters are kept fixed during all the iterations and can be
accelerated to if the parameters are adapted, where is the
number of total iterations.
The second method is the accelerated linearized alternating direction method
of multipliers (LADMM). In addition to the composite convexity, it further
assumes two-block structure on the objective. Different from classic ADMM, our
method allows linearization to the objective and also augmented term to make
the update simple. Assuming strong convexity on one block variable, we show
that LADMM also enjoys convergence with adaptive parameters. This
result is a significant improvement over that in [Goldstein et. al, SIIMS'14],
which requires strong convexity on both block variables and no linearization to
the objective or augmented term.
Numerical experiments are performed on quadratic programming, image
denoising, and support vector machine. The proposed accelerated methods are
compared to nonaccelerated ones and also existing accelerated methods. The
results demonstrate the validness of acceleration and superior performance of
the proposed methods over existing ones
On inexact relative-error hybrid proximal extragradient, forward-backward and Tseng's modified forward-backward methods with inertial effects
In this paper, we propose and study the asymptotic convergence and
nonasymptotic global convergence rates (iteration-complexity) of an inertial
under-relaxed version of the relative-error hybrid proximal extragradient (HPE)
method for solving monotone inclusion problems. We analyze the proposed method
under more flexible assumptions than existing ones on the extrapolation and
relative-error parameters. As applications, we propose and/or study inertial
under-relaxed forward-backward and Tseng's modified forward-backward type
methods for solving structured monotone inclusions
Iteration complexity of an inexact Douglas-Rachford method and of a Douglas-Rachford-Tseng's F-B four-operator splitting method for solving monotone inclusions
In this paper, we propose and study the iteration complexity of an inexact
Douglas-Rachford splitting (DRS) method and a Douglas-Rachford-Tseng's
forward-backward (F-B) splitting method for solving two-operator and
four-operator monotone inclusions, respectively. The former method (although
based on a slightly different mechanism of iteration) is motivated by the
recent work of J. Eckstein and W. Yao, in which an inexact DRS method is
derived from a special instance of the hybrid proximal extragradient (HPE)
method of Solodov and Svaiter, while the latter one combines the proposed
inexact DRS method (used as an outer iteration) with a Tseng's F-B splitting
type method (used as an inner iteration) for solving the corresponding
subproblems. We prove iteration complexity bounds for both algorithms in the
pointwise (non-ergodic) as well as in the ergodic sense by showing that they
admit two different iterations: one that can be embedded into the HPE method,
for which the iteration complexity is known since the work of Monteiro and
Svaiter, and another one which demands a separate analysis. Finally, we perform
simple numerical experiments %on three-operator and four-operator monotone
inclusions to show the performance of the proposed methods when compared with
other existing algorithms
Fast Saddle-Point Algorithm for Generalized Dantzig Selector and FDR Control with the Ordered l1-Norm
In this paper we propose a primal-dual proximal extragradient algorithm to
solve the generalized Dantzig selector (GDS) estimation problem, based on a new
convex-concave saddle-point (SP) reformulation. Our new formulation makes it
possible to adopt recent developments in saddle-point optimization, to achieve
the optimal rate of convergence. Compared to the optimal non-SP
algorithms, ours do not require specification of sensitive parameters that
affect algorithm performance or solution quality. We also provide a new
analysis showing a possibility of local acceleration to achieve the rate of
in special cases even without strong convexity or strong smoothness.
As an application, we propose a GDS equipped with the ordered -norm,
showing its false discovery rate control properties in variable selection.
Algorithm performance is compared between ours and other alternatives,
including the linearized ADMM, Nesterov's smoothing, Nemirovski's mirror-prox,
and the accelerated hybrid proximal extragradient techniques.Comment: In AISTATS 201
DIPPA: An improved Method for Bilinear Saddle Point Problems
This paper studies bilinear saddle point problems , where the
functions are smooth and strongly-convex. When the gradient and proximal
oracle related to and are accessible, optimal algorithms have already
been developed in the literature \cite{chambolle2011first,
palaniappan2016stochastic}. However, the proximal operator is not always easy
to compute, especially in constraint zero-sum matrix games
\cite{zhang2020sparsified}. This work proposes a new algorithm which only
requires the access to the gradients of . Our algorithm achieves a
complexity upper bound which has optimal dependency on the coupling condition
number up to logarithmic factors
A Stochastic Variance-reduced Accelerated Primal-dual Method for Finite-sum Saddle-point Problems
In this paper, we propose a variance-reduced primal-dual algorithm with
Bregman distance for solving convex-concave saddle-point problems with
finite-sum structure and nonbilinear coupling function. This type of problems
typically arises in machine learning and game theory. Based on some standard
assumptions, the algorithm is proved to converge with oracle complexity of
and using constant and non-constant
parameters, respectively where is the number of function components.
Compared with existing methods, our framework yields a significant improvement
over the number of required primal-dual gradient samples to achieve
-accuracy of the primal-dual gap. We tested our method for solving a
distributionally robust optimization problem to show the effectiveness of the
algorithm
An accelerated inexact proximal point method for solving nonconvex-concave min-max problems
This paper presents smoothing schemes for obtaining approximate stationary
points of unconstrained or linearly-constrained composite nonconvex-concave
min-max (and hence nonsmooth) problems by applying well-known algorithms to
composite smooth approximations of the original problems. More specifically, in
the unconstrained (resp. constrained) case, approximate stationary points of
the original problem are obtained by applying, to its composite smooth
approximation, an accelerated inexact proximal point (resp. quadratic penalty)
method presented in a previous paper by the authors. Iteration complexity
bounds for both smoothing schemes are also established. Finally, numerical
results are given to demonstrate the efficiency of the unconstrained smoothing
scheme