19 research outputs found
On the Iteration Complexity of Smoothed Proximal ALM for Nonconvex Optimization Problem with Convex Constraints
It is well-known that the lower bound of iteration complexity for solving
nonconvex unconstrained optimization problems is , which
can be achieved by standard gradient descent algorithm when the objective
function is smooth. This lower bound still holds for nonconvex constrained
problems, while it is still unknown whether a first-order method can achieve
this lower bound. In this paper, we show that a simple single-loop first-order
algorithm called smoothed proximal augmented Lagrangian method (ALM) can
achieve such iteration complexity lower bound. The key technical contribution
is a strong local error bound for a general convex constrained problem, which
is of independent interest
A Superlinear Convergence Framework for Kurdyka-{\L}ojasiewicz Optimization
This work extends the iterative framework proposed by Attouch et al. (in
Math. Program. 137: 91-129, 2013) for minimizing a nonconvex and nonsmooth
function so that the generated sequence possesses a Q-superlinear
convergence rate. This framework consists of a monotone decrease condition, a
relative error condition and a continuity condition, and the first two
conditions both involve a parameter . We justify that any sequence
conforming to this framework is globally convergent when is a
Kurdyka-{\L}ojasiewicz (KL) function, and the convergence has a Q-superlinear
rate of order when is a KL function of exponent
. Then, we illustrate that the iterate sequence
generated by an inexact -order regularization method for composite
optimization problems with a nonconvex and nonsmooth term belongs to this
framework, and consequently, first achieve the Q-superlinear convergence rate
of order for an inexact cubic regularization method to solve this class
of composite problems with KL property of exponent
Variance Reduced Random Relaxed Projection Method for Constrained Finite-sum Minimization Problems
For many applications in signal processing and machine learning, we are
tasked with minimizing a large sum of convex functions subject to a large
number of convex constraints. In this paper, we devise a new random projection
method (RPM) to efficiently solve this problem. Compared with existing RPMs,
our proposed algorithm features two useful algorithmic ideas. First, at each
iteration, instead of projecting onto the subset defined by one of the
constraints, our algorithm only requires projecting onto a half-space
approximation of the subset, which significantly reduces the computational cost
as it admits a closed-form formula. Second, to exploit the structure that the
objective is a sum, variance reduction is incorporated into our algorithm to
further improve the performance. As theoretical contributions, under an error
bound condition and other standard assumptions, we prove that the proposed RPM
converges to an optimal solution and that both optimality and feasibility gaps
vanish at a sublinear rate. We also provide sufficient conditions for the error
bound condition to hold. Experiments on a beamforming problem and a robust
classification problem are also presented to demonstrate the superiority of our
RPM over existing ones
On the local convergence of the semismooth Newton method for composite optimization
Existing superlinear convergence rate of the semismooth Newton method relies
on the nonsingularity of the B-Jacobian. This is a strict condition since it
implies that the stationary point to seek is isolated. In this paper, we
consider a large class of nonlinear equations derived from first-order type
methods for solving composite optimization problems. We first present some
equivalent characterizations of the invertibility of the associated B-Jacobian,
providing easy-to-check criteria for the traditional condition. Secondly, we
prove that the strict complementarity and local error bound condition guarantee
a local superlinear convergence rate. The analysis consists of two steps:
showing local smoothness based on partial smoothness or closedness of the set
of nondifferentiable points of the proximal map, and applying the local error
bound condition to the locally smooth nonlinear equations. Concrete examples
satisfying the required assumptions are presented. The main novelty of the
proposed condition is that it also applies to nonisolated stationary points.Comment: 25 page
An inexact regularized proximal Newton method for nonconvex and nonsmooth optimization
This paper focuses on the minimization of a sum of a twice continuously
differentiable function and a nonsmooth convex function. We propose an
inexact regularized proximal Newton method by an approximation of the Hessian
involving the th power of the KKT residual. For
, we demonstrate the global convergence of the iterate sequence for
the KL objective function and its -linear convergence rate for the KL
objective function of exponent . For , we establish the
global convergence of the iterate sequence and its superlinear convergence rate
of order under an assumption that cluster points satisfy a
local H\"{o}lderian local error bound of order
on the strong stationary point set;
and when cluster points satisfy a local error bound of order on
the common stationary point set, we also obtain the global convergence of the
iterate sequence, and its superlinear convergence rate of order
if . A dual
semismooth Newton augmented Lagrangian method is developed for seeking an
inexact minimizer of subproblem. Numerical comparisons with two
state-of-the-art methods on -regularized Student's -regression,
group penalized Student's -regression, and nonconvex image restoration
confirm the efficiency of the proposed method