Dual Descent ALM and ADMM

Classical primal-dual algorithms attempt to solve $\max_{\mu}\min_{x} \mathcal{L}(x,\mu)$ by alternatively minimizing over the primal variable $x$ through primal descent and maximizing the dual variable $\mu$ through dual ascent. However, when $\mathcal{L}(x,\mu)$ is highly nonconvex with complex constraints in $x$, the minimization over $x$ may not achieve global optimality, and hence the dual ascent step loses its valid intuition. This observation motivates us to propose a new class of primal-dual algorithms for nonconvex constrained optimization with the key feature to reverse dual ascent to a conceptually new dual descent, in a sense, elevating the dual variable to the same status as the primal variable. Surprisingly, this new dual scheme achieves some best iteration complexities for solving nonconvex optimization problems. In particular, when the dual descent step is scaled by a fractional constant, we name it scaled dual descent (SDD), otherwise, unscaled dual descent (UDD). For nonconvex multiblock optimization with nonlinear equality constraints, we propose SDD-ADMM and show that it finds an $\epsilon$-stationary solution in $\mathcal{O}(\epsilon^{-4})$ iterations. The complexity is further improved to $\mathcal{O}(\epsilon^{-3})$ and $\mathcal{O}(\epsilon^{-2})$ under proper conditions. We also propose UDD-ALM, combining UDD with ALM, for weakly convex minimization over affine constraints. We show that UDD-ALM finds an $\epsilon$-stationary solution in $\mathcal{O}(\epsilon^{-2})$ iterations. These complexity bounds for both algorithms either achieve or improve the best-known results in the ADMM and ALM literature. Moreover, SDD-ADMM addresses a long-standing limitation of existing ADMM frameworks

Algorithms for Difference-of-Convex (DC) Programs Based on Difference-of-Moreau-Envelopes Smoothing

In this paper we consider minimization of a difference-of-convex (DC) function with and without linear constraints. We first study a smooth approximation of a generic DC function, termed difference-of-Moreau-envelopes (DME) smoothing, where both components of the DC function are replaced by their respective Moreau envelopes. The resulting smooth approximation is shown to be Lipschitz differentiable, capture stationary points, local, and global minima of the original DC function, and enjoy some growth conditions, such as level-boundedness and coercivity, for broad classes of DC functions. We then develop four algorithms for solving DC programs with and without linear constraints based on the DME smoothing. In particular, for a smoothed DC program without linear constraints, we show that the classic gradient descent method as well as an inexact variant can obtain a stationary solution in the limit with a convergence rate of $\mathcal{O}(K^{-1/2})$, where $K$ is the number of proximal evaluations of both components. Furthermore, when the DC program is explicitly constrained in an affine subspace, we combine the smoothing technique with the augmented Lagrangian function and derive two variants of the augmented Lagrangian method (ALM), named LCDC-ALM and composite LCDC-ALM, focusing on different structures of the DC objective function. We show that both algorithms find an $\epsilon$-approximate stationary solution of the original DC program in $\mathcal{O}(\epsilon^{-2})$ iterations. Comparing to existing methods designed for linearly constrained weakly convex minimization, the proposed ALM-based algorithms can be applied to a broader class of problems, where the objective contains a nonsmooth concave component. Finally, numerical experiments are presented to demonstrate the performance of the proposed algorithms

Decomposition Methods for Global Solutions of Mixed-Integer Linear Programs

This paper introduces two decomposition-based methods for two-block mixed-integer linear programs (MILPs), which break the original problem into a sequence of smaller MILP subproblems. The first method is based on the l1-augmented Lagrangian. The second method is based on the alternating direction method of multipliers. When the original problem has a block-angular structure, the subproblems of the first block have low dimensions and can be solved in parallel. We add reverse-norm cuts and augmented Lagrangian cuts to the subproblems of the second block. For both methods, we show asymptotic convergence to globally optimal solutions and present iteration upper bounds. Numerical comparisons with recent decomposition methods demonstrate the exactness and efficiency of our proposed methods

An ADMM-based Distributed Optimization Method for Solving Security-Constrained AC Optimal Power Flow

In this paper, we study efficient and robust computational methods for solving the security-constrained alternating current optimal power flow (SC-ACOPF) problem, a two-stage nonlinear optimization problem with disjunctive constraints, that is central to the operation of electric power grids. The first-stage problem in SC-ACOPF determines the operation of the power grid in normal condition, while the second-stage problem responds to various contingencies of losing generators, transmission lines, and transformers. The two stages are coupled through disjunctive constraints, which model generators' active and reactive power output changes responding to system-wide active power imbalance and voltage deviations after contingencies. Real-world SC-ACOPF problems may involve power grids with more than 30k buses and 22k contingencies and need to be solved within 10-45 minutes to get a base case solution with high feasibility and reasonably good generation cost. We develop a comprehensive algorithmic framework to solve SC-ACOPF that meets the challenge of speed, solution quality, and computation robustness. In particular, we develop a smoothing technique to approximate disjunctive constraints into a smooth structure which can be handled by interior-point solvers; we design a distributed optimization algorithm to efficiently generate first-stage solutions; we propose a screening procedure to prioritize contingencies; and finally, we develop a reliable and parallel architecture that integrates all algorithmic components. Extensive tests on industry-scale systems demonstrate the superior performance of the proposed algorithms

Decomposition algorithms based on the nonconvex augmented Lagrangian framework

Many problems of recent interest arising from engineering and data sciences go beyond the framework of convex optimization and inevitably need to deal with nonconvex structures. The design of efficient optimization algorithms when non-convexity is present, either in constraints or in the objective, has always been a difficult task. Meanwhile, the sheer amount of data that need to be considered in applications nowadays bring in new computational challenges. In this thesis, we develop new convergence-guaranteed decomposition methods for several important classes of nonconvex problems. The proposed algorithms take advantage of decomposable structures embedded in the original problem. The first part of this thesis is concerned with decomposable structures in constraints. We are motivated by the observation that, for a broad class of nonlinear constrained optimization problems defined over a network, existing distributed algorithms based on the popular alternating direction method of multipliers (ADMM) fail to converge due to certain formulation limitations. To overcome the difficulty, we propose a two-level framework, where the inner level utilizes a structured three-block ADMM to facilitate parallel computations and the outer level applies the classic augmented Lagrange method (ALM) to ensure convergence. We establish global convergence with iteration complexity estimates as well as local convergence results for this new scheme and demonstrate its performance on various nonconvex applications. In particular, we adopt the two-level framework to solve the nonconvex AC optimal power flow (OPF) problem, which is a basic building block in electric power systems. We present extensive numerical experiments on some largest open-sourced power networks (up to 30,000 buses) to demonstrate the speed, robustness, and scalability of the proposed algorithm. The powerful algorithmic frameworks of ALM and ADMM inspire us to further exploit their potentials for solving nonconvex optimization problems. We investigate a new class of dual updates termed scaled dual descent (SDD) within the augmented Lagrangian framework. We propose SDD-ADMM, which combines SDD with ADMM, to solve nonlinear equality-constrained multi-block problems. SDD-ADMM improves the previous state-of-the-art works of ALM and ADMM in several nontrivial perspectives, including new treatment of nonlinear constraints, less restrictions on problem data, and better iteration complexity results. Moreover, SDD-ADMM admits flexible Gauss-Seidel and Jacobi updates on blocks of variables, making the method particularly suitable for distributed computation. The second part of this thesis investigates a specific decomposable structure in the objective, namely, when the objective is a difference-of-convex (DC) function, i.e., a function $F = \phi-g$ where both $\phi$ and $g$ are convex functions. We first study a smoothing technique for a generic DC function $F$, where we replace both $\phi$ and $g$ by their respective Moreau envelopes. The resulting smooth approximation, termed difference of Moreau Envelopes (DME), is shown to be Lipschitz differentiable, capture stationary, local, and global solutions of the original DC function $F$, and enjoy some growth conditions for broad classes of DC functions. In addition, the DME smoothing provides a powerful tool for algorithmic development: first-order updates on the DME deliver a stationary solution of $F$, and we can further combine the DME smoothing with ALM to solve constrained DC programs. An interesting feature that distinguishes DME-based algorithms from existing DC algorithms (DCA) is that we invoke proximal oracles on the negative component $g$ instead of its subgradient oracles, and consequently, the updates on $\phi$ and $g$ can be implemented in parallel.Ph.D

A two-level distributed algorithm for nonconvex constrained optimization

Abstract This paper aims to develop distributed algorithms for nonconvex optimization problems with complicated constraints associated with a network. The network can be a physical one, such as an electric power network, where the constraints are nonlinear power flow equations, or an abstract one that represents constraint couplings between decision variables of different agents. Despite the recent development of distributed algorithms for nonconvex programs, highly complicated constraints still pose a significant challenge in theory and practice. We first identify some difficulties with the existing algorithms based on the alternating direction method of multipliers (ADMM) for dealing with such problems. We then propose a reformulation that enables us to design a two-level algorithm, which embeds a specially structured three-block ADMM at the inner level in an augmented Lagrangian method framework. Furthermore, we prove the global and local convergence as well as iteration complexity of this new scheme for general nonconvex constrained programs, and show that our analysis can be extended to handle more complicated multi-block inner-level problems. Finally, we demonstrate with computation that the new scheme provides convergent and parallelizable algorithms for various nonconvex applications, and is able to complement the performance of the state-of-the-art distributed algorithms in practice by achieving either faster convergence in optimality gap or in feasibility or both