51 research outputs found
GMRES-Accelerated ADMM for Quadratic Objectives
We consider the sequence acceleration problem for the alternating direction
method-of-multipliers (ADMM) applied to a class of equality-constrained
problems with strongly convex quadratic objectives, which frequently arise as
the Newton subproblem of interior-point methods. Within this context, the ADMM
update equations are linear, the iterates are confined within a Krylov
subspace, and the General Minimum RESidual (GMRES) algorithm is optimal in its
ability to accelerate convergence. The basic ADMM method solves a
-conditioned problem in iterations. We give
theoretical justification and numerical evidence that the GMRES-accelerated
variant consistently solves the same problem in iterations
for an order-of-magnitude reduction in iterations, despite a worst-case bound
of iterations. The method is shown to be competitive against
standard preconditioned Krylov subspace methods for saddle-point problems. The
method is embedded within SeDuMi, a popular open-source solver for conic
optimization written in MATLAB, and used to solve many large-scale semidefinite
programs with error that decreases like , instead of ,
where is the iteration index.Comment: 31 pages, 7 figures. Accepted for publication in SIAM Journal on
Optimization (SIOPT
Conic Optimization Theory: Convexification Techniques and Numerical Algorithms
Optimization is at the core of control theory and appears in several areas of
this field, such as optimal control, distributed control, system
identification, robust control, state estimation, model predictive control and
dynamic programming. The recent advances in various topics of modern
optimization have also been revamping the area of machine learning. Motivated
by the crucial role of optimization theory in the design, analysis, control and
operation of real-world systems, this tutorial paper offers a detailed overview
of some major advances in this area, namely conic optimization and its emerging
applications. First, we discuss the importance of conic optimization in
different areas. Then, we explain seminal results on the design of hierarchies
of convex relaxations for a wide range of nonconvex problems. Finally, we study
different numerical algorithms for large-scale conic optimization problems.Comment: 18 page
An Interior-Point-Inspired algorithm for Linear Programs arising in Discrete Optimal Transport
Discrete Optimal Transport problems give rise to very large linear programs
(LP) with a particular structure of the constraint matrix. In this paper we
present a hybrid algorithm that mixes an interior point method (IPM) and column
generation, specialized for the LP originating from the Kantorovich Optimal
Transport problem. Knowing that optimal solutions of such problems display a
high degree of sparsity, we propose a column-generation-like technique to force
all intermediate iterates to be as sparse as possible. The algorithm is
implemented nearly matrix-free. Indeed, most of the computations avoid forming
the huge matrices involved and solve the Newton system using only a much
smaller Schur complement of the normal equations. We prove theoretical results
about the sparsity pattern of the optimal solution, exploiting the graph
structure of the underlying problem. We use these results to mix iterative and
direct linear solvers efficiently, in a way that avoids producing
preconditioners or factorizations with excessive fill-in and at the same time
guaranteeing a low number of conjugate gradient iterations. We compare the
proposed method with two state-of-the-art solvers and show that it can compete
with the best network optimization tools in terms of computational time and
memory usage. We perform experiments with problems reaching more than four
billion variables and demonstrate the robustness of the proposed method
Efficient interior point algorithms for large scale convex optimization problems
Interior point methods (IPMs) are among the most widely used algorithms for
convex optimization problems. They are applicable to a wide range of problems, including
linear, quadratic, nonlinear, conic and semidefinite programming problems,
requiring a polynomial number of iterations to find an accurate approximation of
the primal-dual solution. The formidable convergence properties of IPMs come
with a fundamental drawback: the numerical linear algebra involved becomes
progressively more and more challenging as the IPM converges towards optimality.
In particular, solving the linear systems to find the Newton directions requires
most of the computational effort of an IPM. Proposed remedies to alleviate
this phenomenon include regularization techniques, predictor-corrector schemes,
purposely developed preconditioners, low-rank update strategies, to mention a
few.
For problems of very large scale, this unpleasant characteristic of IPMs becomes
a more and more problematic feature, since any technique used must be efficient
and scalable in order to maintain acceptable computational requirements. In this
Thesis, we deal with convex linear and quadratic problems of large âdimensionâ:
we use this term in a broader sense than just a synonym for âsizeâ of the problem.
The instances considered can be either problems with a large number of variables
and/or constraints but with a sparse structure, or problems with a moderate
number of variables and/or constraints but with a dense structure. Both these
type of problems require very efficient strategies to be used during the algorithm,
even though the corresponding difficulties arise for different reasons.
The first application that we consider deals with a moderate size quadratic
problem where the quadratic term is 100% dense; this problem arises from X-ray
tomographic imaging reconstruction, in particular with the goal of separating the
distribution of two materials present in the observed sample. A novel non-convex
regularizer is introduced for this purpose; convexity of the overall problem is
maintained by careful choice of the parameters. We derive a specialized interior
point method for this problem and an appropriate preconditioner for the normal
equations linear system, to be used without ever forming the fully dense matrices
involved.
The next major contribution is related to the issue of efficiently computing
the Newton direction during IPMs. When an iterative method is applied to
solve the linear equation system in IPMs, the attention is usually placed on
accelerating their convergence by designing appropriate preconditioners, but the
linear solver is applied as a black box with a standard termination criterion
which asks for a sufficient reduction of the residual in the linear system. Such an
approach often leads to an unnecessary âover-solvingâ of linear equations. We
propose new indicators for the early termination of the inner iterations and test
them on a set of large scale quadratic optimization problems. Evidence gathered
from these computational experiments shows that the new technique delivers
significant improvements in terms of inner (linear) iterations and those translate
into significant savings of the IPM solution time.
The last application considered is discrete optimal transport (OT) problems;
these kind of problems give rise to very large linear programs with highly structured
matrices. Solutions of such problems are expected to be sparse, that is only a
small subset of entries in the optimal solution is expected to be nonzero. We derive
an IPM for the standard OT formulation, which exploits a column-generation-like
technique to force all intermediate iterates to be as sparse as possible. We prove
theoretical results about the sparsity pattern of the optimal solution and we
propose to mix iterative and direct linear solvers in an efficient way, to keep
computational time and memory requirement as low as possible. We compare the
proposed method with two state-of-the-art solvers and show that it can compete
with the best network optimization tools in terms of computational time and
memory usage. We perform experiments with problems reaching more than four
billion variables and demonstrate the robustness of the proposed method.
We consider also the optimal transport problem on sparse graphs and present
a primal-dual regularized IPM to solve it. We prove that the introduction of the
regularization allows us to use sparsified versions of the normal equations system
to inexpensively generate inexact IPM directions. The proposed method is shown
to have polynomial complexity and to outperform a very efficient network simplex
implementation, for problems with up to 50 million variables
An inexact dual logarithmic barrier method for solving sparse semidefinite programs
A dual logarithmic barrier method for solving large, sparse semidefinite programs is proposed in this paper. The method avoids any explicit use of the primal variable X and therefore is well-suited to problems with a sparse dual matrix S. It relies on inexact Newton steps in dual space which are computed by the conjugate gradient method applied to the Schur complement of the reduced KKT system. The method may take advantage of low-rank representations of matrices Ai to perform implicit matrix-vector products with the Schur complement matrix and to compute only specific parts of this matrix. This allows the construction of the partial Cholesky factorization of the Schur complement matrix which serves as a good preconditioner for it and permits the method to be run in a matrix-free scheme. Convergence properties of the method are studied and a polynomial complexity result is extended to the case when inexact Newton steps are employed. A Matlab-based implementation is developed and preliminary computational results of applying the method to maximum cut and matrix completion problems are reported
- âŠ