18,380 research outputs found
Exponential Convergence Bounds using Integral Quadratic Constraints
The theory of integral quadratic constraints (IQCs) allows verification of
stability and gain-bound properties of systems containing nonlinear or
uncertain elements. Gain bounds often imply exponential stability, but it can
be challenging to compute useful numerical bounds on the exponential decay
rate. In this work, we present a modification of the classical IQC results of
Megretski and Rantzer that leads to a tractable computational procedure for
finding exponential rate certificates
Semi-definite programming and functional inequalities for Distributed Parameter Systems
We study one-dimensional integral inequalities, with quadratic integrands, on
bounded domains. Conditions for these inequalities to hold are formulated in
terms of function matrix inequalities which must hold in the domain of
integration. For the case of polynomial function matrices, sufficient
conditions for positivity of the matrix inequality and, therefore, for the
integral inequalities are cast as semi-definite programs. The inequalities are
used to study stability of linear partial differential equations.Comment: 8 pages, 5 figure
On the exponential convergence of the Kaczmarz algorithm
The Kaczmarz algorithm (KA) is a popular method for solving a system of
linear equations. In this note we derive a new exponential convergence result
for the KA. The key allowing us to establish the new result is to rewrite the
KA in such a way that its solution path can be interpreted as the output from a
particular dynamical system. The asymptotic stability results of the
corresponding dynamical system can then be leveraged to prove exponential
convergence of the KA. The new bound is also compared to existing bounds
A Unified Analysis of Stochastic Optimization Methods Using Jump System Theory and Quadratic Constraints
We develop a simple routine unifying the analysis of several important
recently-developed stochastic optimization methods including SAGA, Finito, and
stochastic dual coordinate ascent (SDCA). First, we show an intrinsic
connection between stochastic optimization methods and dynamic jump systems,
and propose a general jump system model for stochastic optimization methods.
Our proposed model recovers SAGA, SDCA, Finito, and SAG as special cases. Then
we combine jump system theory with several simple quadratic inequalities to
derive sufficient conditions for convergence rate certifications of the
proposed jump system model under various assumptions (with or without
individual convexity, etc). The derived conditions are linear matrix
inequalities (LMIs) whose sizes roughly scale with the size of the training
set. We make use of the symmetry in the stochastic optimization methods and
reduce these LMIs to some equivalent small LMIs whose sizes are at most 3 by 3.
We solve these small LMIs to provide analytical proofs of new convergence rates
for SAGA, Finito and SDCA (with or without individual convexity). We also
explain why our proposed LMI fails in analyzing SAG. We reveal a key difference
between SAG and other methods, and briefly discuss how to extend our LMI
analysis for SAG. An advantage of our approach is that the proposed analysis
can be automated for a large class of stochastic methods under various
assumptions (with or without individual convexity, etc).Comment: To Appear in Proceedings of the Annual Conference on Learning Theory
(COLT) 201
Design of First-Order Optimization Algorithms via Sum-of-Squares Programming
In this paper, we propose a framework based on sum-of-squares programming to
design iterative first-order optimization algorithms for smooth and strongly
convex problems. Our starting point is to develop a polynomial matrix
inequality as a sufficient condition for exponential convergence of the
algorithm. The entries of this matrix are polynomial functions of the unknown
parameters (exponential decay rate, stepsize, momentum coefficient, etc.). We
then formulate a polynomial optimization, in which the objective is to optimize
the exponential decay rate over the parameters of the algorithm. Finally, we
use sum-of-squares programming as a tractable relaxation of the proposed
polynomial optimization problem. We illustrate the utility of the proposed
framework by designing a first-order algorithm that shares the same structure
as Nesterov's accelerated gradient method
- …