80 research outputs found
Optimality conditions, approximate stationarity, and applications 'a story beyond lipschitzness
Approximate necessary optimality conditions in terms of Frechet subgradients and normals for a rather general optimization problem with a potentially non-Lipschitzian objective function are established with the aid of Ekeland's variational principle, the fuzzy Frechet subdifferential sum rule, and a novel notion of lower semicontinuity relative to a set-valued mapping or set. Feasible points satisfying these optimality conditions are referred to as approximately stationary. As applications, we derive a new general version of the extremal principle. Furthermore, we study approximate stationarity conditions for an optimization problem with a composite objective function and geometric constraints, a qualification condition guaranteeing that approximately stationary points of such a problem are M-stationary, and a multiplier-penalty-method which naturally computes approximately stationary points of the underlying problem. Finally, necessary optimality conditions for an optimal control problem with a non-Lipschitzian sparsity-promoting term in the objective function are established. © The authors
Recommended from our members
Geometric numerical integration for optimisation
In this thesis, we study geometric numerical integration for the optimisation of various classes of functionals. Numerical integration and the study of systems of differential equations have received increased attention within the optimisation community in the last decade, as a means for devising new optimisation schemes as well as to improve our understanding of the dynamics of existing schemes. Discrete gradient methods from geometric numerical integration preserve structures of first-order gradient systems, including the dissipative structure of schemes such as gradient flows, and thus yield iterative methods that are unconditionally dissipative, i.e. decrease the objective function value for all time steps.
We look at discrete gradient methods for optimisation in several settings. First, we provide a comprehensive study of discrete gradient methods for optimisation of continuously differentiable functions. In particular, we prove properties such as well-posedness of the discrete gradient update equation, convergence rates, convergence of the iterates, and propose methods for solving the discrete gradient update equation with superior stability and convergence rates. Furthermore, we present results from numerical experiments which support the theory.
Second, motivated by the existence of derivative-free discrete gradients, and seeking to solve nonsmooth optimisation problems and more generally black-box problems, including for parameter optimisation problems, we propose methods based on the Itoh--Abe discrete gradient method for solving nonconvex, nonsmooth optimisation problems with derivative-free methods. In this setting, we prove well-posedness of the method, and convergence guarantees within the nonsmooth, nonconvex Clarke subdifferential framework for locally Lipschitz continuous functions. The analysis is shown to hold in various settings, namely in the unconstrained and constrained setting, including epi-Lipschitzian constraints, and for stochastic and deterministic optimisation methods.
Building on the work of derivative-free discrete gradient methods and the concept of structure preservation in geometric numerical integration, we consider discrete gradient methods applied to other differential systems with dissipative structures. In particular, we study the inverse scale space flow, linked to the well-known Bregman methods, which are central to variational optimisation problems and regularisation methods for inverse problems. In this setting, we propose and implement derivative-free schemes that exploit structures such as sparsity to achieve superior convergence rates in numerical experiments, and prove convergence guarantees for these methods in the nonsmooth, nonconvex setting. Furthermore, these schemes can be seen as generalisations of the Gauss-Seidel method and successive-over-relaxation.
Finally, we return to parameter optimisation problems, namely nonsmooth bilevel optimisation problems, and propose a framework to employ first-order methods for these problems, when the underlying variational optimisation problem admits a nonsmooth structure in the partial smoothness framework. In this setting, we prove piecewise differentiability of the parameter-dependent solution mapping, and study algorithmic differentiation approaches to evaluating the derivatives. Furthermore, we prove that the algorithmic derivatives converge to the implicit derivatives. Thus we demonstrate that, although some parameter tuning problems must inevitably be treated as black-box optimisation problems, for a large number of variational problems one can exploit the structure of nonsmoothness to perform gradient-based bilevel optimisation
On the Finite-Time Complexity and Practical Computation of Approximate Stationarity Concepts of Lipschitz Functions
We report a practical finite-time algorithmic scheme to compute approximately
stationary points for nonconvex nonsmooth Lipschitz functions. In particular,
we are interested in two kinds of approximate stationarity notions for
nonconvex nonsmooth problems, i.e., Goldstein approximate stationarity (GAS)
and near-approximate stationarity (NAS). For GAS, our scheme removes the
unrealistic subgradient selection oracle assumption in (Zhang et al., 2020,
Assumption 1) and computes GAS with the same finite-time complexity. For NAS,
Davis & Drusvyatskiy (2019) showed that -weakly convex functions admit
finite-time computation, while Tian & So (2021) provided the matching
impossibility results of dimension-free finite-time complexity for first-order
methods. Complement to these developments, in this paper, we isolate a new
class of functions that could be Clarke irregular (and thus not weakly convex
anymore) and show that our new algorithmic scheme can compute NAS points for
functions in that class within finite time. To demonstrate the wide
applicability of our new theoretical framework, we show that -margin SVM,
-layer, and -layer ReLU neural networks, all being Clarke irregular,
satisfy our new conditions.Comment: 20 pages, 3 figures, ICML 202
Inexact proximal methods for weakly convex functions
This paper proposes and develops inexact proximal methods for finding
stationary points of the sum of a smooth function and a nonsmooth weakly convex
one, where an error is present in the calculation of the proximal mapping of
the nonsmooth term. A general framework for finding zeros of a continuous
mapping is derived from our previous paper on this subject to establish
convergence properties of the inexact proximal point method when the smooth
term is vanished and of the inexact proximal gradient method when the smooth
term satisfies a descent condition. The inexact proximal point method achieves
global convergence with constructive convergence rates when the Moreau envelope
of the objective function satisfies the Kurdyka-Lojasiewicz (KL) property.
Meanwhile, when the smooth term is twice continuously differentiable with a
Lipschitz continuous gradient and a differentiable approximation of the
objective function satisfies the KL property, the inexact proximal gradient
method achieves the global convergence of iterates with constructive
convergence rates.Comment: 26 pages, 3 table
A decomposition algorithm for two-stage stochastic programs with nonconvex recourse
In this paper, we have studied a decomposition method for solving a class of
nonconvex two-stage stochastic programs, where both the objective and
constraints of the second-stage problem are nonlinearly parameterized by the
first-stage variable. Due to the failure of the Clarke regularity of the
resulting nonconvex recourse function, classical decomposition approaches such
as Benders decomposition and (augmented) Lagrangian-based algorithms cannot be
directly generalized to solve such models. By exploring an implicitly
convex-concave structure of the recourse function, we introduce a novel
decomposition framework based on the so-called partial Moreau envelope. The
algorithm successively generates strongly convex quadratic approximations of
the recourse function based on the solutions of the second-stage convex
subproblems and adds them to the first-stage master problem. Convergence under
both fixed scenarios and interior samplings is established. Numerical
experiments are conducted to demonstrate the effectiveness of the proposed
algorithm
Nondifferentiable Optimization: Motivations and Applications
IIASA has been involved in research on nondifferentiable optimization since 1976. The Institute's research in this field has been very productive, leading to many important theoretical, algorithmic and applied results. Nondifferentiable optimization has now become a recognized and rapidly developing branch of mathematical programming. To continue this tradition and to review developments in this field IIASA held this Workshop in Sopron (Hungary) in September 1984.
This volume contains selected papers presented at the Workshop. It is divided into four sections dealing with the following topics: (I) Concepts in Nonsmooth Analysis; (II) Multicriteria Optimization and Control Theory; (III) Algorithms and Optimization Methods; (IV) Stochastic Programming and Applications
Design of Low-Order Controllers using Optimization Techniques
In many applications, especially in the process industry, low-level controllers are the workhorses of the automated production lines. The aim of this study has been to provide simple tuning procedures, either optimization-based methods or tuning rules, for design of low-order controllers. The first part of this thesis deals with PID tuning. Design methods or both SISO and MIMO PID controllers based on convex optimization are presented. The methods consist of solving a nonconvex optimization problem by deriving convex approximations of the original problem and solving these iteratively until convergence. The algorithms are fast because of the convex approximations. The controllers obtained minimize low-frequency sensitivity subject to constraints that ensure robustness to process variations and limitations of control signal effort. The second part of this thesis deals with tuning of feedforward controllers. Tuning rules that minimize the integrated-squared-error arising from measurable step disturbances are derived for a controller that can be interpreted as a filtered and possibly time-delayed PD controller. Using a controller structure that decouples the effects of the feedforward and feedback controllers, the controller is optimal both in open and closed loop settings. To improve the high-frequency noise behavior of the feedforward controller, it is proposed that the optimal controller is augmented with a second-order filter. Several aspects on the tuning of this filter are discussed. For systems with PID controllers, the response to step changes in the reference can be improved by introducing set-point weighting. This can be interpreted as feedforward from the reference signal to the control signal. It is shown how these weights can be found by solving a convex optimization problem. Proportional set-point weight that minimizes the integrated-absolute-error was obtained for a batch of over 130 different processes. From these weights, simple tuning rules were derived and the performance was evaluated on all processes in the batch using five different feedback controller tuning methods. The proposed tuning rules could improve the performance by up to 45% with a modest increase in actuation
Recommended from our members
Nonconvex Recovery of Low-complexity Models
Today we are living in the era of big data, there is a pressing need for efficient, scalable and robust optimization methods to analyze the data we create and collect. Although Convex methods offer tractable solutions with global optimality, heuristic nonconvex methods are often more attractive in practice due to their superior efficiency and scalability. Moreover, for better representations of the data, the mathematical model we are building today are much more complicated, which often results in highly nonlinear and nonconvex optimizations problems. Both of these challenges require us to go beyond convex optimization. While nonconvex optimization is extraordinarily successful in practice, unlike convex optimization, guaranteeing the correctness of nonconvex methods is notoriously difficult. In theory, even finding a local minimum of a general nonconvex function is NP-hard – nevermind the global minimum.
This thesis aims to bridge the gap between practice and theory of nonconvex optimization, by developing global optimality guarantees for nonconvex problems arising in real-world engineering applications, and provable, efficient nonconvex optimization algorithms. First, this thesis reveals that for certain nonconvex problems we can construct a model specialized initialization that is close to the optimal solution, so that simple and efficient methods provably converge to the global solution with linear rate. These problem include sparse basis learning and convolutional phase retrieval. In addition, the work has led to the discovery of a broader class of nonconvex problems – the so-called ridable saddle functions. Those problems possess characteristic structures, in which (i) all local minima are global, (ii) the energy landscape does not have any ''flat'' saddle points. More interestingly, when data are large and random, this thesis reveals that many problems in the real world are indeed ridable saddle, those problems include complete dictionary learning and generalized phase retrieval. For each of the aforementioned problems, the benign geometric structure allows us to obtain global recovery guarantees by using efficient optimization methods with arbitrary initialization
Variational Methods for Evolution
The meeting focused on the last advances in the applications of variational methods to evolution problems governed by partial differential equations. The talks covered a broad range of topics, including large deviation and variational principles, rate-independent evolutions and gradient flows, heat flows in metric-measure spaces, propagation of fracture, applications of optimal transport and entropy-entropy dissipation methods, phase-transitions, viscous approximation, and singular-perturbation problems
- …