Global optimization for nonconvex optimization problems

Duality is one of the most successful ideas in modern science [46] [91]. It is essential in natural phenomena, particularly, in physics and mathematics [39] [94] [96]. In this thesis, we consider the canonical duality theory for several classes of optimization problems.The first problem that we consider is a general sum of fourth-order polynomial minimization problem. This problem arises extensively in engineering and science, including database analysis, computational biology, sensor network communications, nonconvex mechanics, and ecology. We first show that this global optimization problem is actually equivalent to a discretized minimal potential variational problem in large deformation mechanics. Therefore, a general analytical solution is proposed by using the canonical duality theory.The second problem that we consider is a nonconvex quadratic-exponential optimization problem. By using the canonical duality theory, the nonconvex primal problem in n-dimensional space can be converted into a one-dimensional canonical dual problem, which is either a concave maximization or a convex minimization problem with zero duality gap. Several examples are solved so as to illustrate the applicability of the theory developed.The third problem that we consider is quadratic minimization problems subjected to either box or integer constraints. Results show that these nonconvex problems can be converted into concave maximization dual problems over convex feasible spaces without duality gap and the Boolean integer programming problem is actually equivalent to a critical point problem in continuous space. These dual problems can be solved under certain conditions. Both existence and uniqueness of the canonical dual solutions are presented. A canonical duality algorithm is presented and applications are illustrated.The fourth problem that we consider is a quadratic discrete value selection problem subjected to inequality constraints. The problem is first transformed into a quadratic 0-1 integer programming problem. The dual problem is thus constructed by using the canonical duality theory. Under appropriate conditions, this dual problem is a maximization problem of a concave function over a convex continuous space. Theoretical results show that the canonical duality theory can either provide a global optimization solution, or an optimal lower bound approximation to this NP-hard problem. Numerical simulation studies, including some relatively large scale problems, are carried out so as to demonstrate the effectiveness and efficiency of the canonical duality method. An open problem for understanding NP-hard problems is proposed.The fifth problem that we consider is a mixed-integer quadratic minimization problem with fixed cost terms. We show that this well-known NP-hard problem in R2n can be transformed into a continuous concave maximization dual problem over a convex feasible subset of Rn with zero duality gap. We also discuss connections between the proposed canonical duality theory approach and the classical Lagrangian duality approach. The resulting canonical dual problem can be solved under certain conditions, by traditional convex programming methods. Conditions for the existence and uniqueness of global optimal solutions are presented. An application to a decoupled mixed-integer problem is used to illustrate the derivation of analytic solutions for globally minimizing the objective function. Numerical examples for both decoupled and general mixed-integral problems are presented, and an open problem is proposed for future study.The sixth problem that we consider is a general nonconvex quadratic minimization problem with nonconvex constraints. By using the canonical dual transformation, the nonconvex primal problem can be converted into a canonical dual problem (i.e., either a concave maximization problem with zero duality gap). Illustrative applications to quadratic minimization with multiple quadratic constraints, box/integer constraints, and general nonconvex polynomial constraints are discussed, along with insightful connections to classical Lagrangian duality. Conditions for ensuring the existence and uniqueness of global optimal solutions are presented. Several numerical examples are solved.The seventh problem that we consider is a general nonlinear algebraic system. By using the least square method, the nonlinear system of m quadratic equations in n-dimensional space is first formulated as a nonconvex optimization problem. We then prove that, by using the canonical duality theory, this nonconvex problem is equivalent to a concave maximization problem in Rm, which can be solved by well-developed convex optimization techniques. Both existence and uniqueness of global optimal solutions are discussed, and several illustrative examples are presented.The eighth problem that we consider is a general sensor network localization problem. It is shown that by the canonical duality theory, this nonconvex minimization problem is equivalent to a concave maximization problem over a convex set in a symmetrical matrix space, and hence can be solved by combining a perturbation technique with existing optimization techniques. Applications are illustrated and results show that the proposed method is potentially a powerful one for large-scale sensor network localization problems

Practical implementation of nonlinear time series methods: The TISEAN package

Nonlinear time series analysis is becoming a more and more reliable tool for the study of complicated dynamics from measurements. The concept of low-dimensional chaos has proven to be fruitful in the understanding of many complex phenomena despite the fact that very few natural systems have actually been found to be low dimensional deterministic in the sense of the theory. In order to evaluate the long term usefulness of the nonlinear time series approach as inspired by chaos theory, it will be important that the corresponding methods become more widely accessible. This paper, while not a proper review on nonlinear time series analysis, tries to make a contribution to this process by describing the actual implementation of the algorithms, and their proper usage. Most of the methods require the choice of certain parameters for each specific time series application. We will try to give guidance in this respect. The scope and selection of topics in this article, as well as the implementational choices that have been made, correspond to the contents of the software package TISEAN which is publicly available from http://www.mpipks-dresden.mpg.de/~tisean . In fact, this paper can be seen as an extended manual for the TISEAN programs. It fills the gap between the technical documentation and the existing literature, providing the necessary entry points for a more thorough study of the theoretical background.Comment: 27 pages, 21 figures, downloadable software at http://www.mpipks-dresden.mpg.de/~tisea

Learning fast, accurate, and stable closures of a kinetic theory of an active fluid

Important classes of active matter systems can be modeled using kinetic theories. However, kinetic theories can be high dimensional and challenging to simulate. Reduced-order representations based on tracking only low-order moments of the kinetic model serve as an efficient alternative, but typically require closure assumptions to model unrepresented higher-order moments. In this study, we present a learning framework based on neural networks that exploit rotational symmetries in the closure terms to learn accurate closure models directly from kinetic simulations. The data-driven closures demonstrate excellent a-priori predictions comparable to the state-of-the-art Bingham closure. We provide a systematic comparison between different neural network architectures and demonstrate that nonlocal effects can be safely ignored to model the closure terms. We develop an active learning strategy that enables accurate prediction of the closure terms across the entire parameter space using a single neural network without the need for retraining. We also propose a data-efficient training procedure based on time-stepping constraints and a differentiable pseudo-spectral solver, which enables the learning of stable closures suitable for a-posteriori inference. The coarse-grained simulations equipped with data-driven closure models faithfully reproduce the mean velocity statistics, scalar order parameters, and velocity power spectra observed in simulations of the kinetic theory. Our differentiable framework also facilitates the estimation of parameters in coarse-grained descriptions conditioned on data

Polynomial distribution functions on bounded closed intervals

The thesis explores several topics, related to polynomial distribution functions and their densities on [0,1]M, including polynomial copula functions and their densities. The contribution of this work can be subdivided into two areas. - Studying the characterization of the extreme sets of polynomial densities and copulas, which is possible due to the Choquet theorem. - Development of statistical methods that utilize the fact that the density is polynomial (which may or may not be an extreme density). With regard to the characterization of the extreme sets, we first establish that in all dimensions the density of an extreme distribution function is an extreme density. As a consequence, characterizing extreme distribution functions is equivalent to characterizing extreme densities, which is easier analytically. We provide the full constructive characterization of the Choquet-extreme polynomial densities in the univariate case, prove several necessary and sufficient conditions for the extremality of densities in arbitrary dimension, provide necessary conditions for extreme polynomial copulas, and prove characterizing duality relationships for polynomial copulas. We also introduce a special case of reflexive polynomial copulas. Most of the statistical methods we consider are restricted to the univariate case. We explore ways to construct univariate densities by mixing the extreme ones, propose non-parametric and ML estimators of polynomial densities. We introduce a new procedure to calibrate the mixing distribution and propose an extension of the standard method of moments to pinned density moment matching. As an application of the multivariate polynomial copulas, we introduce polynomial coupling and explore its application to convolution of coupled random variables. The introduction is followed by a summary of the contributions of this thesis and the sections, dedicated first to the univariate case, then to the general multivariate case, and then to polynomial copula densities. Each section first presents the main results, followed by the literature review

High-dimensional Bayesian optimization with intrinsically low-dimensional response surfaces

Bayesian optimization is a powerful technique for the optimization of expensive black-box functions. It is used in a wide range of applications such as in drug and material design and training of machine learning models, e.g. large deep networks. We propose to extend this approach to high-dimensional settings, that is where the number of parameters to be optimized exceeds 10--20. In this thesis, we scale Bayesian optimization by exploiting different types of projections and the intrinsic low-dimensionality assumption of the objective function. We reformulate the problem in a low-dimensional subspace and learn a response surface and maximize an acquisition function in this low-dimensional projection. Contributions include i) a probabilistic model for axis-aligned projections, such as the quantile-Gaussian process and ii) a probabilistic model for learning a feature space by means of manifold Gaussian processes. In the latter contribution, we propose to learn a low-dimensional feature space jointly with (a) the response surface and (b) a reconstruction mapping. Finally, we present empirical results against well-known baselines in high-dimensional Bayesian optimization and provide possible directions for future research in this field.Open Acces

Approche variationnelle de la fonction rang : relaxation convexe, sous-différentiation généralisée, régularisation-approximation de Moreau

Dans ce mémoire de these, nous étudions la fonction rang du point de vue variationnel. La raison pour laquelle nous nous intéressons à cette fonction est qu'elle apparaît comme une fonction objectif (ou comme fonction contrainte) dans divers problèmes d'optimisation moderne, par exemple: complétion de matrices, analyse de données statistiques, acquisition parcimonieuse de données, etc. Dans certains cas particuliers, les problèmes de minimisation de la fonction rang peuvent être résolus en utilisant la décomposition en valeurs singulières. Mais, en géneral, les problèmes de minimisation de la fonction rang sont « NP-difficiles ». Nous proposons ici quelques propriétés de la fonction rang du point de vue variationnel: des démonstrations supplémentaires pour son enveloppe convexe fermée (restreinte à des boules spectrales), les expressions des sous-différentiels généralisés et la régularisation-approximation au sens de Moreau. Puis, dans le dernier chapitre, nous revenons sur une notion dont la définition ressemble à celle de la fonction rang, la fonction cp-rang.In this dissertation, we consider the rank function from the variational point of view. The reason why we are interested in this function is that it appears as an objective (or constraint) function in various modern optimization problems, such as: low rank matrix completion, multivariate statistical data analysis, compressed sensing, etc. In some particular cases, the rank minimization problems can be solved by using the singular value decomposition of matrices or can be reduced to the solution of linear systems. But in general, the rank minimization problems is known to be « NP-hard ». We provide here several properties of the rank function from the variational point of view: additional proofs for its closed convex relaxation, the expressions of its generalized subdifferentials and the explicit expression of its Moreau regularization-approximation form. Then, in the last chapter, we revisit a notion whose definition resembles that of the rank, the cp-rank function

A Brief Review on Mathematical Tools Applicable to Quantum Computing for Modelling and Optimization Problems in Engineering

Since its emergence, quantum computing has enabled a wide spectrum of new possibilities and advantages, including its efficiency in accelerating computational processes exponentially. This has directed much research towards completely novel ways of solving a wide variety of engineering problems, especially through describing quantum versions of many mathematical tools such as Fourier and Laplace transforms, differential equations, systems of linear equations, and optimization techniques, among others. Exploration and development in this direction will revolutionize the world of engineering. In this manuscript, we review the state of the art of these emerging techniques from the perspective of quantum computer development and performance optimization, with a focus on the most common mathematical tools that support engineering applications. This review focuses on the application of these mathematical tools to quantum computer development and performance improvement/optimization. It also identifies the challenges and limitations related to the exploitation of quantum computing and outlines the main opportunities for future contributions. This review aims at offering a valuable reference for researchers in fields of engineering that are likely to turn to quantum computing for solutions. Doi: 10.28991/ESJ-2023-07-01-020 Full Text: PD

Real Algebraic Geometry With a View Toward Moment Problems and Optimization

Continuing the tradition initiated in MFO workshop held in 2014, the aim of this workshop was to foster the interaction between real algebraic geometry, operator theory, optimization, and algorithms for systems control. A particular emphasis was given to moment problems through an interesting dialogue between researchers working on these problems in finite and infinite dimensional settings, from which emerged new challenges and interdisciplinary applications

Theory and Methods for Large Spatial Data

Correlated Gaussian processes are of central importance to the study of time series, spatial statistics, computer experiments, and many machine learning models. Large spatially or temporally indexed datasets bring with them a host of computational and mathematical challenges. Parameter estimation of these processes often relies on maximum likelihood, which for Gaussian processes involves manipulations of the covariance matrix including solving systems of equations and determinant calculations. The score function, on the other hand, avoids direct calculation of the determinant, but still requires solving a large number of linear equations. We propose an equivalent kernel approximation to the score function of a stationary Gaussian process. A nugget effect is required for the approximation. We suggest two approximations, and for large sample sizes, our proposals are fast, accurate, and compare well against existing approaches.We then present a method for simulating time series of high frequency wind data calibrated by real data. The method provides and fits a parametric model for local wind directions by embedding them into the angular projection of a bivariate normal. Incorporating a temporal autocorrelation structure in that normal induces a continuous angular correlation over time in the simulated wind directions. The final joint model for speed and direction can be decomposed into the simulation of a single multivariate normal and a series of transformations thereof, allowing for fast and easy repeated generations of long time series. This is compared to a state of the art approach for simulating angular time series of swapping between discrete regimes of wind direction, a method that does not fully translate to high frequency data