ABS Algorithms for Linear Systems and Optimization

We present a review and bibliography of the main results obtained during a research on ABS (Abaffy, Broyden, Spedicato) methods.Comment: 38 page

Machine Learning of Space-Fractional Differential Equations

Data-driven discovery of "hidden physics" -- i.e., machine learning of differential equation models underlying observed data -- has recently been approached by embedding the discovery problem into a Gaussian Process regression of spatial data, treating and discovering unknown equation parameters as hyperparameters of a modified "physics informed" Gaussian Process kernel. This kernel includes the parametrized differential operators applied to a prior covariance kernel. We extend this framework to linear space-fractional differential equations. The methodology is compatible with a wide variety of fractional operators in $\mathbb{R}^d$ and stationary covariance kernels, including the Matern class, and can optimize the Matern parameter during training. We provide a user-friendly and feasible way to perform fractional derivatives of kernels, via a unified set of d-dimensional Fourier integral formulas amenable to generalized Gauss-Laguerre quadrature. The implementation of fractional derivatives has several benefits. First, it allows for discovering fractional-order PDEs for systems characterized by heavy tails or anomalous diffusion, bypassing the analytical difficulty of fractional calculus. Data sets exhibiting such features are of increasing prevalence in physical and financial domains. Second, a single fractional-order archetype allows for a derivative of arbitrary order to be learned, with the order itself being a parameter in the regression. This is advantageous even when used for discovering integer-order equations; the user is not required to assume a "dictionary" of derivatives of various orders, and directly controls the parsimony of the models being discovered. We illustrate on several examples, including fractional-order interpolation of advection-diffusion and modeling relative stock performance in the S&P 500 with alpha-stable motion via a fractional diffusion equation.Comment: 26 pages, 10 figures. In v2, a minor change to the formatting of a handful of references was made in the bibliography; the main text was unchanged. In v3, minor improvements were made to the exposition; more details about motivation, examples, optimization, and relation to previous works were give

Derivative-free optimization methods

In many optimization problems arising from scientific, engineering and artificial intelligence applications, objective and constraint functions are available only as the output of a black-box or simulation oracle that does not provide derivative information. Such settings necessitate the use of methods for derivative-free, or zeroth-order, optimization. We provide a review and perspectives on developments in these methods, with an emphasis on highlighting recent developments and on unifying treatment of such problems in the non-linear optimization and machine learning literature. We categorize methods based on assumed properties of the black-box functions, as well as features of the methods. We first overview the primary setting of deterministic methods applied to unconstrained, non-convex optimization problems where the objective function is defined by a deterministic black-box oracle. We then discuss developments in randomized methods, methods that assume some additional structure about the objective (including convexity, separability and general non-smooth compositions), methods for problems where the output of the black-box oracle is stochastic, and methods for handling different types of constraints

Canonical Duality-Triality Theory: Bridge Between Nonconvex Analysis/Mechanics and Global Optimization in Complex Systems

Canonical duality-triality is a breakthrough methodological theory, which can be used not only for modeling complex systems within a unified framework, but also for solving a wide class of challenging problems from real-world applications. This paper presents a brief review on this theory, its philosophical origin, physics foundation, and mathematical statements in both finite and infinite dimensional spaces, with emphasizing on its role for bridging the gap between nonconvex analysis/mechanics and global optimization. Special attentions are paid on unified understanding the fundamental difficulties in large deformation mechanics, bifurcation/chaos in nonlinear science, and the NP-hard problems in global optimization, as well as the theorems, methods, and algorithms for solving these challenging problems. Misunderstandings and confusions on some basic concepts, such as objectivity, nonlinearity, Lagrangian, and generalized convexities are discussed and classified. Breakthrough from recent challenges and conceptual mistakes by M. Voisei, C. Zalinescu and his co-worker are addressed. Some open problems and future works in global optimization and nonconvex mechanics are proposed.Comment: 43 pages, 4 figures. appears in Mathematics and Mechanics of Solids, 201

Deep Neural Networks as 0-1 Mixed Integer Linear Programs: A Feasibility Study

Deep Neural Networks (DNNs) are very popular these days, and are the subject of a very intense investigation. A DNN is made by layers of internal units (or neurons), each of which computes an affine combination of the output of the units in the previous layer, applies a nonlinear operator, and outputs the corresponding value (also known as activation). A commonly-used nonlinear operator is the so-called rectified linear unit (ReLU), whose output is just the maximum between its input value and zero. In this (and other similar cases like max pooling, where the max operation involves more than one input value), one can model the DNN as a 0-1 Mixed Integer Linear Program (0-1 MILP) where the continuous variables correspond to the output values of each unit, and a binary variable is associated with each ReLU to model its yes/no nature. In this paper we discuss the peculiarity of this kind of 0-1 MILP models, and describe an effective bound-tightening technique intended to ease its solution. We also present possible applications of the 0-1 MILP model arising in feature visualization and in the construction of adversarial examples. Preliminary computational results are reported, aimed at investigating (on small DNNs) the computational performance of a state-of-the-art MILP solver when applied to a known test case, namely, hand-written digit recognition.Comment: submitted to an international conferenc

Reinforcement Learning for Batch Bioprocess Optimization

Bioprocesses have received a lot of attention to produce clean and sustainable alternatives to fossil-based materials. However, they are generally difficult to optimize due to their unsteady-state operation modes and stochastic behaviours. Furthermore, biological systems are highly complex, therefore plant-model mismatch is often present. To address the aforementioned challenges we propose a Reinforcement learning based optimization strategy for batch processes. In this work, we applied the Policy Gradient method from batch-to-batch to update a control policy parametrized by a recurrent neural network. We assume that a preliminary process model is available, which is exploited to obtain a preliminary optimal control policy. Subsequently, this policy is updatedbased on measurements from thetrueplant. The capabilities of our proposed approach were tested on three case studies (one of which is nonsmooth) using a more complex process model for thetruesystemembedded with adequate process disturbance. Lastly, we discussed the advantages and disadvantages of this strategy compared against current existing approaches such as nonlinear model predictive control

Relaxations of AC Maximal Load Delivery for Severe Contingency Analysis

This work considers the task of finding an AC-feasible operating point of a severely damaged transmission network while ensuring that a maximal amount of active power loads can be delivered. This AC Maximal Load Delivery (AC-MLD) task is a nonconvex nonlinear optimization problem that is incredibly challenging to solve on large-scale transmission system datasets. This work demonstrates that convex relaxations of the AC-MLD problem provide a reliable and scalable method for finding high-quality bounds on the amount of active power that can be delivered in the AC-MLD problem. To demonstrate their effectiveness, the solution methods proposed in this work are rigorously evaluated on 1000 N-k scenarios on seven power networks ranging in size from 70 to 6000 buses. The most effective relaxation of the AC-MLD problem converges in less than 20 seconds on commodity computing hardware for all 7000 of the scenarios considered.Comment: The problem considered in this work is also known as AC Minimal Load-Sheddin

Generating Reflectance Curves from sRGB Triplets

The color sensation evoked by an object depends on both the spectral power distribution of the illumination and the reflectance properties of the object being illuminated. The color sensation can be characterized by three color-space values, such as XYZ, RGB, HSV, L*a*b*, etc. It is straightforward to compute the three values given the illuminant and reflectance curves. The converse process of computing a reflectance curve given the color-space values and the illuminant is complicated by the fact that an infinite number of different reflectance curves can give rise to a single set of color-space values (metamerism). This paper presents five algorithms for generating a reflectance curve from a specified sRGB triplet, written for a general audience. The algorithms are designed to generate reflectance curves that are similar to those found with naturally occurring colored objects. The computed reflectance curves are compared to a database of thousands of reflectance curves measured from paints and pigments available both commercially and in nature, and the similarity is quantified. One particularly useful application of these algorithms is in the field of computer graphics, where modeling color transformations sometimes requires wavelength-specific information, such as when modeling subtractive color mixture.Comment: v3 minor editing to clarify some points, and some webpage link updates, v4 adds the LHTSS method, v5 indicates LHTSS should be preferred to ILLSS generall

A theory on the absence of spurious solutions for nonconvex and nonsmooth optimization

We study the set of continuous functions that admit no spurious local optima (i.e. local minima that are not global minima) which we term \textit{global functions}. They satisfy various powerful properties for analyzing nonconvex and nonsmooth optimization problems. For instance, they satisfy a theorem akin to the fundamental uniform limit theorem in the analysis regarding continuous functions. Global functions are also endowed with useful properties regarding the composition of functions and change of variables. Using these new results, we show that a class of nonconvex and nonsmooth optimization problems arising in tensor decomposition applications are global functions. This is the first result concerning nonconvex methods for nonsmooth objective functions. Our result provides a theoretical guarantee for the widely-used $\ell_1$ norm to avoid outliers in nonconvex optimization.Comment: 22 pages, 13 figure

On Unified Modeling, Canonical Duality-Triality Theory, Challenges and Breakthrough in Optimization

A unified model is addressed for general optimization problems in multi-scale complex systems. Based on necessary conditions and basic principles in physics, the canonical duality-triality theory is presented in a precise way to include traditional duality theories and popular methods as special applications. Two conjectures on NP-hardness are discussed, which should play important roles for correctly understanding and efficiently solving challenging real-world problems. Applications are illustrated for both nonconvex continuous optimization and mixed integer nonlinear programming. Misunderstandings and confusion on some basic concepts, such as objectivity, nonlinearity, Lagrangian, and Lagrange multiplier method are discussed and classified. Breakthrough from recent false challenges by C. Z\u{a}linescu and his co-workers are addressed. This paper will bridge a significant gap between optimization and multi-disciplinary fields of applied math and computational sciences.Comment: 28 pages, 2 figure