Nonlinear Integer Programming

Research efforts of the past fifty years have led to a development of linear integer programming as a mature discipline of mathematical optimization. Such a level of maturity has not been reached when one considers nonlinear systems subject to integrality requirements for the variables. This chapter is dedicated to this topic. The primary goal is a study of a simple version of general nonlinear integer problems, where all constraints are still linear. Our focus is on the computational complexity of the problem, which varies significantly with the type of nonlinear objective function in combination with the underlying combinatorial structure. Numerous boundary cases of complexity emerge, which sometimes surprisingly lead even to polynomial time algorithms. We also cover recent successful approaches for more general classes of problems. Though no positive theoretical efficiency results are available, nor are they likely to ever be available, these seem to be the currently most successful and interesting approaches for solving practical problems. It is our belief that the study of algorithms motivated by theoretical considerations and those motivated by our desire to solve practical instances should and do inform one another. So it is with this viewpoint that we present the subject, and it is in this direction that we hope to spark further research.Comment: 57 pages. To appear in: M. J\"unger, T. Liebling, D. Naddef, G. Nemhauser, W. Pulleyblank, G. Reinelt, G. Rinaldi, and L. Wolsey (eds.), 50 Years of Integer Programming 1958--2008: The Early Years and State-of-the-Art Surveys, Springer-Verlag, 2009, ISBN 354068274

Fast Non-Parametric Learning to Accelerate Mixed-Integer Programming for Online Hybrid Model Predictive Control

Today's fast linear algebra and numerical optimization tools have pushed the frontier of model predictive control (MPC) forward, to the efficient control of highly nonlinear and hybrid systems. The field of hybrid MPC has demonstrated that exact optimal control law can be computed, e.g., by mixed-integer programming (MIP) under piecewise-affine (PWA) system models. Despite the elegant theory, online solving hybrid MPC is still out of reach for many applications. We aim to speed up MIP by combining geometric insights from hybrid MPC, a simple-yet-effective learning algorithm, and MIP warm start techniques. Following a line of work in approximate explicit MPC, the proposed learning-control algorithm, LNMS, gains computational advantage over MIP at little cost and is straightforward for practitioners to implement

Empirical Bounds on Linear Regions of Deep Rectifier Networks

We can compare the expressiveness of neural networks that use rectified linear units (ReLUs) by the number of linear regions, which reflect the number of pieces of the piecewise linear functions modeled by such networks. However, enumerating these regions is prohibitive and the known analytical bounds are identical for networks with same dimensions. In this work, we approximate the number of linear regions through empirical bounds based on features of the trained network and probabilistic inference. Our first contribution is a method to sample the activation patterns defined by ReLUs using universal hash functions. This method is based on a Mixed-Integer Linear Programming (MILP) formulation of the network and an algorithm for probabilistic lower bounds of MILP solution sets that we call MIPBound, which is considerably faster than exact counting and reaches values in similar orders of magnitude. Our second contribution is a tighter activation-based bound for the maximum number of linear regions, which is particularly stronger in networks with narrow layers. Combined, these bounds yield a fast proxy for the number of linear regions of a deep neural network.Comment: AAAI 202

Stability Verification of Neural Network Controllers using Mixed-Integer Programming

We propose a framework for the stability verification of Mixed-Integer Linear Programming (MILP) representable control policies. This framework compares a fixed candidate policy, which admits an efficient parameterization and can be evaluated at a low computational cost, against a fixed baseline policy, which is known to be stable but expensive to evaluate. We provide sufficient conditions for the closed-loop stability of the candidate policy in terms of the worst-case approximation error with respect to the baseline policy, and we show that these conditions can be checked by solving a Mixed-Integer Quadratic Program (MIQP). Additionally, we demonstrate that an outer and inner approximation of the stability region of the candidate policy can be computed by solving an MILP. The proposed framework is sufficiently general to accommodate a broad range of candidate policies including ReLU Neural Networks (NNs), optimal solution maps of parametric quadratic programs, and Model Predictive Control (MPC) policies. We also present an open-source toolbox in Python based on the proposed framework, which allows for the easy verification of custom NN architectures and MPC formulations. We showcase the flexibility and reliability of our framework in the context of a DC-DC power converter case study and investigate its computational complexity

On the Relationship Between the Value Function and the Efficient Frontier of a Mixed Integer Linear Optimization Problem

In this paper, we investigate the connection between the efficient frontier (EF) of a general multiobjective mixed integer linear optimization problem (MILP) and the so-called restricted value function (RVF) of a closely related single-objective MILP. We demonstrate that the EF of the multiobjective MILP is comprised of points on the boundary of the epigraph of the RVF so that any description of the EF suffices to describe the RVF and vice versa. In the first part of the paper, we describe the mathematical structure of the RVF, including characterizing the set of points at which it is differentiable, the gradients at such points, and the subdifferential at all nondifferentiable points. Because of the close relationship of the RVF to the EF, we observe that methods for constructing so-called value functions and methods for constructing the EF of a multiobjective optimization problem, each of which have been developed in separate communities, are effectively interchangeable. By exploiting this relationship, we propose a generalized cutting plane algorithm for constructing the EF of a multiobjective MILP based on a generalization of an existing algorithm for constructing the classical value function. We prove that the algorithm is finite under a standard boundedness assumption and comes with a performance guarantee if terminated early

Two-Stage Stochastic Mixed Integer Linear Optimization

The primary focus of this dissertation is on optimization problems that involve uncertainty unfolding over time. In many real-world decisions, the decision-maker has to decide in the face of uncertainty. After the outcome of the uncertainty is observed, she can correct her initial decision by taking some corrective actions at a later time stage. These problems are known as stochastic optimization problems with recourse. In the case that the number of time stages is limited to two, these problems are referred to as two-stage stochastic optimization problems. We focus on this class of optimization problems in this dissertation. The optimization problem that is solved before the realization of uncertainty is called the first-stage problem and the problem solved to make a corrective action on the initial decision is called the second-stage problem. The decisions made in the second- stage are affected by both the first-stage decisions and the realization of random variables. Consequently, the two-stage problem can be viewed as a parametric optimization problem which involves the so-called value function of the second-stage problem. The value function describes the change in optimal objective value as the right-hand side is varied and understanding it is crucial to developing solution methods for two-stage optimization problems.In the first part of this dissertation, we study the value function of a MILP. We review the structural properties of the value function and its construction methods. We con- tribute by proposing a discrete representation of the MILP value function. We show that the structure of the MILP value function arises from two other optimization problems that are constructed from its discrete and continuous components. We show that our representation can explain certain structural properties of the MILP value function such as the sets over which the value function is convex. We then provide a simplification of the Jeroslow Formula obtained by applying our results. Finally, we describe a cutting plane algorithm for its construction and determine the conditions under which the pro- posed algorithm is finite.Traditionally, the solution methods developed for two-stage optimization problems consider the problem where the second-stage problem involves only continuous variables. In the recent years, however, two-stage problems with integer variables in the second- stage have been visited in several studies. These problems are important in practice and arise in several applications in supply chain, finance, forestry and disaster management, among others. The second part of this dissertation concerns the development and implementation of a solution method for the two-stage optimization problem where both the first and second stage involve mixed integer variables. We describe a generalization of the classical Benders’ method for solving mixed integer two-stage stochastic linear optimization problems. We employ the strong dual functions encoded in the branch-and-bound trees resulting from solution of the second-stage problem. We show that these can be used effectively within a Benders’ framework and describe a method for obtaining all required dual functions from a single, continuously refined branch-and-bound tree that is used to warm start the solution procedure for each subproblem.Finally, we provide details on the implementation of our proposed algorithm. The implementation allows for construction of several approximations of the value function of the second-stage problem. We use different warm-starting strategies within our proposed algorithm to solve the second-stage problems, including solving all second-stage problems with a single tree. We provide computational results on applying these strategies to the stochastic server problems (SSLP) from the stochastic integer programming test problem library (SIPLIB)