A Machine Learning Approach to Solving Large Bilevel and Stochastic Programs: Application to Cycling Network Design

We present a novel machine learning-based approach to solving bilevel programs that involve a large number of independent followers, which as a special case include two-stage stochastic programming. We propose an optimization model that explicitly considers a sampled subset of followers and exploits a machine learning model to estimate the objective values of unsampled followers. Unlike existing approaches, we embed machine learning model training into the optimization problem, which allows us to employ general follower features that can not be represented using leader decisions. We prove bounds on the optimality gap of the generated leader decision as measured by the original objective function that considers the full follower set. We then develop follower sampling algorithms to tighten the bounds and a representation learning approach to learn follower features, which can be used as inputs to the embedded machine learning model. Using synthetic instances of a cycling network design problem, we compare the computational performance of our approach versus baseline methods. Our approach provides more accurate predictions for follower objective values, and more importantly, generates leader decisions of higher quality. Finally, we perform a real-world case study on cycling infrastructure planning, where we apply our approach to solve a network design problem with over one million followers. Our approach presents favorable performance compared to the current cycling network expansion practices

Bilevel optimisation with embedded neural networks: Application to scheduling and control integration

Scheduling problems requires to explicitly account for control considerations in their optimisation. The literature proposes two traditional ways to solve this integrated problem: hierarchical and monolithic. The monolithic approach ignores the control level's objective and incorporates it as a constraint into the upper level at the cost of suboptimality. The hierarchical approach requires solving a mathematically complex bilevel problem with the scheduling acting as the leader and control as the follower. The linking variables between both levels belong to a small subset of scheduling and control decision variables. For this subset of variables, data-driven surrogate models have been used to learn follower responses to different leader decisions. In this work, we propose to use ReLU neural networks for the control level. Consequently, the bilevel problem is collapsed into a single-level MILP that is still able to account for the control level's objective. This single-level MILP reformulation is compared with the monolithic approach and benchmarked against embedding a nonlinear expression of the neural networks into the optimisation. Moreover, a neural network is used to predict control level feasibility. The case studies involve batch reactor and sequential batch process scheduling problems. The proposed methodology finds optimal solutions while largely outperforming both approaches in terms of computational time. Additionally, due to well-developed MILP solvers, adding ReLU neural networks in a MILP form marginally impacts the computational time. The solution's error due to prediction accuracy is correlated with the neural network training error. Overall, we expose how - by using an existing big-M reformulation and being careful about integrating machine learning and optimisation pipelines - we can more efficiently solve the bilevel scheduling-control problem with high accuracy.Comment: 18 page

Fuzzy Bi-level Decision-Making Techniques: A Survey

© 2016 the authors. Bi-level decision-making techniques aim to deal with decentralized management problems that feature interactive decision entities distributed throughout a bi-level hierarchy. A challenge in handling bi-level decision problems is that various uncertainties naturally appear in decision-making process. Significant efforts have been devoted that fuzzy set techniques can be used to effectively deal with uncertain issues in bi-level decision-making, known as fuzzy bi-level decision-making techniques, and researchers have successfully gained experience in this area. It is thus vital that an instructive review of current trends in this area should be conducted, not only of the theoretical research but also the practical developments. This paper systematically reviews up-to-date fuzzy bi-level decisionmaking techniques, including models, approaches, algorithms and systems. It also clusters related technique developments into four main categories: basic fuzzy bi-level decision-making, fuzzy bi-level decision-making with multiple optima, fuzzy random bi-level decision-making, and the applications of bi-level decision-making techniques in different domains. By providing state-of-the-art knowledge, this survey paper will directly support researchers and practitioners in their understanding of developments in theoretical research results and applications in relation to fuzzy bi-level decision-making techniques

Integrated Optimization And Learning Methods Of Predictive And Prescriptive Analytics

A typical decision problem optimizes one or more objectives subject to a set of constraints on its decision variables. Most real-world decision problems contain uncertain parameters. The exponential growth of data availability, ease of accessibility in computational power, and more efficient optimization techniques have paved the way for machine learning tools to effectively predict these uncertain parameters. Traditional machine learning models measure the quality of predictions based on the closeness between true and predicted values and ignore decision problems involving uncertain parameters for which predicted values are treated as the true values.Standard approaches passing point estimates of machine learning models into decision problems as replacement of uncertain parameters lose the connection between predictive and prescriptive tasks. Recently developed methods to strengthen the bond between predictive and prescriptive tasks still rely on either first predict, then optimize strategy or use approximation techniques in integrating predictive and prescriptive tasks. We develop an integrated framework for performing predictive and prescriptive analytics concurrently to realize the best prescriptive performance under uncertainty. This framework is applicable to all prescriptive tasks involving uncertainty. Further, it is scalable to handle integrated predictive and prescriptive tasks with reasonable computational effort and enables users to apply decomposition algorithms for large-scale problems. The framework also accommodates prediction tasks ranging from simple regression to more complex black-box neural network models. The integrated optimization framework is composed of two integration approaches. The first approach integrates regression-based prediction and mathematical programming-based prescription tasks as a bilevel program. While the lower-level problem prescribes decisions based on the predicted outcome for a specific observation, the upper-level evaluates the quality of decisions with respect to true values. The upper-level problem can be considered as a prescriptive error, and the goal is to minimize this prescriptive error. In order to achieve the same performance in external data sets (test) compared to internal data sets (train), we offer different approaches to control the prescription generalization error associated with out-of-sample observation. We develop a decomposition algorithm for large-scale problems by leveraging a progressive hedging algorithm to solve the resulting bilevel formulation. The second approach integrates the learning of neural network-based prediction and optimization tasks as a nested neural network. While the predictive neural network promotes decisions based on predicted outcomes, the prescriptive neural network evaluates the quality of predicted decisions with respect to true values. We also propose a weight initialization process for nested neural networks and build a decomposition algorithm for large-scale problems. Our results for the example problems validate the performance of our proposed integrated predictive and prescriptive optimization and training frameworks. With customarily generated synthetic data sets, proposed methods surpass all of the first predict, then optimize approaches and recently developed approximate integration methods for both in-sample and out of sample data sets. We also observe how the proposed generalization error controlling approach improves results in out of sample data sets. Customarily generated synthetic data pairs at different levels of correlation and non-linearity graphically show us how different methods converge to each other