Towards Better Integration of Surrogate Models and Optimizers

Surrogate-Assisted Evolutionary Algorithms (SAEAs) have been proven to be very effective in solving (synthetic and real-world) computationally expensive optimization problems with a limited number of function evaluations. The two main components of SAEAs are: the surrogate model and the evolutionary optimizer, both of which use parameters to control their respective behavior. These parameters are likely to interact closely, and hence the exploitation of any such relationships may lead to the design of an enhanced SAEA. In this chapter, as a first step, we focus on Kriging and the Efficient Global Optimization (EGO) framework. We discuss potentially profitable ways of a better integration of model and optimizer. Furthermore, we investigate in depth how different parameters of the model and the optimizer impact optimization results. In particular, we determine whether there are any interactions between these parameters, and how the problem characteristics impact optimization results. In the experimental study, we use the popular Black-Box Optimization Benchmarking (BBOB) testbed. Interestingly, the analysis finds no evidence for significant interactions between model and optimizer parameters, but independently their performance has a significant interaction with the objective function. Based on our results, we make recommendations on how best to configure EGO

Polynomial Response Surface Approximations for the Multidisciplinary Design Optimization of a High Speed Civil Transport

Surrogate functions have become an important tool in multidisciplinary design optimization to deal with noisy functions, high computational cost, and the practical difficulty of integrating legacy disciplinary computer codes. A combination of mathematical, statistical, and engineering techniques, well known in other contexts, have made polynomial surrogate functions viable for MDO. Despite the obvious limitations imposed by sparse high fidelity data in high dimensions and the locality of low order polynomial approximations, the success of the panoply of techniques based on polynomial response surface approximations for MDO shows that the implementation details are more important than the underlying approximation method (polynomial, spline, DACE, kernel regression, etc.). This paper surveys some of the ancillary techniques—statistics, global search, parallel computing, variable complexity modeling—that augment the construction and use of polynomial surrogates

Machine Learning Methods to Estimate Whole-Brain Effective Connectome for ASD Identification

Functional Magnetic Resonance Imaging (fMRI) is widely used to study neural-developmental diseases such as Autism Spectrum Disorder (ASD). There are mainly two types of connectome to analyze fMRI: the Functional Connectome (FC) and the Effective Connectome (EC). FC is typically derived as the correlation between fMRI time-series from different brain regions, while EC is derived by fitting the measurement time-series to the Dynamical Causal Model (DCM) described by a system of Ordinary Differential Equations (ODEs). FC is typically easier to compute yet can not reveal the causal relations among brain regions; EC reveals the causal relations yet is much harder to compute and is more sensitive to observation noise. Therefore, this dissertation aims to propose a generic framework for estimation of EC, and identify ASD from fMRI based on EC. First, we propose the Model Driven Learning Framework (MDL) for parameter estimation in the continuous models. MDL iteratively performs three steps: 1) forward simulation according to prior knowledge of the model, 2) backward pass to derive the gradient of parameters, 3) update of parameters based on gradient information. We derive various methods to solve each step in MDL. Specifically, for step 2), we identify the inaccuracy of existing gradient estimation methods for continuous time models (e.g. ODEs): the adjoint method has numerical errors in reverse-mode integration; the naive method suffers from a redundantly deep computation graph. We propose a series of new methods which guarantee the numerical accuracy with a low memory cost. For step 3), we propose the AdaBelief optimizer, which is a generic first-order adaptive optimizer that simultaneously achieves fast convergence, good generalization and training stability. Furthermore, we show that an asynchronous version of AdaBelief achieves provably weaker convergence condition and faster convergence rate. We show that our MDL significantly accelerates the fitting of DCM and estimation of EC. To deal with the limited data and improve generalization of the classifier, we propose the Surrogate Gap Guided Sharpness-Aware Minimization (GSAM). GSAM is based on the observation that poor generalization often comes with a sharp loss surface of the model, and improves generalization by jointly minimizing the training loss and the curvature of the loss surface. Finally, we apply the proposed MDL to estimate whole-brain EC for fMRI, and performed group comparison to identify FC and EC edges that are related to ASD. Next, we apply the estimated EC for the identification of ASD. Specifically, we conducted experiments with both resting-state fMRI and task fMRI data, and compare the predictive power of FC and EC in both cases. Furthermore, we apply GSAM to further improve the generalization performance, which significantly improves the classification performance and reduces the dominant eigenvalue of the Hessian of the network. In summary, we apply the proposed framework for effective connectome analysis, and improve the identification of ASD from fMRI data

Is One Hyperparameter Optimizer Enough?

Hyperparameter tuning is the black art of automatically finding a good combination of control parameters for a data miner. While widely applied in empirical Software Engineering, there has not been much discussion on which hyperparameter tuner is best for software analytics. To address this gap in the literature, this paper applied a range of hyperparameter optimizers (grid search, random search, differential evolution, and Bayesian optimization) to defect prediction problem. Surprisingly, no hyperparameter optimizer was observed to be `best' and, for one of the two evaluation measures studied here (F-measure), hyperparameter optimization, in 50\% cases, was no better than using default configurations. We conclude that hyperparameter optimization is more nuanced than previously believed. While such optimization can certainly lead to large improvements in the performance of classifiers used in software analytics, it remains to be seen which specific optimizers should be applied to a new dataset.Comment: 7 pages, 2 columns, accepted for SWAN1

Particle swarm algorithm with adaptive constraint handling and integrated surrogate model for the management of petroleum fields

This paper deals with the development of effective techniques to automatically obtain the optimum management of petroleum fields aiming to increase the oil production during a given concession period of exploration. The optimization formulations of such a problem turn out to be highly multimodal, and may involve constraints. In this paper, we develop a robust particle swarm algorithm coupled with a novel adaptive constraint-handling technique to search for the global optimum of these formulations. However, this is a population-based method, which therefore requires a high number of evaluations of an objective function. Since the performance evaluation of a given management scheme requires a computationally expensive high-fidelity simulation, it is not practicable to use it directly to guide the search. In order to overcome this drawback, a Kriging surrogate model is used, which is trained offline via evaluations of a High-Fidelity simulator on a number of sample points. The optimizer then seeks the optimum of the surrogate model

Fast calculation of multiobjective probability of improvement and expected improvement criteria for Pareto optimization

The use of surrogate based optimization (SBO) is widely spread in engineering design to reduce the number of computational expensive simulations. However, "real-world" problems often consist of multiple, conflicting objectives leading to a set of competitive solutions (the Pareto front). The objectives are often aggregated into a single cost function to reduce the computational cost, though a better approach is to use multiobjective optimization methods to directly identify a set of Pareto-optimal solutions, which can be used by the designer to make more efficient design decisions (instead of weighting and aggregating the costs upfront). Most of the work in multiobjective optimization is focused on multiobjective evolutionary algorithms (MOEAs). While MOEAs are well-suited to handle large, intractable design spaces, they typically require thousands of expensive simulations, which is prohibitively expensive for the problems under study. Therefore, the use of surrogate models in multiobjective optimization, denoted as multiobjective surrogate-based optimization, may prove to be even more worthwhile than SBO methods to expedite the optimization of computational expensive systems. In this paper, the authors propose the efficient multiobjective optimization (EMO) algorithm which uses Kriging models and multiobjective versions of the probability of improvement and expected improvement criteria to identify the Pareto front with a minimal number of expensive simulations. The EMO algorithm is applied on multiple standard benchmark problems and compared against the well-known NSGA-II, SPEA2 and SMS-EMOA multiobjective optimization methods

Data Driven Surrogate Based Optimization in the Problem Solving Environment WBCSim

Large scale, multidisciplinary, engineering designs are always difficult due to the complexity and dimensionality of these problems. Direct coupling between the analysis codes and the optimization routines can be prohibitively time consuming due to the complexity of the underlying simulation codes. One way of tackling this problem is by constructing computationally cheap(er) approximations of the expensive simulations, that mimic the behavior of the simulation model as closely as possible. This paper presents a data driven, surrogate based optimization algorithm that uses a trust region based sequential approximate optimization (SAO) framework and a statistical sampling approach based on design of experiment (DOE) arrays. The algorithm is implemented using techniques from two packages—SURFPACK and SHEPPACK that provide a collection of approximation algorithms to build the surrogates and three different DOE techniques—full factorial (FF), Latin hypercube sampling (LHS), and central composite design (CCD)—are used to train the surrogates. The results are compared with the optimization results obtained by directly coupling an optimizer with the simulation code. The biggest concern in using the SAO framework based on statistical sampling is the generation of the required database. As the number of design variables grows, the computational cost of generating the required database grows rapidly. A data driven approach is proposed to tackle this situation, where the trick is to run the expensive simulation if and only if a nearby data point does not exist in the cumulatively growing database. Over time the database matures and is enriched as more and more optimizations are performed. Results show that the proposed methodology dramatically reduces the total number of calls to the expensive simulation runs during the optimization process