Engineering design applications of surrogate-assisted optimization techniques

The construction of models aimed at learning the behaviour of a system whose responses to inputs are expensive to measure is a branch of statistical science that has been around for a very long time. Geostatistics has pioneered a drive over the last half century towards a better understanding of the accuracy of such ‘surrogate’ models of the expensive function. Of particular interest to us here are some of the even more recent advances related to exploiting such formulations in an optimization context. While the classic goal of the modelling process has been to achieve a uniform prediction accuracy across the domain, an economical optimization process may aim to bias the distribution of the learning budget towards promising basins of attraction. This can only happen, of course, at the expense of the global exploration of the space and thus finding the best balance may be viewed as an optimization problem in itself. We examine here a selection of the state of-the-art solutions to this type of balancing exercise through the prism of several simple, illustrative problems, followed by two ‘real world’ applications: the design of a regional airliner wing and the multi-objective search for a low environmental impact hous

Transfer Learning for Multi-surrogate-model Optimization

Surrogate-model-based optimization is widely used to solve black-box optimization problems if the evaluation of a target system is expensive. However, when the optimization budget is limited to a single or several evaluations, surrogate-model-based optimization may not perform well due to the lack of knowledge about the search space. In this case, transfer learning helps to get a good optimization result due to the usage of experience from the previous optimization runs. And if the budget is not strictly limited, transfer learning is capable of improving the final results of black-box optimization. The recent work in surrogate-model-based optimization showed that using multiple surrogates (i.e., applying multi-surrogate-model optimization) can be extremely efficient in complex search spaces. The main assumption of this thesis suggests that transfer learning can further improve the quality of multi-surrogate-model optimization. However, to the best of our knowledge, there exist no approaches to transfer learning in the multi-surrogate-model context yet. In this thesis, we propose an approach to transfer learning for multi-surrogate-model optimization. It encompasses an improved method of defining the expediency of knowledge transfer, adapted multi-surrogate-model recommendation, multi-task learning parameter tuning, and few-shot learning techniques. We evaluated the proposed approach with a set of algorithm selection and parameter setting problems, comprising mathematical functions optimization and the traveling salesman problem, as well as random forest hyperparameter tuning over OpenML datasets. The evaluation shows that the proposed approach helps to improve the quality delivered by multi-surrogate-model optimization and ensures getting good optimization results even under a strictly limited budget.:1 Introduction 1.1 Motivation 1.2 Research objective 1.3 Solution overview 1.4 Thesis structure 2 Background 2.1 Optimization problems 2.2 From single- to multi-surrogate-model optimization 2.2.1 Classical surrogate-model-based optimization 2.2.2 The purpose of multi-surrogate-model optimization 2.2.3 BRISE 2.5.0: Multi-surrogate-model-based software product line for parameter tuning 2.3 Transfer learning 2.3.1 Definition and purpose of transfer learning 2.4 Summary of the Background 3 Related work 3.1 Questions to transfer learning 3.2 When to transfer: Existing approaches to determining the expediency of knowledge transfer 3.2.1 Meta-features-based approaches 3.2.2 Surrogate-model-based similarity 3.2.3 Relative landmarks-based approaches 3.2.4 Sampling landmarks-based approaches 3.2.5 Similarity threshold problem 3.3 What to transfer: Existing approaches to knowledge transfer 3.3.1 Ensemble learning 3.3.2 Search space pruning 3.3.3 Multi-task learning 3.3.4 Surrogate model recommendation 3.3.5 Few-shot learning 3.3.6 Other approaches to transferring knowledge 3.4 How to transfer (discussion): Peculiarities and required design decisions for the TL implementation in multi-surrogate-model setup 3.4.1 Peculiarities of model recommendation in multi-surrogate-model setup 3.4.2 Required design decisions in multi-task learning 3.4.3 Few-shot learning problem 3.5 Summary of the related work analysis 4 Transfer learning for multi-surrogate-model optimization 4.1 Expediency of knowledge transfer 4.1.1 Experiments’ similarity definition as a variability point 4.1.2 Clustering to filter the most suitable experiments 4.2 Dynamic model recommendation in multi-surrogate-model setup 4.2.1 Variable recommendation granularity 4.2.2 Model recommendation by time and performance criteria 4.3 Multi-task learning 4.4 Implementation of the proposed concept 4.5 Conclusion of the proposed concept 5 Evaluation 5.1 Benchmark suite 5.1.1 APSP for the meta-heuristics 5.1.2 Hyperparameter optimization of the Random Forest algorithm 5.2 Environment setup 5.3 Evaluation plan 5.4 Baseline evaluation 5.5 Meta-tuning for a multi-task learning approach 5.5.1 Revealing the dependencies between the parameters of multi-task learning and its performance 5.5.2 Multi-task learning performance with the best found parameters 5.6 Expediency determination approach 5.6.1 Expediency determination as a variability point 5.6.2 Flexible number of the most similar experiments with the help of clustering 5.6.3 Influence of the number of initial samples on the quality of expediency determination 5.7 Multi-surrogate-model recommendation 5.8 Few-shot learning 5.8.1 Transfer of the built surrogate models’ combination 5.8.2 Transfer of the best configuration 5.8.3 Transfer from different experiment instances 5.9 Summary of the evaluation results 6 Conclusion and Future wor

Multi-objective network optimization: models, methods, and applications

There can be an array of planning objectives to consider when identifying alternatives for using, modifying, or restoring natural or built environments. In this respect, multi-objective network optimization models can provide decision support to both managers and users of the system. While there can be an infinite number of feasible solutions to any multi-objective optimization problem in large networks (e.g., urban transportation systems), the efficient ones are usually more desirable in the decision-making process. However, identification of efficient solutions can be challenging in practical applications. To address this issue, this dissertation details mathematical formulations and solution algorithms for a range of real-world planning problems in the context of intelligent transportation systems, vehicle routing problem, natural conservation and landscape connectivity. While the combination of objectives being optimized is unique for each application, the underlying phenomena involves modeling movement between origins and destinations of a networked system. To demonstrate the type of insights that can be achieved using these modeling approaches, the location and number of times solutions appear in different realizations of system and given different solution approaches (e.g., exact and approximate methods) are visualized on network using a commercial geographic information system

A Multi-Agent Architecture for the Design of Hierarchical Interval Type-2 Beta Fuzzy System

This paper presents a new methodology for building and evolving hierarchical fuzzy systems. For the system design, a tree-based encoding method is adopted to hierarchically link low dimensional fuzzy systems. Such tree structural representation has by nature a flexible design offering more adjustable and modifiable structures. The proposed hierarchical structure employs a type-2 beta fuzzy system to cope with the faced uncertainties, and the resulting system is called the Hierarchical Interval Type-2 Beta Fuzzy System (HT2BFS). For the system optimization, two main tasks of structure learning and parameter tuning are applied. The structure learning phase aims to evolve and learn the structures of a population of HT2BFS in a multiobjective context taking into account the optimization of both the accuracy and the interpretability metrics. The parameter tuning phase is applied to refine and adjust the parameters of the system. To accomplish these two tasks in the most optimal and faster way, we further employ a multi-agent architecture to provide both a distributed and a cooperative management of the optimization tasks. Agents are divided into two different types based on their functions: a structure agent and a parameter agent. The main function of the structure agent is to perform a multi-objective evolutionary structure learning step by means of the Multi-Objective Immune Programming algorithm (MOIP). The parameter agents have the function of managing different hierarchical structures simultaneously to refine their parameters by means of the Hybrid Harmony Search algorithm (HHS). In this architecture, agents use cooperation and communication concepts to create high-performance HT2BFSs. The performance of the proposed system is evaluated by several comparisons with various state of art approaches on noise-free and noisy time series prediction data sets and regression problems. The results clearly demonstrate a great improvement in the accuracy rate, the convergence speed and the number of used rules as compared with other existing approaches

Simultaneous Optimal Uncertainty Apportionment and Robust Design Optimization of Systems Governed by Ordinary Differential Equations

The inclusion of uncertainty in design is of paramount practical importance because all real-life systems are affected by it. Designs that ignore uncertainty often lead to poor robustness, suboptimal performance, and higher build costs. Treatment of small geometric uncertainty in the context of manufacturing tolerances is a well studied topic. Traditional sequential design methodologies have recently been replaced by concurrent optimal design methodologies where optimal system parameters are simultaneously determined along with optimally allocated tolerances; this allows to reduce manufacturing costs while increasing performance. However, the state of the art approaches remain limited in that they can only treat geometric related uncertainties restricted to be small in magnitude. This work proposes a novel framework to perform robust design optimization concurrently with optimal uncertainty apportionment for dynamical systems governed by ordinary differential equations. The proposed framework considerably expands the capabilities of contemporary methods by enabling the treatment of both geometric and non-geometric uncertainties in a unified manner. Additionally, uncertainties are allowed to be large in magnitude and the governing constitutive relations may be highly nonlinear. In the proposed framework, uncertainties are modeled using Generalized Polynomial Chaos and are solved quantitatively using a least-square collocation method. The computational efficiency of this approach allows statistical moments of the uncertain system to be explicitly included in the optimization-based design process. The framework formulates design problems as constrained multi-objective optimization problems, thus enabling the characterization of a Pareto optimal trade-off curve that is off-set from the traditional deterministic optimal trade-off curve. The Pareto off-set is shown to be a result of the additional statistical moment information formulated in the objective and constraint relations that account for the system uncertainties. Therefore, the Pareto trade-off curve from the new framework characterizes the entire family of systems within the probability space; consequently, designers are able to produce robust and optimally performing systems at an optimal manufacturing cost. A kinematic tolerance analysis case-study is presented first to illustrate how the proposed methodology can be applied to treat geometric tolerances. A nonlinear vehicle suspension design problem, subject to parametric uncertainty, illustrates the capability of the new framework to produce an optimal design at an optimal manufacturing cost, accounting for the entire family of systems within the associated probability space. This case-study highlights the general nature of the new framework which is capable of optimally allocating uncertainties of multiple types and with large magnitudes in a single calculation

Interoperability and computational framework for simulating open channel hydraulics: application to sensitivity analysis and calibration of Gironde Estuary model

Water resource management is of crucial societal and economic importance, requiring a strong capacity for anticipating environmental change. Progress in physical process knowledge, numerical methods and computational power, allows us to address hydro-environmental problems of growing complexity. Modeling of river and marine flows is no exception. With the increase in IT resources, environmental modeling is evolving to meet the challenges of complex real-world problems. This paper presents a new distributed Application Programming Interface (API) of the open source TELEMAC-MASCARET system to run hydro-environmental simulations with the help of the interoperability concept. Use of the API encourages and facilitates the combination of worldwide reference environmental libraries with the hydro-informatic system. Consequently, the objective of the paper is to promote the interoperability concept for studies dealing with such issues as uncertainty propagation, global sensitivity analysis, optimization, multi-physics or multi-dimensional coupling. To illustrate the capability of the API, an operational problem for improving the navigation capacity of the Gironde Estuary is presented. The API potential is demonstrated in a re-calibration context. The API is used for a multivariate sensitivity analysis to quickly reveal the most influential parameters which can then be optimally calibrated with the help of a data assimilation technique