State-of-the-Art and Comparative Review of Adaptive Sampling Methods for Kriging

Metamodels aim to approximate characteristics of functions or systems from the knowledge extracted on only a finite number of samples. In recent years kriging has emerged as a widely applied metamodeling technique for resource-intensive computational experiments. However its prediction quality is highly dependent on the size and distribution of the given training points. Hence, in order to build proficient kriging models with as few samples as possible adaptive sampling strategies have gained considerable attention. These techniques aim to find pertinent points in an iterative manner based on information extracted from the current metamodel. A review of adaptive schemes for kriging proposed in the literature is presented in this article. The objective is to provide the reader with an overview of the main principles of adaptive techniques, and insightful details to pertinently employ available tools depending on the application at hand. In this context commonly applied strategies are compared with regards to their characteristics and approximation capabilities. In light of these experiments, it is found that the success of a scheme depends on the features of a specific problem and the goal of the analysis. In order to facilitate the entry into adaptive sampling a guide is provided. All experiments described herein are replicable using a provided open source toolbox. © 2020, The Author(s)

Universal Prediction Distribution for Surrogate Models

International audienceThe use of surrogate models instead of computationally expensive simulation codes is very convenient in engineering. Roughly speaking, there are two kinds of surrogate models: the deterministic and the probabilistic ones. These last are generally based on Gaussian assumptions. The main advantage of probabilistic approach is that it provides a measure of uncertainty associated with the surrogate model in the whole space. This uncertainty is an efficient tool to construct strategies for various problems such as prediction enhancement, optimization or inversion.In this paper, we propose a universal method to define a measure of uncertainty suitable for any surrogate model either deterministic or probabilistic. It relies on Cross-Validation (CV) sub-models predictions. This empirical distribution may be computed in much more general frames than the Gaussian one. So that it is called the Universal Prediction distribution (UP distribution).It allows the definition of many sampling criteria. We give and study adaptive sampling techniques for global refinement and an extension of the so-called Efficient Global Optimization (EGO) algorithm. We also discuss the use of the UP distribution for inversion problems. The performances of these new algorithms are studied both on toys models and on an engineering design problem

Maximin Designs for Computer Experiments.

Decision processes are nowadays often facilitated by simulation tools. In the field of engineering, for example, such tools are used to simulate the behavior of products and processes. Simulation runs, however, are often very time-consuming, and, hence, the number of simulation runs allowed is limited in practice. The problem then is to determine which simulation runs to perform such that the maximal amount of information about the product or process is obtained. This problem is addressed in the first part of the thesis. It is proposed to use so-called maximin Latin hypercube designs and many new results for this class of designs are obtained. In the second part, the case of multiple interrelated simulation tools is considered and a framework to deal with such tools is introduced. Important steps in this framework are the construction and the use of coordination methods and of nested designs in order to control the dependencies present between the various simulation tools

Machine Learning of Hazard Model Simulations for use in Probabilistic Risk Assessments

This study explored the use of machine learning to generate metamodel approximations of a physics-based fire hazard model called Consolidated Fire and Smoke Transport (CFAST). The motivation to generate accurate and efficient metamodels is to improve modeling realism in probabilistic risk assessments where computational burden has prevented broader application of high fidelity codes. The process involved scenario definition, generating training data by iteratively running the hazard model over a range of input space, exploratory data analysis and feature selection, an initial testing of a broad set of metamodel types, and finally metamodel selection and tuning. The study identified several factors that should be considered when metamodeling a physics-based computer code. First, the input space should be limited to a manageable scale and number of parameters; otherwise generating sufficient training data becomes infeasible. Second, there is a relationship between the physics being characterized and the metamodel types that will successfully mimic those physics. Finally, metamodel accuracy and efficiency must be balanced against initial development costs. Once developed, trained metamodels are portable and can be applied by many users over a range of modeling conditions. The Idaho National Laboratory software called RAVEN was used to facilitate the analysis. Twenty five (25) metamodel types were investigated for their potential to mimic CFAST-calculated maximum upper layer temperature and its timing. Linear metamodels struggled to predict with accuracy because the physics of fire are non-linear. k-nearest neighbor (kNN) model tuning generated a k =4 model that fit the vast majority of CFAST calculations within 10% for both maximum upper layer temperature and its timing. This model showed good generalization with use of 10-fold cross validation. The resulting kNN model was compared to algebraic models typically used in fire probabilistic risk assessments. The algebraic models were generally conservative relative to CFAST; whereas the kNN model closely mimicked CFAST. This illustrates the potential of metamodels to improve modeling realism over the simpler models often selected for computational feasibility. While the kNN metamodel is a simplification of the higher fidelity CFAST code, the error introduced is quantifiable and can be explicitly considered in applications of the metamodel

Optimal Design of Validation Experiments for Calibration and Validation of Complex Numerical Models

As prediction of the performance and behavior of complex engineering systems shifts from a primarily empirical-based approach to the use of complex physics-based numerical models, the role of experimentation is evolving to calibrate, validate, and quantify uncertainty of the numerical models. Oftentimes, these experiments are expensive, placing importance on selecting experimental settings to efficiently calibrate the numerical model with a limited number of experiments. The aim of this thesis is to reduce the experimental resources required to reach predictive maturity in complex numerical models by (i) aiding experimenters in determining the optimal settings for experiments, and (ii) aiding the model developers in assessing the predictive maturity of numerical models through a new, more refined coverage metric. Numerical model predictions entail uncertainties, primarily caused by imprecisely known input parameter values and biases, primarily caused by simplifications and idealizations in the model. Hence, calibration of numerical models involves not only updating of parameter values but also inferring the discrepancy bias, or empirically trained error model. Training of this error model throughout the domain of applicability becomes possible when experiments conducted at varying settings are available. Of course, for the trained discrepancy bias to be meaningful and a numerical model to be predictively mature, the validation experiments must sufficiently cover the operational domain. Otherwise, poor training of the discrepancy bias and overconfidence in model predictions may result. Thus, coverage metrics are used to quantify the ability of a set of validation experiments to represent an entire operation domain. This thesis is composed of two peer-reviewed journal articles. The first article focuses on the optimal design of validation experiments. The ability to improve the predictive maturity of a plasticity material model is assessed for several index-based and distance-based batch sequential design selection criteria through a detailed analysis of discrepancy bias and coverage. Furthermore, the effect of experimental uncertainty, complexity of discrepancy bias, and initial experimental settings on the performance of each criterion is evaluated. Lastly, a technique that integrates index-based and distance-based selection criteria to both exploit the available knowledge regarding the discrepancy bias and explore the operational domain is evaluated. This article is published in Structural and Multidisciplinary Optimization in 2013. The second article is focused on developing a coverage metric. Four characteristics of an exemplar coverage metric are identified and the ability of coverage metrics from the literature to satisfy the four criteria is evaluated. No existing coverage metric is determined to satisfy all four criteria. As a solution, a new coverage metric is proposed which exhibits satisfactory performance in all four criteria. The performance of the proposed coverage metric is compared to the existing coverage metrics using an application to the plasticity material model as well as a high-dimensional Rosenbrock function. This article is published in Mechanical Systems and Signal Processing in 2014