Optimal Design of Validation Experiments for Calibration and Validation of Complex Numerical Models

As prediction of the performance and behavior of complex engineering systems shifts from a primarily empirical-based approach to the use of complex physics-based numerical models, the role of experimentation is evolving to calibrate, validate, and quantify uncertainty of the numerical models. Oftentimes, these experiments are expensive, placing importance on selecting experimental settings to efficiently calibrate the numerical model with a limited number of experiments. The aim of this thesis is to reduce the experimental resources required to reach predictive maturity in complex numerical models by (i) aiding experimenters in determining the optimal settings for experiments, and (ii) aiding the model developers in assessing the predictive maturity of numerical models through a new, more refined coverage metric. Numerical model predictions entail uncertainties, primarily caused by imprecisely known input parameter values and biases, primarily caused by simplifications and idealizations in the model. Hence, calibration of numerical models involves not only updating of parameter values but also inferring the discrepancy bias, or empirically trained error model. Training of this error model throughout the domain of applicability becomes possible when experiments conducted at varying settings are available. Of course, for the trained discrepancy bias to be meaningful and a numerical model to be predictively mature, the validation experiments must sufficiently cover the operational domain. Otherwise, poor training of the discrepancy bias and overconfidence in model predictions may result. Thus, coverage metrics are used to quantify the ability of a set of validation experiments to represent an entire operation domain. This thesis is composed of two peer-reviewed journal articles. The first article focuses on the optimal design of validation experiments. The ability to improve the predictive maturity of a plasticity material model is assessed for several index-based and distance-based batch sequential design selection criteria through a detailed analysis of discrepancy bias and coverage. Furthermore, the effect of experimental uncertainty, complexity of discrepancy bias, and initial experimental settings on the performance of each criterion is evaluated. Lastly, a technique that integrates index-based and distance-based selection criteria to both exploit the available knowledge regarding the discrepancy bias and explore the operational domain is evaluated. This article is published in Structural and Multidisciplinary Optimization in 2013. The second article is focused on developing a coverage metric. Four characteristics of an exemplar coverage metric are identified and the ability of coverage metrics from the literature to satisfy the four criteria is evaluated. No existing coverage metric is determined to satisfy all four criteria. As a solution, a new coverage metric is proposed which exhibits satisfactory performance in all four criteria. The performance of the proposed coverage metric is compared to the existing coverage metrics using an application to the plasticity material model as well as a high-dimensional Rosenbrock function. This article is published in Mechanical Systems and Signal Processing in 2014

Resource Allocation Framework: Validation of Numerical Models of Complex Engineering Systems against Physical Experiments

An increasing reliance on complex numerical simulations for high consequence decision making is the motivation for experiment-based validation and uncertainty quantification to assess, and when needed, to improve the predictive capabilities of numerical models. Uncertainties and biases in model predictions can be reduced by taking two distinct actions: (i) increasing the number of experiments in the model calibration process, and/or (ii) improving the physics sophistication of the numerical model. Therefore, decision makers must select between further code development and experimentation while allocating the finite amount of available resources. This dissertation presents a novel framework to assist in this selection between experimentation and code development for model validation strictly from the perspective of predictive capability. The reduction and convergence of discrepancy bias between model prediction and observation, computed using a suitable convergence metric, play a key role in the conceptual formulation of the framework. The proposed framework is demonstrated using two non-trivial case study applications on the Preston-Tonks-Wallace (PTW) code, which is a continuum-based plasticity approach to modeling metals, and the ViscoPlastic Self-Consistent (VPSC) code which is a mesoscopic plasticity approach to modeling crystalline materials. Results show that the developed resource allocation framework is effective and efficient in path selection (i.e. experimentation and/or code development) resulting in a reduction in both model uncertainties and discrepancy bias. The framework developed herein goes beyond path selection in the validation of numerical models by providing a methodology for the prioritization of optimal experimental settings and an algorithm for prioritization of code development. If the path selection algorithm selects the experimental path, optimal selection of the settings at which these physical experiments are conducted as well as the sequence of these experiments is vital to maximize the gain in predictive capability of a model. The Batch Sequential Design (BSD) is a methodology utilized in this work to achieve the goal of selecting the optimal experimental settings. A new BSD selection criterion, Coverage Augmented Expected Improvement for Predictive Stability (C-EIPS), is developed to minimize the maximum reduction in the model discrepancy bias and coverage of the experiments within the domain of applicability. The functional form of the new criterion, C-EIPS, is demonstrated to outperform its predecessor, the EIPS criterion, and the distance-based criterion when discrepancy bias is high and coverage is low, while exhibiting a comparable performance to the distance-based criterion in efficiently maximizing the predictive capability of the VPSC model as discrepancy decreases and coverage increases. If the path selection algorithm selects the code development path, the developed framework provides an algorithm for the prioritization of code development efforts. In coupled systems, the predictive accuracy of the simulation hinges on the accuracy of individual constituent models. Potential improvement in the predictive accuracy of the simulation that can be gained through improving a constituent model depends not only on the relative importance, but also on the inherent uncertainty and inaccuracy of that particular constituent. As such, a unique and quantitative code prioritization index (CPI) is proposed to accomplish the task of prioritizing code development efforts, and its application is demonstrated on a case study of a steel frame with semi-rigid connections. Findings show that the CPI is effective in identifying the most critical constituent of the coupled system, whose improvement leads to the highest overall enhancement of the predictive capability of the coupled model

PREDICTIVE MATURITY OF INEXACT AND UNCERTAIN STRONGLY COUPLED NUMERICAL MODELS

The Computer simulations are commonly used to predict the response of complex systems in many branches of engineering and science. These computer simulations involve the theoretical foundation, numerical modeling and supporting experimental data, all of which contain their associated errors. Furthermore, real-world problems are generally complex in nature, in which each phenomenon is described by the respective constituent models representing different physics and/or scales. The interactions between such constituents are typically complex in nature, such that the outputs of a particular constituent may be the inputs for one or more constituents. Thus, the natural question then arises concerning the validity of these complex computer model predictions, especially in cases where these models are executed in support of high-consequence decision making. The overall accuracy and precision of the coupled system is then determined by the accuracy and precision of both the constituents and the coupling interface. Each constituent model has its own uncertainty and bias error. Furthermore, the coupling interface also brings in a similar spectrum of uncertainties and bias errors due to unavoidably inexact and incomplete data transfer between the constituents. This dissertation contributes to the established knowledge of partitioned analysis by investigating the numerical uncertainties, validation and uncertainty quantification of strongly coupled inexact and uncertain models. The importance of this study lies in the urgent need for gaining a better understanding of the simulations of coupled systems, such as those in multi-scale and multi-physics applications, and to identify the limitations due to uncertainty and bias errors in these models

Experiment-Based Validation and Uncertainty Quantification of Partitioned Models: Improving Predictive Capability of Multi-Scale Plasticity Models

Partitioned analysis involves coupling of constituent models that resolve their own scales or physics by exchanging inputs and outputs in an iterative manner. Through partitioning, simulations of complex physical systems are becoming evermore present in scientific modeling, making Verification and Validation of partitioned models for the purpose of quantifying the predictive capability of their simulations increasingly important. Parameterization of the constituent models as well as the coupling interface requires a significant amount of information about the system, which is often imprecisely known. Consequently, uncertainties as well as biases in constituent models and their interface lead to concerns about the accumulation and compensation of these uncertainties and errors during the iterative procedures of partitioned analysis. Furthermore, partitioned analysis relies on the availability of reliable constituent models for each component of a system. When a constituent is unavailable, assumptions must be made to represent the coupling relationship, often through uncertain parameters that are then calibrated. This dissertation contributes to the field of computational modeling by presenting novel methods that take advantage of the transparency of partitioned analysis to compare constituent models with separate-effect experiments (measurements contained to the constituent domain) and coupled models with integral-effect experiments (measurements capturing behavior of the full system). The methods developed herein focus on these two types of experiments seeking to maximize the information that can be gained from each, thus progressing our capability to assess and improve the predictive capability of partitioned models through inverse analysis. The importance of this study stems from the need to make coupled models available for widespread use for predicting the behavior of complex systems with confidence to support decision-making in high-risk scenarios. Methods proposed herein address the challenges currently limiting the predictive capability of coupled models through a focused analysis with available experiments. Bias-corrected partitioned analysis takes advantage of separate-effect experiments to reduce parametric uncertainty and quantify systematic bias at the constituent level followed by an integration of bias-correction to the coupling framework, thus ‘correcting’ the constituent model during coupling iterations and preventing the accumulation of errors due to the final predictions. Model bias is the result of assumptions made in the modeling process, often due to lack of understanding of the underlying physics. Such is the case when a constituent model of a system component is entirely unavailable and cannot be developed due to lack of knowledge. However, if this constituent model were to be available and coupled to existing models of the other system components, bias in the coupled system would be reduced. This dissertation proposes a novel statistical inference method for developing empirical constituent models where integral-effect experiments are used to infer relationships missing from system models. Thus, the proposed inverse analysis may be implemented to infer underlying coupled relationships, not only improving the predictive capability of models by producing empirical constituents to allow for coupling, but also advancing our fundamental understanding of dependencies in the coupled system. Throughout this dissertation, the applicability and feasibility of the proposed methods are demonstrated with advanced multi-scale and multi-physics material models simulating complex material behaviors under extreme loading conditions, thus specifically contributing advancements to the material modeling community

Representation learning for uncertainty-aware clinical decision support

Over the last decade, there has been an increasing trend towards digitalization in healthcare, where a growing amount of patient data is collected and stored electronically. These recorded data are known as electronic health records. They are the basis for state-of-the-art research on clinical decision support so that better patient care can be delivered with the help of advanced analytical techniques like machine learning. Among various technical fields in machine learning, representation learning is about learning good representations from raw data to extract useful information for downstream prediction tasks. Deep learning, a crucial class of methods in representation learning, has achieved great success in many fields such as computer vision and natural language processing. These technical breakthroughs would presumably further advance the research and development of data analytics in healthcare. This thesis addresses clinically relevant research questions by developing algorithms based on state-of-the-art representation learning techniques. When a patient visits the hospital, a physician will suggest a treatment in a deterministic manner. Meanwhile, uncertainty comes into play when the past statistics of treatment decisions from various physicians are analyzed, as they would possibly suggest different treatments, depending on their training and experiences. The uncertainty in clinical decision-making processes is the focus of this thesis. The models developed for supporting these processes will therefore have a probabilistic nature. More specifically, the predictions are predictive distributions in regression tasks and probability distributions over, e.g., different treatment decisions, in classification tasks. The first part of the thesis is concerned with prescriptive analytics to provide treatment recommendations. Apart from patient information and treatment decisions, the outcome after the respective treatment is included in learning treatment suggestions. The problem setting is known as learning individualized treatment rules and is formulated as a contextual bandit problem. A general framework for learning individualized treatment rules using data from observational studies is presented based on state-of-the-art representation learning techniques. From various offline evaluation methods, it is shown that the treatment policy in our proposed framework can demonstrate better performance than both physicians and competitive baselines. Subsequently, the uncertainty-aware regression models in diagnostic and predictive analytics are studied. Uncertainty-aware deep kernel learning models are proposed, which allow the estimation of the predictive uncertainty by a pipeline of neural networks and a sparse Gaussian process. By considering the input data structure, respective models are developed for diagnostic medical image data and sequential electronic health records. Various pre-training methods from representation learning are adapted to investigate their impacts on the proposed models. Through extensive experiments, it is shown that the proposed models delivered better performance than common architectures in most cases. More importantly, uncertainty-awareness of the proposed models is illustrated by systematically expressing higher confidence in more accurate predictions and less confidence in less accurate ones. The last part of the thesis is about missing data imputation in descriptive analytics, which provides essential evidence for subsequent decision-making processes. Rather than traditional mean and median imputation, a more advanced solution based on generative adversarial networks is proposed. The presented method takes the categorical nature of patient features into consideration, which enables the stabilization of the adversarial training. It is shown that the proposed method can better improve the predictive accuracy compared to traditional imputation baselines