Monotonicity-Preserving Bootstrapped Kriging Metamodels for Expensive Simulations

Kriging (Gaussian process, spatial correlation) metamodels approximate the Input/Output (I/O) functions implied by the underlying simulation models; such metamodels serve sensitivity analysis and optimization, especially for computationally expensive simulations. In practice, simulation analysts often know that the I/O function is monotonic. To obtain a Kriging metamodel that preserves this known shape, this article uses bootstrapping (or resampling). Parametric bootstrapping assuming normality may be used in deterministic simulation, but this article focuses on stochastic simulation (including discrete-event simulation) using distribution-free bootstrapping. In stochastic simulation, the analysts should simulate each input combination several times to obtain a more reliable average output per input combination. Nevertheless, this average still shows sampling variation, so the Kriging metamodel does not need to interpolate the average outputs. Bootstrapping provides a simple method for computing a noninterpolating Kriging model. This method may use standard Kriging software, such as the free Matlab toolbox called DACE. The method is illustrated through the M/M/1 simulation model with as outputs either the estimated mean or the estimated 90% quantile; both outputs are monotonic functions of the traffic rate, and have nonnormal distributions. The empirical results demonstrate that monotonicity-preserving bootstrapped Kriging may give higher probability of covering the true simulation output, without lengthening the confidence interval.Queues

Certified Monotonic Neural Networks

Learning monotonic models with respect to a subset of the inputs is a desirable feature to effectively address the fairness, interpretability, and generalization issues in practice. Existing methods for learning monotonic neural networks either require specifically designed model structures to ensure monotonicity, which can be too restrictive/complicated, or enforce monotonicity by adjusting the learning process, which cannot provably guarantee the learned model is monotonic on selected features. In this work, we propose to certify the monotonicity of the general piece-wise linear neural networks by solving a mixed integer linear programming problem.This provides a new general approach for learning monotonic neural networks with arbitrary model structures. Our method allows us to train neural networks with heuristic monotonicity regularizations, and we can gradually increase the regularization magnitude until the learned network is certified monotonic. Compared to prior works, our approach does not require human-designed constraints on the weight space and also yields more accurate approximation. Empirical studies on various datasets demonstrate the efficiency of our approach over the state-of-the-art methods, such as Deep Lattice Networks

Statistical strategies for constructing health risk models with multiple pollutants and their interactions: possible choices and comparisons

Abstract Background As public awareness of consequences of environmental exposures has grown, estimating the adverse health effects due to simultaneous exposure to multiple pollutants is an important topic to explore. The challenges of evaluating the health impacts of environmental factors in a multipollutant model include, but are not limited to: identification of the most critical components of the pollutant mixture, examination of potential interaction effects, and attribution of health effects to individual pollutants in the presence of multicollinearity. Methods In this paper, we reviewed five methods available in the statistical literature that are potentially helpful for constructing multipollutant models. We conducted a simulation study and presented two data examples to assess the performance of these methods on feature selection, effect estimation and interaction identification using both cross-sectional and time-series designs. We also proposed and evaluated a two-step strategy employing an initial screening by a tree-based method followed by further dimension reduction/variable selection by the aforementioned five approaches at the second step. Results Among the five methods, least absolute shrinkage and selection operator regression performs well in general for identifying important exposures, but will yield biased estimates and slightly larger model dimension given many correlated candidate exposures and modest sample size. Bayesian model averaging, and supervised principal component analysis are also useful in variable selection when there is a moderately strong exposure-response association. Substantial improvements on reducing model dimension and identifying important variables have been observed for all the five statistical methods using the two-step modeling strategy when the number of candidate variables is large. Conclusions There is no uniform dominance of one method across all simulation scenarios and all criteria. The performances differ according to the nature of the response variable, the sample size, the number of pollutants involved, and the strength of exposure-response association/interaction. However, the two-step modeling strategy proposed here is potentially applicable under a multipollutant framework with many covariates by taking advantage of both the screening feature of an initial tree-based method and dimension reduction/variable selection property of the subsequent method. The choice of the method should also depend on the goal of the study: risk prediction, effect estimation or screening for important predictors and their interactions.http://deepblue.lib.umich.edu/bitstream/2027.42/112386/1/12940_2013_Article_691.pd

Post-Processing Methods to Enforce Monotonic Constraints in Ant Colony Classification Algorithms

Most classification algorithms ignore existing domain knowledge during model construction, which can decrease the model's comprehensibility and increase the likelihood of model rejection due to users losing trust in the models they use. One approach to encapsulate this domain knowledge is monotonic constraints. This paper proposes new monotonic pruners to enforce monotonic constraints on models created by an existing ACO algorithm in a post-processing stage. We compare the effectiveness of the new pruners against an existing post-processing approach that also enforce constraints. Additionally, we also compare the effectiveness of both these post-processing procedures in isolation and in conjunction with favouring constraints in the learning phase. Our results show that our proposed pruners outperform the existing post-processing approach and the combination of favouring and enforcing constraints at different stages of the model construction process is the most effective solution

Monotone Models for Prediction in Data Mining.

This dissertation studies the incorporation of monotonicity constraints as a type of domain knowledge into a data mining process. Monotonicity constraints are enforced at two stages¿data preparation and data modeling. The main contributions of the research are a novel procedure to test the degree of monotonicity of a real data set, a greedy algorithm to transform non-monotone into monotone data, and extended and novel approaches for building monotone decision models. The results from simulation and real case studies show that enforcing monotonicity can considerably improve knowledge discovery and facilitate the decision-making process for end-users by deriving more accurate, stable and plausible decision models.

Constitutive Modelling of Non-Linear Isotropic Elasticity Using Deep Regression Neural Networks

Deep neural networks (DNNs) have emerged as a promising approach for constitutive modelling of advanced materials in computational mechanics. However, achieving physically realistic and stable numerical simulations with DNNs can be challenging, especially when dealing with large deformations, that can lead to non-convergence effects in the presence of local stretch/stress peaks. This PhD dissertation introduces a novel approach for data-driven modelling of non-linear compressible isotropic materials, focusing on predicting the large deformation response of 3D specimens. The proposed methodology formulates the underlying hyperelastic deformation problem in the principal space using principal stretches and principal stresses, in which the corresponding constitutive relation is captured by a deep neural network surrogate model. To ensure constitutive requirements of the model while preserving the robustness of underlying numerical solution schemes, several physics-motivated constraints are imposed on the architecture of the DNN, such as objectivity, growth condition, normalized condition, and Hill’s inequalities. Furthermore, the prediction phase utilizes a constitutive blending approach to overcome divergence in the Newton-Raphson process, which can occur when solving boundary value problems using the Finite Element Method. The work also presents a machine learning finite element pipeline for modelling non-linear compressible isotropic materials, involving determining automatically the hyperparameters, training, and integrating the ANN operator into the finite element solver using symbolic representation. The proposed formalism has been tested through numerical benchmarks, demonstrating its ability to describe non-trivial load-deformation trajectories of 3D test specimens accurately. Overall, the thesis presents a complete and general formalism for data-driven modelling of non-linear compressible isotropic materials that overcomes the limitations of existing approaches.9. Industry, innovation and infrastructur

Discovering Regression and Classification Rules with Monotonic Constraints Using Ant Colony Optimization

Data mining is a broad area that encompasses many different tasks from the supervised classification and regression tasks to unsupervised association rule mining and clustering. A first research thread in this thesis is the introduction of new Ant Colony Optimization (ACO)-based algorithms that tackle the regression task in data mining, exploring three different learning strategies: Iterative Rule Learning, Pittsburgh and Michigan strategies. The Iterative Rule Learning strategy constructs one rule at a time, where the best rule created by the ant colony is added to the rule list at each iteration, until a complete rule list is created. In the Michigan strategy, each ant constructs a single rule and from this collection of rules a niching algorithm combines the rules to create the final rule list. Finally, in the Pittsburgh strategy each ant constructs an entire rule list at each iteration, with the best list constructed by an ant in any iteration representing the final model. The most successful Pittsburgh-based Ant-Miner-Reg_PB algorithm, among the three variants, has been shown to be competitive against a well-known regression rule induction algorithm from the literature. The second research thread pursued involved incorporating existing domain knowledge to guide the construction of models as it is rare to find new domains that nothing is known about. One type of domain knowledge that occurs frequently in real world data-sets is monotonic constraints which capture increasing or decreasing trends within the data. In this thesis, monotonic constraints have been introduced into ACO-based rule induction algorithms for both classification and regression tasks. The enforcement of monotonic constraints has been implemented as a two step process. The first is a soft constraint preference in the model construction phase. This is followed by a hard constraint post-processing pruning suite to ensure the production of monotonic models. The new algorithms presented here have been shown to maintain and improve their predictive power when compared to non-monotonic rule induction algorithms

EFFECT OF COGNITIVE BIASES ON HUMAN UNDERSTANDING OF RULE-BASED MACHINE LEARNING MODELS

PhDThis thesis investigates to what extent do cognitive biases a ect human understanding of interpretable machine learning models, in particular of rules discovered from data. Twenty cognitive biases (illusions, e ects) are analysed in detail, including identi cation of possibly e ective debiasing techniques that can be adopted by designers of machine learning algorithms and software. This qualitative research is complemented by multiple experiments aimed to verify, whether, and to what extent, do selected cognitive biases in uence human understanding of actual rule learning results. Two experiments were performed, one focused on eliciting plausibility judgments for pairs of inductively learned rules, second experiment involved replication of the Linda experiment with crowdsourcing and two of its modi cations. Altogether nearly 3.000 human judgments were collected. We obtained empirical evidence for the insensitivity to sample size e ect. There is also limited evidence for the disjunction fallacy, misunderstanding of and , weak evidence e ect and availability heuristic. While there seems no universal approach for eliminating all the identi ed cognitive biases, it follows from our analysis that the e ect of many biases can be ameliorated by making rule-based models more concise. To this end, in the second part of thesis we propose a novel machine learning framework which postprocesses rules on the output of the seminal association rule classi cation algorithm CBA [Liu et al, 1998]. The framework uses original undiscretized numerical attributes to optimize the discovered association rules, re ning the boundaries of literals in the antecedent of the rules produced by CBA. Some rules as well as literals from the rules can consequently be removed, which makes the resulting classi er smaller. Benchmark of our approach on 22 UCI datasets shows average 53% decrease in the total size of the model as measured by the total number of conditions in all rules. Model accuracy remains on the same level as for CBA