Couckuyt, Ivo

Dhaene, Tom

Gorissen, Dirk

Knockaert, Luc

Nguyen, Huu Minh

Saeys, Yvan

English

Ghent University Academic Bibliography

Avoiding overfitting in surrogate modeling: an alternative approachKeywords: Surrogate modeling, LRM, cross-validation, overfittingHuu Minh Nguyen huuminh.nguyen@ugent.beIvo Couckuyt ivo.couckuyt@ugent.beYvan Saeys yvan.saeys@ugent.beLuc Knockaert luc.knockaert@ugent.beTom Dhaene tom.dhaene@ugent.beGhent University - IBBT, Sint-Pietersnieuwstraat 25, 9000, Gent, BelgiumDirk Gorissen dirk.gorissen@soton.ac.ukUniversity of Southampton, Room 2041, Building 25, Highfield Campus, School of Engineering Sciences, Uni-versity of Southampton, SO17 1BJ, UK1. IntroductionIn many simulation applications , performing routinedesign tasks such as visualization, design space explo-ration or sensitivity analysis quickly becomes imprac-tical due to the (relatively) high cost of computing asingle design (Forrester et al., 2008). Therefore, in afirst design step, surrogate models are often used asreplacements for the real simulator to speed up thedesign process (Queipo et al., 2005). Surrogate mod-els are mathematical models which try to generalizethe complex behavior of the system of interest, from alimited set of data samples to unseen data and this asaccurately as possible. Examples of surrogate modelsare Artificial Neural Networks (ANN), Support Vec-tor Machines (SVM), Kriging models and Radial BasisFunction (RBF) models. Surrogate models are used inmany types of applications, however in this work weconcentrate on noiseless simulation data, as opposedto measurement data or data coming from stochasticsimulators.An important consideration when constructing thesurrogate model is the selection of suitable hyperpa-rameters, as they determine the behavior of the model.Finding a good set of hyperparameters is however non-trivial, as it requires estimating the generalization abil-ity of the model. When dealing with sparse data, thesearch for good hyperparameters becomes even harder.Bad hyperparameters will lead to models with hightraining accuracy (as shown in Fig. 1(a)) but whichAppearing in Proceedings of the 20th Machine Learningconference of Belgium and The Netherlands. Copyright2011 by the authors(s)/owner(s).fail to capture the true behavior (Fig. 1(c)) and in-stead exhibiting artificial ripples and bumps.A common strategy for optimizing model hyperparam-eters when only a limited amount of training data isavailable, is cross-validation. However, because manymodels have to be trained, performing cross-validationcan be quite time and resource consuming, especiallyif the cost of model building is high. Moreover, cross-validation is not always efficient at preventing artificialmodel behavior (Gorissen et al., 2009).We present in this work a new generic auxiliary modelselection measure, called the Linear Reference Model(LRM), which is designed to be fast to compute andwhich reduces the chance of overfitting. LRM identi-fies regions where the model exhibits complex behav-ior (such as oscillations) but lacks the data to supportthis and penalizes the model accordingly. Overfittedregions are identified by comparing the predicted out-put values of the surrogate model to that of a local lin-ear interpolation. Large deviations are an indicationof overfitting, and models are penalized more heavily ifthey diverge more from the local linear interpolation.Note that the risk of underfitting is usually neglible asthe high accuracy typically required in surrogate mod-eling can only achieved by using high complexity mod-els, in which case LRM will only reduce their tendencyto overfit but never to the extent that the models willunderfit. Fig. 1(b) shows the effect of applying theLRM measure. Although the training accuracy of themodel is now worse, the intermediate surrogate modelis better at capturing the true behavior of the system.The LRM score is calculated as follows. First, a Delau-Avoiding overfitting in surrogate modeling: an alternative approach(a) Overfitted model exhibit-ing artificial behavior(b) Minimum LRM score(c) True functionFigure 1. surrogate models of the input noise current(√i2in) of a Low Noise Amplifier (Gorissen et al., 2009)generated with different model selection criteria. The dotsrepresent a sparse intermediate training during model con-struction (7×7 samples).100 150 200 250 30000.10.20.30.40.50.60.70.8Number of samplesTrue error (RRSE)  SampleError + LRMCrossValidation (5 folds)Figure 2. Academic example error on the independent testset as function of the number of data samples for ”cross-validation” and ”LRM in conjuction with the in-sampleerror”.nay tesselation of the input space is contructed afterwhich, for each simplex given by the tesselation, a hy-perplane through the corresponding samples is built.These hyperplanes form the local linear interpolationwhich will be used as reference for the surrogate model.Next, the surrogate model is compared with the locallinear interpolation at every simplex, and the differ-ence between the two is calculated. The LRM scorefor the surrogate model is then simply the average dif-ference over all simplices.We applied the LRM measure to both an analytic aca-demic example and a real world application (Goris-sen et al., 2009) using an adaptive model buildingscheme. In this scheme, a sequential sampling algo-rithm adds a small number of new samples to thetraining data at each iteration after which the surro-gate model is rebuilt. Both cross-validation and LRMare then used to evaluate the updated models. Ourexperiments show that, in this context, the accura-cies of the models selected by LRM converge fasterand are better or comparable to accuracies of modelsselected using cross-validation. This is illustrated inFig. 2 for the academic example, where the accuracyon an independent test set is plotted as a function ofthe number of selected samples. When the numberof training samples is relatively small and the modelsare prone to overfitting, LRM (in combination withthe in-sample error) achieves much lower errors thancross-validation. As the number of training samplesincreases, the difference in accuracy of the models se-lected by both approaches diminishes. However, LRMachieves this accuracy at much reduced computationalcost and can thus provide an interesting alternativeto cross-validation in simulation-based engineering de-sign.ReferencesForrester, A., Sobester, A., & Keane, A. (2008). Engi-neering design via surrogate modelling: A practicalguide. Wiley.Gorissen, D., De Tommasi, L., Crombecq, K., &Dhaene, T. (2009). Sequential modeling of a lownoise amplifier with neural networks and activelearning. Neural Computing and Applications, 18,485–494.Queipo, N., Haftka, R., Shyy, W., Goel, T.,Vaidyanathan, R., & Tucker, P. (2005). Surrogate-based analysis and optimization. Progress inAerospace Sciences, 41, 1–28.

Avoiding overfitting in surrogate modeling: an alternative approach

https://biblio.ugent.be/publication/1268170/file/2952289.pdf

Avoiding overfitting in surrogate modeling: an alternative approach

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography