BACKGROUND: Prediction e.g. of project cost is an important concern in software engineering. PROBLEM: Although many empirical validations of software engineering prediction systems have been published, no one approach dominates and sense-making of conflicting empirical results is proving challenging. METHOD: We propose a new approach to evaluating competing prediction systems based upon an unbiased statistic (Standardised Accuracy), analysis of results relative to the baseline technique of guessing and calculation of effect sizes. RESULTS: Two empirical studies are revisited and the published results are shown to be misleading when re-analysed using our new approach. CONCLUSION: Biased statistics such as MMRE are deprecated. By contrast our approach leads to valid results. Such steps will greatly assist in performing future meta-analyses

Shepperd, M

English

Brunel University Research Archive

New Ideas and Emerging Research: Evaluating PredictionSystem AccuracyMartin ShepperdSchool of Computing, IS & MathsBrunel UniversityUxbridge UB8 3PH, United Kingdommartin.shepperd@brunel.ac.ukStephen G. MacDonellSERL, Computing & Mathematical SciencesAUT University, Private Bag 92006Auckland 1142, New Zealandstephen.macdonell@aut.ac.nzABSTRACTBACKGROUND: Prediction e.g. of project cost is an im-portant concern in software engineering.PROBLEM: Although many empirical validations of soft-ware engineering prediction systems have been published,no one approach dominates and sense-making of conflictingempirical results is proving challenging.METHOD: We propose a new approach to evaluating com-peting prediction systems based upon an unbiased statistic(Standardised Accuracy), analysis of results relative to thebaseline technique of guessing and calculation of effect sizes.RESULTS: Two empirical studies are revisited and the pub-lished results are shown to be misleading when re-analysedusing our new approach.CONCLUSION: Biased statistics such as MMRE are depre-cated. By contrast our approach leads to valid results. Suchsteps will greatly assist in performing future meta-analyses.Categories and Subject DescriptorsD.2.9 [Management]: Cost estimationGeneral TermsManagement, MeasurementKeywordsSoftware project management, empirical analysis, predictionsystem, prediction quality, forecasting, accuracy.1. INTRODUCTIONAn important feature of any engineering discipline is theability to make timely and accurate predictions, and in or-der to do so we need prediction systems. Moreover we needto evaluate such models or prediction systems. Software en-gineering is no exception. However, judging predictive accu-racy is a subtle task and our failure to do so appropriatelyInformal Technical Note: February 7, 2011is a contributor to the present situation of a lack of con-clusion stability across studies [13] and inconclusive resultsfrom systematic reviews [7].So we find ourselves in a situation of increasing numbersof models and modelling techniques being offered, frequentlywith contradictory claims and results, for a wide range ofdata sets [9]. The picture is further complicated by the useof different accuracy statistics and validation schemes. Theconsequence is difficulties in sense making and increasingnumbers of researchers commenting upon the challenge ofconclusion instability.In order to illustrate the challenges and show our pro-posed solution we revisit two published empirical studies ofsoftware project effort prediction systems and demonstratehow traditional evaluation techniques are misleading andhow evaluation should be handled. The selected studies in-volved the first author (MS) and two contrasting data sets.Study 1 was published in an international conference withan acceptance rate of approximately 30% and Study 2 in theIEEE Transactions on Software Engineering.2. METHODIn order to bring some generality to our discussion we pro-pose the following framework. We validate some predictionsystem P over a data set D using some accuracy statisticS. Empirical evaluation can be seen as an attempt to es-tablish a set of preference relations such that S(P1, D) 4S(P2, D), ..., S(Pn, D). In this paper we restrict the discus-sion to predicting some continuous1 output that is denotedY , however, in principle the argument also applies to classi-fiers where the output is categorical.When establishing these preference relations we need to beconcerned with some other questions. For a given accuracystatistic S and data set D:1. Does the prediction system P1 outperform na¨ıve guess-ing, a special case of a prediction system that we de-note P0. In other words is P1 ≺ P0 statistically signif-icant for some pre-determined value of α?2. Is the difference P1 ≺ P2 significant for some pre-determined value of α?3. Is the effect size large enough to justify P1 ≺ P2 inpractice?1Strictly speaking we also include the absolute scalar typei.e. counting.Researchers have tended to focus on the second questiontypically by testing for the difference in means or medians.Exceptions are Jørgensen [3] who used sample mean pro-ductivity as fairly simple benchmark and Miller [10] whoemphasises the need to estimate effect size in order to cal-culate the power of an experiment.Typically statistics such as MMRE have been used as theaccuracy statistic S for continuous prediction systems, whereMMRE is given as: Pn1 |(yi − yˆi)|/yin(1)and n is the number of cases in D. Obviously for classifiersdifferent accuracy statistics e.g. AUC would be more appro-priate.Unfortunately it has been shown that this popular pre-diction accuracy statistic is flawed [8] in that it is a biasedestimator of central tendency of the residuals of a predictionsystem because it is an asymmetric measure. Foss et al. [2]also highlighted problems associated with MMRE by meansof simulation.The fundamental variable of interest is the residual orprediction error, yi − yˆi. As has been indicated there arepotentially a number properties of this variable, however,for the present we assume the focus is upon central tendencyrather than say bias or spread. As prediction system biasis not a concern, we use absolute residuals and for a set ofpredictions, mean absolute residual (MAR). This measureof centre is unbiased since it is not based on ratios unlikeMMRE which must be bounded by zero in one direction andunbounded in the other.However, MAR does have the disadvantage that it is hardto interpret and comparisons cannot be made across datasets since the residuals are not standardised. Therefore wepropose to measure accuracy as the MAR relative to na¨ıveguessing P0 hence we offer a standardised accuracy measureSA for prediction technique Pi:SAPi = 1− MARPiMARP0(2)where MARP0 is the mean value of a large number, typically1000, runs of na¨ıve guessing. This is defined as – to predict ayˆ for the target case randomly sample over all the remainingcases and take yˆ = yRAND. This is the most na¨ıve approachpossible without being perverse. It also provides a relevantbaseline irrespective of the exact form of P1. Over manyruns the MARP0 will converge on simply using the samplemean. The advantage of not using the sample mean is onecan estimate the distribution of MARs for determining like-lihood of any observed MAR value along with the varianceof MAR. Note that whilst SA, like MMRE is a ratio, this isnot problematic since we are only interested in one directioni.e. better than random.The interpretation of SA is that the ratio represents howmuch better Pi is than na¨ıve guessing. Clearly a negativevalue would be worrisome and close to zero discouraging!To judge the effect size we use a standardised measure dueto Glass [11] which is:∆ =MARPi −MARP0sP0(3)where sP0 is the sample standard deviation of the na¨ıveguessing strategy. Note we do not use a pooled measure asin Cohen’s d since (i) we cannot assume the variances ofPi and P0 are homogenous and (ii) the comparison is withrespect to the control i.e. na¨ıve guessing2. Even if comparingbetween two prediction systems the rationale still tends to beP1 represents the status quo with which P2 is to contrastedand hence P1 is effectively a control.Having defined a standardised accuracy measure SA andan effect size measure ∆ we are now in a position to revisitsome typical empirical validation studies of project effortprediction systems and pose our three questions.3. STUDY 1: THE ATKINSON DATA SETThis is a small data set of telecoms projects that uses real-time function points as a size estimator. It was one of twodata sets employed by the replication study (Study 1) [12] ofa proposed regression to the mean (R2M) prediction method[4]. The details are not important, suffice to say that the aimof Study 1 was to empirically compare the accuracy of R2Mwith an estimation-by-analogy (EBA) prediction system asa baseline. The reported accuracy statistics were MAR andMMRE (for interpretation purposes).Table 1: Atkinson Data Set ResultsPrediction Method MAR MMRE SAEBA-prod 331.6 99% -17%R2M-prod 291.6 84% -3%NG 283.0 86.2% 0%NG0.05 210.8 56.8% 26%Expert judgment 117.5 24.1% 58%Table 1 gives the results for SA sorted and expressed asa percentage including EBA-prod and R2M-prod which wasthe main theme for study Study 1. However, we also add theresults for na¨ıve guessing (P0) and the 5% quantile for thecumulative distribution of MAR values from 1000 runs of P0(the histogram is shown in Fig. 1). An encouraging, and tobe expected, observation is the near Gaussian distributionwhich contrasts with a more skewed MMRE distributionfrom the same 1000 runs (see Fig. 2). Note also that MMREdoes not preserve the same ranking of prediction systems.We can see that the two methods under review (EBA-prodand R2M-prod) both perform worse than na¨ıve guessing. Itis therefore pointless to even consider whether the differencesbetween the two methods are significant because one wouldbe better off guessing. In a sense one might argue we havenon-prediction systems. Although not included in Study1, we add results using predictions made by experts at thetime which formed part of the data set. We see that expertjudgment is best and a 58% improvement over guessing.4. STUDY 2: THEDESHARNAISDATA SETStudy 2 considers nine data sets in an empirical compari-son of stepwise regression (SWR) prediction, intended as abenchmark, and estimation by analogy (EBA). We choosethe Desharnais data set out of the nine as it is substan-tially larger than Atkinson (77 cf. 16 cases) and because it2One note of caution is that Glass’s ∆ is known to be abiased estimator for small sample sizes or if there are largediscrepancies in sample sizes, in which case Hedges’s g mightbe preferred.Histogram of MAR values from Naive Guessing for the Atkinson Data SetResults[, 1]Frequency150 200 250 300 350 400 450050100150Figure 1:MMRE values from Naive Guessing for the Atkinson Data SetResults[, 2]Frequency0.4 0.6 0.8 1.0 1.2 1.4050100150200Figure 2:is widely used [9]. The main accuracy statistic employed inStudy 2 [14] was MMRE.Table 2: Desharnais Data Set ResultsPrediction Method MAR MMRE SANG 4149 142% 0%NG0.05 3556 110% 13.9%EBA, k = 3 2289 63% 44.9%SWR 2022 71% 51.3%EBA-FSS, k = 3 1682 41% 55%Table 2 sets out the prediction results from SWR, EBAand, in addition, we add EBA that uses a better (more re-cent) feature subset selection method (EBA-FSS) [6]. Tothese prediction systems are added the baseline techniqueof na¨ıve guessing and the 5% quantile (NG0.05).In order to obtain the residuals we re-ran the regressionanalysis and EBA keeping the procedure as similar as pos-sible to [14]. Due to changes in the EBA tool over the in-tervening 12 years our results differ slightly from Study 2although the rankings are preserved. Running a paired one-tailed t-test yields p ≈ 0.3 which of course is not significant.However, in fairness to the original study, the argument wasmade with respect to the overall analysis of nine data sets.We also see the bias of MMRE since it indicates EBA isto be preferred to the SWR prediction system yet by exam-ining the residuals and MAR we see this is fallacious. Fur-thermore, SWR seeks to minimise the sum of the squaresof the residuals so using MMRE is inappropriate. Exam-ining the SA measure we see that both prediction systemsimprove upon guessing and fall beyond the 0.05 confidencelevel. SWR is the more accurate technique and offers a51.3% improvement over guessing. We observe that the dif-ference in SA between SWR and EBA is small (6.4%) andin the opposite direction to that indicated by MMRE i.e.PSWR ≺ PEBA.Table 3: Desharnais Effect SizesPrediction MAR MMRE ∆ wrt ∆ wrtMethod SD SD P0 SWRNG 4220 258% n.a. n.a.EBA, k=3 2684 82% 0.436 -0.123SWR 2171 118% 0.499 n.a.Next we consider effect size, ∆ (see Table 3) using Eqn.3. Cohen [1] suggested that one might interpret a d or ∆of 0.2 as indicating a small effect, 0.5 as medium and 0.8 aslarge. Note that ∆ may exceed 1. First, we see that theimprovement in accuracy over guessing due to using SWRor EBA tends towards a medium effect size. Clearly eachapproach is doing substantially better than na¨ıve guessingbut then this is not saying a lot! Study 2 set out to compareEBA with SWR as a benchmark so we recalculate the ∆’swith respect to SWR and use the SWR MARSD value. Herewe see a small, statistically insignificant, negative effect sizebetween SWR and EBA.The above analysis uses MAR as the accuracy statisticfor the reasons discussed previously, however, to indicateyet another disadvantage with MMRE we provide the stan-dardised standard deviations (i.e. mean/SD) in Table 4. Wesee that in all cases the standardised variance is greater forMMRE than for MAR. This means that any effects due todifferences in prediction system are harder to detect sincethey will be masked by variance inherent in the accuracystatistic.Table 4: Variance due to MAR cf. MMREP MAR/SD MAR MMRE / SD MMRENG 1.16 2.35EBA, k=3 1.80 4.10SWR 2.04 3.645. DISCUSSION AND CONCLUSIONSFirst we need to stress that our purpose is not the evalua-tion of specific prediction systems but rather the question ofhow one evaluates prediction systems. From the foregoingit is clear that there are a number of problems with how wepresently view and analyse results from empirical validationstudies of competing prediction systems. We have re-visitedtwo such studies and show that the published results — bothstudies underwent peer reviewing for prestigious outlets —were misleading.For Study 1 the absence of some fundamental baseline(P0) meant that the accuracy statistics were hard to inter-pret (MAR=292 and MMRE=84%) and so the problem thatboth techniques were substantially worse than guessing wasoverlooked. In such a situation the relative improvement ofR2M-prod is irrelevant and in any case is most likely theconsequence of the technique converging on using the sam-ple mean as the correlation between predicted and actualproductivity tends to zero.In Study 2, unlike Study 1, the prediction techniques per-form significantly better than P0. Here however, the prob-lem is reliance on a biased accuracy statistic MMRE whichreverses the preference relation between EBA and SWR.Further the effect with respect to P0 would barely be con-sidered medium and the difference between EBA and SWRis negligible.However before both papers are consigned to the scrapheapof history it should be added that both papers use multipledata sets and so some form of meta-analysis is required be-fore commenting upon the overall conclusions of each study.So where do we go from here? First, biased accuracystatistics should be absolutely deprecated. We see no goodpurpose in using MMRE. We have proposed a new stan-dardised measure SA that is unbiased, enables meaningfulinterpretation and can be used across different data sets.Second, validation studies should address the three ques-tions we pose in this paper, namely is P1 better than guess-ing (P0), is P1 significantly better than P2 and is the effectsize large enough to be of practical import? Biased statis-tics and noisy data sets will lead to high levels of varianceresulting in the between-prediction system variance beingdifficult to detect. Finally, this approach will better supportmeta-analysis via effect sizes even when different responsevariables have been used [5].AcknowledgmentsMartin Shepperd was supported by the UK Engineering andPhysical Sciences Research Council (EPSRC) under grantEP/H050329/1. The authors wish to thank Magne Jør-gensen, Barbara Kitchenham, Carolyn Mair and Audris Mockusfor helpful comments and discussions.6. REFERENCES[1] J. Cohen. Statistical power analysis for the behavioralsciences. Lawrence Earlbaum Associates., Hillsdale,NJ, 2nd edition, 1988.[2] T. Foss, E. Stensrud, B. Kitchenham, and I. Myrtveit.A simulation study of the model evaluation criterionmmre. IEEE Transactions on Software Engineering,29(11):985–995, 2003.[3] M. Jørgensen. Experience with the accuracy ofsoftware maintenance task effort prediction models.IEEE Transactions on Software Engineering,21(8):674–681, 1995.[4] M. Jørgensen, U. Indahl, and D. I. K. Sjøberg.Software effort estimation by analogy and “regressiontoward the mean”. J. of Systems & Software,68(3):253–262, 2003.[5] V. Kampenes, T. Dyb˚a, J.E. Hannay, and D.I.K.Sjøberg. A systematic review of effect size in softwareengineering experiments. Information & SoftwareTechnology, 49:1073–1086, 2007.[6] C. Kirsopp, M.J. Shepperd, and J. Hart. Searchheuristics, case-based reasoning and software projecteffort prediction. In Genetic and EvolutionaryComputation Conf., New York, 2002. AAAI.[7] B. Kitchenham, P. Brereton, D. Budgen, M. Turner,J. Bailey, and S. Linkman. Systematic literaturereviews in software engineering – a systematicliterature review. Information & Software Technology,51(1):7–15, 2009.[8] B.A. Kitchenham, S.G. MacDonell, L. Pickard, andM.J. Shepperd. What accuracy statistics reallymeasure. IEE Proceedings - Software Engineering,148(3):81–85, 2001.[9] C. Mair, M. Shepperd, and M. Jørgensen. An analysisof data sets used to train and validate cost predictionsystems. In PROMISE 2005, St Louis, MI, 2005. ACMComputer Press.[10] J. Miller, J. Daly, M. Wood, M. Roper, andA. Brooks. Statistical power and its subcomponents -missing and misunderstood concepts in empiricalsoftware engineering research. Information & SoftwareTechnology, 39(4):285–295, 1997.[11] R. Rosenthal. Parametric measures of effect size. InH. Cooper and L. Hedges, editors, The Handbook ofResearch Synthesis. Sage, New York, 1994.[12] M. Shepperd and M. Cartwright. A replication of theuse of regression towards the mean (r2m) as anadjustment to effort estimation models. In 11th IEEEIntl. Softw. Metrics Symp. (Metrics05), Como, Italy,2005. Computer Society Press.[13] M.J. Shepperd. Software project economics: aroadmap. In Future of Software Engineering (ICSE2007), Minneapolis, 2007. ACM.[14] M.J. Shepperd and C. Schofield. Estimating softwareproject effort using analogies. IEEE Transactions onSoftware Engineering, 23(11):736–743, 1997.

A replication of the use of regression towards the mean (r2m) as an adjustment to effort estimation models.

A replication of the use of regression towards the mean (r2m) as an adjustment to eort estimation models.

A simulation study of the model evaluation criterion mmre.

A systematic review of eect size in software engineering experiments.

A systematic review of effect size in software engineering experiments.

An analysis of data sets used to train and validate cost prediction systems.

Estimating software project effort using analogies.

Estimating software project eort using analogies.

Experience with the accuracy of software maintenance task effort prediction models.

Experience with the accuracy of software maintenance task eort prediction models.

Parametric measures of eect size.

Parametric measures of effect size.

Search heuristics, case-based reasoning and software project effort prediction.

Search heuristics, case-based reasoning and software project eort prediction.

Software effort estimation by analogy and “regression toward the mean”.

Software eort estimation by analogy and \regression toward the mean&quot;.

Software project economics: a roadmap.

Statistical power analysis for the behavioral sciences. Lawrence Earlbaum Associates.,

Statistical power and its subcomponents -missing and misunderstood concepts in empirical software engineering research.

Systematic literature reviews in software engineering – a systematic literature review.

Systematic literature reviews in software engineering { a systematic literature review.

What accuracy statistics really measure.

New ideas and emerging research: evaluating prediction system accuracy

New ideas and emerging research: evaluating prediction system accuracy

Abstract

Similar works

Full text

Available Versions

Brunel University Research Archive