Evaluating prediction systems in software project estimation

This is the Pre-print version of the Article - Copyright @ 2012 ElsevierContext: Software engineering has a problem in that when we empirically evaluate competing prediction systems we obtain conflicting results. Objective: To reduce the inconsistency amongst validation study results and provide a more formal foundation to interpret results with a particular focus on continuous prediction systems. Method: A new framework is proposed for evaluating competing prediction systems based upon (1) an unbiased statistic, Standardised Accuracy, (2) testing the result likelihood relative to the baseline technique of random ‘predictions’, that is guessing, and (3) calculation of effect sizes. Results: Previously published empirical evaluations of prediction systems are re-examined and the original conclusions shown to be unsafe. Additionally, even the strongest results are shown to have no more than a medium effect size relative to random guessing. Conclusions: Biased accuracy statistics such as MMRE are deprecated. By contrast this new empirical validation framework leads to meaningful results. Such steps will assist in performing future meta-analyses and in providing more robust and usable recommendations to practitioners.Martin Shepperd was supported by the UK Engineering and Physical Sciences Research Council (EPSRC) under Grant EP/H050329

Integrate the GM(1,1) and Verhulst models to predict software stage effort

This is the author's accepted manuscript. The final published article is available from the link below. Copyright @ 2009 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other users, including reprinting/ republishing this material for advertising or promotional purposes, creating new collective works for resale or redistribution to servers or lists, or reuse of any copyrighted components of this work in other works.Software effort prediction clearly plays a crucial role in software project management. In keeping with more dynamic approaches to software development, it is not sufficient to only predict the whole-project effort at an early stage. Rather, the project manager must also dynamically predict the effort of different stages or activities during the software development process. This can assist the project manager to reestimate effort and adjust the project plan, thus avoiding effort or schedule overruns. This paper presents a method for software physical time stage-effort prediction based on grey models GM(1,1) and Verhulst. This method establishes models dynamically according to particular types of stage-effort sequences, and can adapt to particular development methodologies automatically by using a novel grey feedback mechanism. We evaluate the proposed method with a large-scale real-world software engineering dataset, and compare it with the linear regression method and the Kalman filter method, revealing that accuracy has been improved by at least 28% and 50%, respectively. The results indicate that the method can be effective and has considerable potential. We believe that stage predictions could be a useful complement to whole-project effort prediction methods.National Natural Science Foundation of China and the Hi-Tech Research and Development Program of Chin

Making inferences with small numbers of training sets

A potential methodological problem with empirical studies that assess project effort prediction system is discussed. Frequently, a hold-out strategy is deployed so that the data set is split into a training and a validation set. Inferences are then made concerning the relative accuracy of the different prediction techniques under examination. This is typically done on very small numbers of sampled training sets. It is shown that such studies can lead to almost random results (particularly where relatively small effects are being studied). To illustrate this problem, two data sets are analysed using a configuration problem for case-based prediction and results generated from 100 training sets. This enables results to be produced with quantified confidence limits. From this it is concluded that in both cases using less than five training sets leads to untrustworthy results, and ideally more than 20 sets should be deployed. Unfortunately, this raises a question over a number of empirical validations of prediction techniques, and so it is suggested that further research is needed as a matter of urgency

Data accumulation and software effort prediction

BACKGROUND: In reality project managers are constrained by the incremental nature of data collection. Specifically, project observations are accumulated one project at a time. Likewise within-project data are accumulated one stage or phase at a time. However, empirical researchers have given limited attention to this perspective. PROBLEM: Consequently, our analyses may be biased. On the one hand, our predictions may be optimistic due to the availability of the entire data set, but on the other hand pessimistic due to the failure to capitalize upon the temporal nature of the data. Our goals are (i) to explore the impact of ignoring time when building cost prediction models and (ii) to show the benefits of re-estimating using completed phase data during a project. METHOD: Using a small industrial data set of sixteen software projects from a single organization we compare predictive models developed using a time-aware approach with a more traditional leave-one-out analysis. We then investigate the impact of using requirements, design and implementation phase data on estimating subsequent phase effort. RESULTS: First, we find that failure to take the temporal nature of data into account leads to unreliable estimates of their predictive efficacy. Second, for this organization, prior-phase effort data could be used to improve the management of subsequent process tasks. CONCLUSION: We should collect time-related data and use it in our analyses. Failure to do so may lead to incorrect conclusions being drawn, and may also inhibit industrial take up of our research work

Responsiveness summary for the remedial investigation/feasibility study for management of the bulk wastes at the Weldon Spring quarry, Weldon Spring, Missouri

The US Department of Energy (DOE) is responsible for conducting remedial actions at the Weldon Spring site in St. Charles County, Missouri, under its Surplus Facilities Management Program. The site consists of a quarry and a chemical plant area located about 6.4 km (4 mi) northeast of the quarry. The quarry is surrounded by the Weldon Spring Wildfire Area and is near an alluvial well field that constitutes a major source of potable water for St. Charles County; the nearest supply well is located about 0.8 km (0.5 mi) southeast of the quarry. From 1942 to 1969, the quarry was used for the disposal of various radioactively and chemically contaminated materials. Bulk wastes in the quarry consist of contaminated soils and sediments, rubble, metal debris, and equipment. As part of overall site remediation, DOE is proposing to conduct an interim remedial action at the quarry to manage the radioactively and chemically contaminated bulk wastes contained therein. Potential remedial action alternatives for managing the quarry bulk wastes have been evaluated consistent with US Environmental Protection Agency (EPA) guidance for conducting remedial actions under the Comprehensive Environmental Response, Compensation, and Liability Act (CERCLA), as amended. The contents of these documents were developed in consultation with EPA Region VII and the state of Missouri and reflect the focused scope defined for this interim remedial action. 9 refs

Programming the assembly of carboxylic acid-functionalised hybrid polyoxometalates

We report here the straightforward synthesis and characterisation of a series Anderson-type hybrid polyoxometalates in high yield, functionalised with carboxylic acid following the reaction of anhydride precursors with the starting hybrid cluster ([n-N(C4H9)4]3[MnMo6O18((OCH2)3CNH2)2]). Seven new structures have been obtained, five of which have acid-terminated ligands. Six of these structures have been isolated with a yield higher than 80% with high purity. This reaction is limited by the bulkiness of the anhydride used; this effect can be employed to selectively synthesise one isomer out of three other possibilities. The acid groups and aromatic platforms attached to the clusters can act as building tools to bridge several length scales and engineer molecular packing within the crystal structure. The presence of acids should also change the hydrophilicity of the clusters, and therefore the way they interact with hydrophilic surfaces. We also show a potential relationship between the acid group interaction in the packing diagram and the cluster’s tendency to interact with a hydrophilic surface. In addition to reporting a derived synthetic path to new acid-terminated Mn-Anderson-type hybrids, we describe here a new way to program self-assembly motifs of these compounds in the crystal structure and at interfaces

A metamorphic inorganic framework that can be switched between eight single-crystalline states

The design of highly flexible framework materials requires organic linkers, whereas inorganic materials are more robust but inflexible. Here, by using linkable inorganic rings made up of tungsten oxide (P8W48O184) building blocks, we synthesized an inorganic single crystal material that can undergo at least eight different crystal-to-crystal transformations, with gigantic crystal volume contraction and expansion changes ranging from −2,170 to +1,720 Å3 with no reduction in crystallinity. Not only does this material undergo the largest single crystal-to-single crystal volume transformation thus far reported (to the best of our knowledge), the system also shows conformational flexibility while maintaining robustness over several cycles in the reversible uptake and release of guest molecules switching the crystal between different metamorphic states. This material combines the robustness of inorganic materials with the flexibility of organic frameworks, thereby challenging the notion that flexible materials with robustness are mutually exclusive