RECOMMENDATIONS FOR IMPROVING SOFTWARE COST ESTIMATION IN DOD ACQUISITION

Acquisition initiatives within the Department of Defense (DOD) are becoming increasingly reliant on software. While the DOD has ample experience in estimating costs of hardware acquisition, expertise in estimating software acquisition costs is lacking. The objective of this capstone project is to summarize the current software cost estimating methods, analyze existing software cost estimating models, and suggest areas and methods for improvement. To accomplish this, surveys were conducted to gather program cost data, which was run through existing cost estimating models. From here, the outputs were compared to actual program costs. This established a baseline for the effectiveness of existing methods and guided suggestions for areas of improvement. The Software Resource Data Reports (SRDR) data used seemed to have spurious data reporting from at least one source, and the base cost estimation models were not found to be sufficiently accurate in our study. The capstone finds that calibrating the cost models to the data available improved those models dramatically. In all, the capstone recommends performing data realism checks upon SRDR submissions to ensure data accuracy and calibrating cost models for each contractor with the available data before using them to estimate DOD Acquisition costs.Civilian, Department of the ArmyCivilian, Department of the ArmyCivilian, Department of the ArmyCivilian, Department of the ArmyApproved for public release. Distribution is unlimited

Software Development Effort Estimation Using Regression Fuzzy Models

Software effort estimation plays a critical role in project management. Erroneous results may lead to overestimating or underestimating effort, which can have catastrophic consequences on project resources. Machine-learning techniques are increasingly popular in the field. Fuzzy logic models, in particular, are widely used to deal with imprecise and inaccurate data. The main goal of this research was to design and compare three different fuzzy logic models for predicting software estimation effort: Mamdani, Sugeno with constant output and Sugeno with linear output. To assist in the design of the fuzzy logic models, we conducted regression analysis, an approach we call regression fuzzy logic. State-of-the-art and unbiased performance evaluation criteria such as standardized accuracy, effect size and mean balanced relative error were used to evaluate the models, as well as statistical tests. Models were trained and tested using industrial projects from the International Software Benchmarking Standards Group (ISBSG) dataset. Results showed that data heteroscedasticity affected model performance. Fuzzy logic models were found to be very sensitive to outliers. We concluded that when regression analysis was used to design the model, the Sugeno fuzzy inference system with linear output outperformed the other models.Comment: This paper has been accepted in January 2019 in Computational Intelligence and Neuroscience Journal (In Press

Adopting the Appropriate Performance Measures for Soft Computing-based Estimation by Analogy

Soft Computing based estimation by analogy is a lucrative research domain for the software engineering research community. There are a considerable number of models proposed in this research area. Therefore, researchers are of interest to compare the models to identify the best one for software development effort estimation. This research showed that most of the studies used mean magnitude of relative error (MMRE) and percentage of prediction (PRED) for the comparison of their estimation models. Still, it was also found in this study that there are quite a number of criticisms done on accuracy statistics like MMRE and PRED by renowned authors. It was found that MMRE is an unbalanced, biased, and inappropriate performance measure for identifying the best among competing estimation models. The accuracy statistics, e.g., MMRE and PRED, are still adopted in the evaluation criteria by the domain researchers, stating the reason for “widely used,” which is not a valid reason. This research study identified that, since there is no practical solution provided so far, which could replace MMRE and PRED, the researchers are adopting these measures. The approach of partitioning the large dataset into subsamples was tried in this paper using estimation by analogy (EBA) model. One small and one large dataset were considered for it, such as Desharnais and ISBSG release 11. The ISBSG dataset is a large dataset concerning Desharnais. The ISBSG dataset was partitioned into subsamples. The results suggested that when the large datasets are partitioned, the MMRE produces the same or nearly the same results, which it produces for the small dataset. It is observed that the MMRE can be trusted as a performance metric if the large datasets are partitioned into subsamples

An Intelligent Framework for Estimating Software Development Projects using Machine Learning

The IT industry has faced many challenges related to software effort and cost estimation. A cost assessment is conducted after software effort estimation, which benefits customers as well as developers. The purpose of this paper is to discuss various methods for the estimation of software effort and cost in the context of software engineering, such as algorithmic methods, expert judgment methods, analogy-based estimation methods, and machine learning methods, as well as their different aspects. In spite of this, estimation of the effort involved in software development are subject to uncertainty. Several methods have been developed in the literature for improving estimation accuracy, many of which involve the use of machine learning techniques. A machine learning framework is proposed in this paper to address this challenging problem. In addition to being completely independent of algorithmic models and estimation problems, this framework also features a modular architecture. It has high interpretability, learning capability, and robustness to imprecise and uncertain inputs

Calibration and Validation of the Checkpoint Model to the Air Force Electronic Systems Center Software Database

This research effort focused on the calibration and validation of CHECKPOINT Version 2.3.1, a computerized software cost estimating tool, to the USAF Electronic Systems Center (ESC) software database. This thesis is a direct follow-on to a 1996 CHECKPOINT study at the Air Force Institute of Technology, which successfully calibrated and validated CHECKPOINT to the SMC software database. While this research generally parallels the methodology in the aforementioned study, it offers advancements in the CHECKPOINT calibration and validation procedure, and it refines the data stratification process and the statistical analyses employed. After stratifying the ESC software database into ten usable data sets, the author calibrated and validated the CHECKPOINT model on each data set. Although the results of this study exhibited occasional improvements in estimating accuracy for both the calibration and validation subsets, the model generally failed to satisfy the accuracy criteria used to assess overall calibration success and estimating accuracy (MMRE0.75). Thus, the CHECKPOINT model was not successfully calibrated or validated to the 1997 version of the ESC database. The results of this study illuminate the need for complete, accurate and homogeneous data as a requirement for a successful calibration and validation effort

Calibration and Validation of the Sage Software Cost/Schedule Estimating System to United States Air Force Databases

This research entailed calibration and validation of the SAGE Software Cost/Schedule Estimating System, Version 1.7 as a means to improve estimating accuracy for DoD software-intensive systems, and thereby introduce stability into software system development. SAGE calibration consisted of using historical data from completed projects at the Space and Missile Systems Center (SMC) and the Electronic Systems Center (ESC) to derive average performance factors (i.e., calibration factors) for pre-defined categories of projects. A project was categorized for calibration by either its primary application or by the contractor that developed it. The intent was to determine the more appropriate categorization for calibration. SAGE validation consisted of using the derived calibration factors to predict completed efforts, not used in deriving the factors. Statistical resampling employing Monte Carlo simulation was used to calibrate and validate the model on each possible combination of a category\u27s projects. Three statistical measures were employed to measure model performance in default and calibrated estimating modes. SAGE generally did not meet pre-established criteria for estimating accuracy, although the model demonstrated some improvement with calibration. Calibration of projects categorized by contractor resulted in better calibrated model performance than calibration of projects categorized by application. This categorization is suggested for future consideration

Parametric software effort estimation based on optimizing correction factors and multiple linear regression

Context: Effort estimation is one of the essential phases that must be accurately predicted in the early stage of software project development. Currently, solving problems that affect the estimation accuracy of Use Case Points-based methods is still a challenge to be addressed. Objective: This paper proposes a parametric software effort estimation model based on Optimizing Correction Factors and Multiple Regression Models to minimize the estimation error and the influence of unsystematic noise, which has not been considered in previous studies. The proposed method takes advantage of the Least Squared Regression models and Multiple Linear Regression models on the Use Case Points-based elements. Method: We have conducted experimental research to evaluate the estimation accuracy of the proposed method and compare it with three previous related methods, i.e., 1) the baseline estimation method – Use Case Points, 2) Optimizing Correction Factors, and 3) Algorithmic Optimization Method. Experiments were performed on datasets (Dataset D1, Dataset D2, and Dataset D3). The estimation accuracy of the methods was analysed by applying various unbiased evaluation criteria and statistical tests. Results: The results proved that the proposed method outperformed the other methods in improving estimation accuracy. Statistically, the results proved to be significantly superior to the three compared methods based on all tested datasets. Conclusion: Based on our obtained results, the proposed method has a high estimation capability and is considered a helpful method for project managers during the estimation phase. The correction factors are considered in the estimation process. AuthorFaculty of Applied Informatics, Tomas Bata University in Zlin [IGA/CebiaTech/2021/001, RO30216002025

Adopting the appropriate performance measures for soft computing based estimation by analogy

Soft Computing based estimation by analogy is a lucrative research domain for the software engineering research community. There are a considerable number of models proposed in this research area. Therefore, researchers are of interest to compare the models to identify the best one for software development effort estimation. This research showed that most of the studies used mean magnitude of relative error (MMRE) and percentage of prediction (PRED) for the comparison of their estimation models. Still, it was also found in this study that there are quite a number of criticisms done on accuracy statistics like MMRE and PRED by renowned authors. It was found that MMRE is an unbalanced, biased, and inappropriate performance measure for identifying the best among competing estimation models. The accuracy statistics, e.g., MMRE and PRED, are still adopted in the evaluation criteria by the domain researchers, stating the reason for "widely used, " which is not a valid reason. This research study identified that, since there is no practical solution provided so far, which could replace MMRE and PRED, the researchers are adopting these measures. The approach of partitioning the large dataset into subsamples was tried in this paper using estimation by analogy (EBA) model. One small and one large dataset were considered for it, such as Desharnais and ISBSG release 11. The ISBSG dataset is a large dataset concerning Desharnais. The ISBSG dataset was partitioned into subsamples. The results suggested that when the large datasets are partitioned, the MMRE produces the same or nearly the same results, which it produces for the small dataset. It is observed that the MMRE can be trusted as a performance metric if the large datasets are partitioned into subsamples