Effort Estimation for Service-Oriented Computing Environments

The concept of service in Service-Oriented Architecture (SOA) makes possible to introduce other ideas like service composition, governance and virtualization. Each of these ideas, when exercised to an enterprise level, provides benefits in terms of cost and performance. These ideas bring many new opportunities for the project managers in making the estimates of effort required to produce SOA systems. This is because the SOA systems are different from traditional software projects and there is a lack of efficient metrics and models for providing a high level of confidence in effort estimation. Thus, in this paper, an efficient estimation methodology has been presented based on analyzing the development phases of past SOA based software systems. The objective of this paper is twofold: first, to study and analyze the development phases of some past SOA based systems; second, to propose estimation metrics based on these analyzed parameters. The proposed methodology is facilitated from the use of four regression(s) based estimation models. The validation of the proposed methodology is cross checked by comparing the predictive accuracy, using some commonly used performance measurement indicators and box-plots evaluation. The evaluation results of the study (using industrial data collected from 10 SOA based software systems) show that the effort estimates obtained using the multiple linear regression model are more accurate and indicate an improvement in performance than the other used regression models

Cost and Risk Considerations for Test and Evaluation of Unmanned and Autonomous Systems of Systems

The evolutionary nature of Unmanned and Autonomous systems of systems (UASoS) acquisition needs to be matched by evolutionary test capabilities yet to be developed. As part of this effort we attempt to understand the cost and risk considerations for UASoS Test and Evaluation (T&E) and propose the development of a parametric cost model to conduct trade-off analyses. This paper focuses on understanding the need for effort estimation for UASoS, the limitations of existing cost estimation models, and how our effort can be merged with the cost estimation processes. We present the prioritization of both technical and organizational cost drivers. We note that all drivers associated with time constraints, integration, complexity, understanding of architecture and requirements are rated highly, while those regarding stakeholders and team cohesion are rated as medium. We intend for our cost model approach to provide management guidance to the T&E community in estimating the effort required for UASoS T&E

New Effort and Schedule Estimation Models for Agile Processes in the U.S. DoD

Excerpt from the Proceedings of the Nineteenth Annual Acquisition Research SymposiumThe DoD’s new software acquisition pathway prioritizes speed of delivery, advocating agile software processes. Estimating the cost and schedule of agile software projects is critical at an early phase to establish baseline budgets and to select competitive bidders. The challenge is that common ag-ile sizing measures such as story points and user stories are not practical for early estimation as these are often reported after contract award in DoD. This study provides a set of parametric effort and schedule estimation models for agile projects using a sizing measure that is available before proposal evaluation based on data from 36 DoD agile projects. The results suggest that initial software requirements, defined as the sum of functions and external interfaces, is an effective sizing measure for early estimation of effort and schedule of agile projects. The models’ accuracy improves when application domain groups and peak staff are added as inputs.Approved for public release; distribution is unlimited

New Effort and Schedule Estimation Models for Agile Processes in the U.S. DoD

Excerpt from the Proceedings of the Nineteenth Annual Acquisition Research SymposiumThe DoD’s new software acquisition pathway prioritizes speed of delivery, advocating agile software processes. Estimating the cost and schedule of agile software projects is critical at an early phase to establish baseline budgets and to select competitive bidders. The challenge is that common ag-ile sizing measures such as story points and user stories are not practical for early estimation as these are often reported after contract award in DoD. This study provides a set of parametric effort and schedule estimation models for agile projects using a sizing measure that is available before proposal evaluation based on data from 36 DoD agile projects. The results suggest that initial software requirements, defined as the sum of functions and external interfaces, is an effective sizing measure for early estimation of effort and schedule of agile projects. The models’ accuracy improves when application domain groups and peak staff are added as inputs.Approved for public release; distribution is unlimited

Software Size and Effort Estimation from Use Case Diagrams Using Regression and Soft Computing Models

In this research, we propose a novel model to predict software size and effort from use case diagrams. The main advantage of our model is that it can be used in the early stages of the software life cycle, and that can help project managers efficiently conduct cost estimation early, thus avoiding project overestimation and late delivery among other benefits. Software size, productivity, complexity and requirements stability are the inputs of the model. The model is composed of six independent sub-models which include non-linear regression, linear regression with a logarithmic transformation, Radial Basis Function Neural Network (RBFNN), Multilayer Perceptron Neural Network (MLP), General Regression Neural Network (GRNN) and a Treeboost model. Several experiments were conducted to train and test the model based on the size of the training and testing data points. The neural network models were evaluated against regression models as well as two other models that conduct software estimation from use case diagrams. Results show that our model outperforms other relevant models based on five evaluation criteria. While the performance of each of the six sub-models varies based on the size of the project dataset used for evaluation, it was concluded that the non-linear regression model outperforms the linear regression model. As well, the GRNN model exceeds other neural network models. Furthermore, experiments demonstrated that the Treeboost model can be efficiently used to predict software effort

Software Development Effort Estimation Using Regression Fuzzy Models

Software effort estimation plays a critical role in project management. Erroneous results may lead to overestimating or underestimating effort, which can have catastrophic consequences on project resources. Machine-learning techniques are increasingly popular in the field. Fuzzy logic models, in particular, are widely used to deal with imprecise and inaccurate data. The main goal of this research was to design and compare three different fuzzy logic models for predicting software estimation effort: Mamdani, Sugeno with constant output and Sugeno with linear output. To assist in the design of the fuzzy logic models, we conducted regression analysis, an approach we call regression fuzzy logic. State-of-the-art and unbiased performance evaluation criteria such as standardized accuracy, effect size and mean balanced relative error were used to evaluate the models, as well as statistical tests. Models were trained and tested using industrial projects from the International Software Benchmarking Standards Group (ISBSG) dataset. Results showed that data heteroscedasticity affected model performance. Fuzzy logic models were found to be very sensitive to outliers. We concluded that when regression analysis was used to design the model, the Sugeno fuzzy inference system with linear output outperformed the other models.Comment: This paper has been accepted in January 2019 in Computational Intelligence and Neuroscience Journal (In Press

Make the most of your samples : Bayes factor estimators for high-dimensional models of sequence evolution

Background: Accurate model comparison requires extensive computation times, especially for parameter-rich models of sequence evolution. In the Bayesian framework, model selection is typically performed through the evaluation of a Bayes factor, the ratio of two marginal likelihoods (one for each model). Recently introduced techniques to estimate (log) marginal likelihoods, such as path sampling and stepping-stone sampling, offer increased accuracy over the traditional harmonic mean estimator at an increased computational cost. Most often, each model's marginal likelihood will be estimated individually, which leads the resulting Bayes factor to suffer from errors associated with each of these independent estimation processes. Results: We here assess the original 'model-switch' path sampling approach for direct Bayes factor estimation in phylogenetics, as well as an extension that uses more samples, to construct a direct path between two competing models, thereby eliminating the need to calculate each model's marginal likelihood independently. Further, we provide a competing Bayes factor estimator using an adaptation of the recently introduced stepping-stone sampling algorithm and set out to determine appropriate settings for accurately calculating such Bayes factors, with context-dependent evolutionary models as an example. While we show that modest efforts are required to roughly identify the increase in model fit, only drastically increased computation times ensure the accuracy needed to detect more subtle details of the evolutionary process. Conclusions: We show that our adaptation of stepping-stone sampling for direct Bayes factor calculation outperforms the original path sampling approach as well as an extension that exploits more samples. Our proposed approach for Bayes factor estimation also has preferable statistical properties over the use of individual marginal likelihood estimates for both models under comparison. Assuming a sigmoid function to determine the path between two competing models, we provide evidence that a single well-chosen sigmoid shape value requires less computational efforts in order to approximate the true value of the (log) Bayes factor compared to the original approach. We show that the (log) Bayes factors calculated using path sampling and stepping-stone sampling differ drastically from those estimated using either of the harmonic mean estimators, supporting earlier claims that the latter systematically overestimate the performance of high-dimensional models, which we show can lead to erroneous conclusions. Based on our results, we argue that highly accurate estimation of differences in model fit for high-dimensional models requires much more computational effort than suggested in recent studies on marginal likelihood estimation