Evidence-based defect assessment and prediction for software product lines

The systematic reuse provided by software product lines provides opportunities to achieve increased quality and reliability as a product line matures. This has led to a widely accepted assumption that as a product line evolves, its reliability improves. However, evidence in terms of empirical investigation of the relationship among change, reuse and reliability in evolving software product lines is lacking. To address the problem this work investigates: 1) whether reliability as measured by post-deployment failures improves as the products and components in a software product line change over time, and 2) whether the stabilizing effect of shared artifacts enables accurate prediction of failure-prone files in the product line. The first part of this work performs defect assessment and investigates defect trends in Eclipse, an open-source software product line. It analyzes the evolution of the product line over time in terms of the total number of defects, the percentage of severe defects and the relationship between defects and changes. The second part of this work explores prediction of failure-prone files in the Eclipse product line to determine whether prediction improves as the product line evolves over time. In addition, this part investigates the effect of defect and data collection periods on the prediction performance. The main contributions of this work include findings that the majority of files with severe defects are reused files rather than new files, but that common components experience less change than variation components. The work also found that there is a consistent set of metrics which serve as prominent predictors across multiple products and reuse categories over time. Classification of post-release, failure-prone files using change data for the Eclipse product line gives better recall and false positive rates as compared to classification using static code metrics. The work also found that on-going change in product lines hinders the ability to predict failure-prone files, and that predicting post-release defects using pre-release change data for the Eclipse case study is difficult. For example, using more data from the past to predict future failure-prone files does not necessarily give better results than using data only from the recent past. The empirical investigation of product line change and defect data leads to an improved understanding of the interplay among change, reuse and reliability as a product line evolves

Results and experiences from an empirical study of fault reports in industrial projects

Abstract. Faults introduced into systems during development are costly to fix, and especially so for business-critical systems. These systems are developed using common development practices, but have high requirements for dependability. This paper reports on an ongoing investigation of fault reports from Norwegian IT companies, where the aim is to seek a better understanding on faults that have been found during development and how this may affect the quality of the system. Our objective in this paper is to investigate the fault profiles of four business-critical commercial projects to explore if there are differences in the way faults appear in different systems. We have conducted an empirical study by collecting fault reports from several industrial projects, comparing findings from projects where components and reuse have been core strategies with more traditional development projects. Findings show that some specific fault types are generally dominant across reports from all projects, and that some fault types are rated as more severe than others. 1