The Co-Evolution of Test Maintenance and Code Maintenance through the lens of Fine-Grained Semantic Changes

Automatic testing is a widely adopted technique for improving software quality. Software developers add, remove and update test methods and test classes as part of the software development process as well as during the evolution phase, following the initial release. In this work we conduct a large scale study of 61 popular open source projects and report the relationships we have established between test maintenance, production code maintenance, and semantic changes (e.g, statement added, method removed, etc.). performed in developers' commits. We build predictive models, and show that the number of tests in a software project can be well predicted by employing code maintenance profiles (i.e., how many commits were performed in each of the maintenance activities: corrective, perfective, adaptive). Our findings also reveal that more often than not, developers perform code fixes without performing complementary test maintenance in the same commit (e.g., update an existing test or add a new one). When developers do perform test maintenance, it is likely to be affected by the semantic changes they perform as part of their commit. Our work is based on studying 61 popular open source projects, comprised of over 240,000 commits consisting of over 16,000,000 semantic change type instances, performed by over 4,000 software engineers.Comment: postprint, ICSME 201

The temporality of technical debt introduction on new code and confounding factors

Code Technical Debt (TD) is intentionally or unintentionally created when developers introduce inefficiencies in the codebase. This can be attributed to various reasons such as heavy workload, tight delivery schedule, or developers’ lack of experience. Since a software system grows mostly through the addition of new code, it is interesting to study how TD fluctuates along this process. Specifically, in this paper, we investigate: (a) the temporality of code TD introduction in new code, i.e., whether the introduction of TD is stable across the lifespan of the project, or if its evolution presents spikes; and (b) the relation of TD introduction to the development team’s workload in a given period, as well as to the experience of the development team. To answer these questions, we have performed a case study on 47 open-source projects from two well-known ecosystems (Apache and Eclipse) as well as additional isolated projects from GitHub (not selected from a specific ecosystem) and inspected the number of TD issues introduced in 6-month sliding temporal windows. The results of the study suggested that: (a) overall, the number of TD issues introduced through new code is a stable measure, although it presents spikes; and (b) the number of commits performed, as well as developers’ experience are not strongly correlated to the number of introduced TD issues.</p

Is "Better Data" Better than "Better Data Miners"? (On the Benefits of Tuning SMOTE for Defect Prediction)

We report and fix an important systematic error in prior studies that ranked classifiers for software analytics. Those studies did not (a) assess classifiers on multiple criteria and they did not (b) study how variations in the data affect the results. Hence, this paper applies (a) multi-criteria tests while (b) fixing the weaker regions of the training data (using SMOTUNED, which is a self-tuning version of SMOTE). This approach leads to dramatically large increases in software defect predictions. When applied in a 5*5 cross-validation study for 3,681 JAVA classes (containing over a million lines of code) from open source systems, SMOTUNED increased AUC and recall by 60% and 20% respectively. These improvements are independent of the classifier used to predict for quality. Same kind of pattern (improvement) was observed when a comparative analysis of SMOTE and SMOTUNED was done against the most recent class imbalance technique. In conclusion, for software analytic tasks like defect prediction, (1) data pre-processing can be more important than classifier choice, (2) ranking studies are incomplete without such pre-processing, and (3) SMOTUNED is a promising candidate for pre-processing.Comment: 10 pages + 2 references. Accepted to International Conference of Software Engineering (ICSE), 201

Is "Better Data" Better than "Better Data Miners"? (On the Benefits of Tuning SMOTE for Defect Prediction)

We report and fix an important systematic error in prior studies that ranked classifiers for software analytics. Those studies did not (a) assess classifiers on multiple criteria and they did not (b) study how variations in the data affect the results. Hence, this paper applies (a) multi-criteria tests while (b) fixing the weaker regions of the training data (using SMOTUNED, which is a self-tuning version of SMOTE). This approach leads to dramatically large increases in software defect predictions. When applied in a 5*5 cross-validation study for 3,681 JAVA classes (containing over a million lines of code) from open source systems, SMOTUNED increased AUC and recall by 60% and 20% respectively. These improvements are independent of the classifier used to predict for quality. Same kind of pattern (improvement) was observed when a comparative analysis of SMOTE and SMOTUNED was done against the most recent class imbalance technique. In conclusion, for software analytic tasks like defect prediction, (1) data pre-processing can be more important than classifier choice, (2) ranking studies are incomplete without such pre-processing, and (3) SMOTUNED is a promising candidate for pre-processing.Comment: 10 pages + 2 references. Accepted to International Conference of Software Engineering (ICSE), 201

A Novel Machine Learning Approach for Bug Prediction

AbstractWith the growing complexities of the software, the number of potential bugs is also increasing rapidly. These bugs hinder the rapid software development cycle. Bugs, if left unresolved, might cause problems in the long run. Also, without any prior knowledge about the location and the number of bugs, managers may not be able to allocate resources in an efficient way. In order to overcome this problem, researchers have devised numerous bug prediction approaches so far. The problem with the existing models is that the researchers have not been able to arrive at an optimized set of metrics. So, in this paper, we make an attempt to select the minimal number of best performing metrics, thereby keeping the model both simple and accurate at the same time. Most of the bug prediction models use regression for prediction and since regression is a technique to best approximate the training data set, the approximations don’t always fit well with the test data set. Keeping this in mind, we propose an algorithm to predict the bug proneness index using marginal R square values. Though regressions are performed as intermediary steps in this algorithm, the underlying logic is different in nature when compared with the models using regressions alone

A Quantitative Analysis of Open Source Software Code Quality: Insights from Metric Distributions

Code quality is a crucial construct in open-source software (OSS) with three dimensions: maintainability, reliability, and functionality. To accurately measure them, we divide 20 distinct metrics into two types: 1) threshold-type metrics that influence code quality in a monotonic manner; 2) non-threshold-type metrics that lack a monotonic relationship to evaluate. We propose a distribution-based method to provide scores for metrics, which demonstrates great explainability on OSS adoption. Our empirical analysis includes more than 36,460 OSS projects and their raw metrics from SonarQube and CK. Our work contributes to the understanding of the multi-dimensional construct of code quality and its metric measurements

Too Trivial To Test? An Inverse View on Defect Prediction to Identify Methods with Low Fault Risk

Background. Test resources are usually limited and therefore it is often not possible to completely test an application before a release. To cope with the problem of scarce resources, development teams can apply defect prediction to identify fault-prone code regions. However, defect prediction tends to low precision in cross-project prediction scenarios. Aims. We take an inverse view on defect prediction and aim to identify methods that can be deferred when testing because they contain hardly any faults due to their code being "trivial". We expect that characteristics of such methods might be project-independent, so that our approach could improve cross-project predictions. Method. We compute code metrics and apply association rule mining to create rules for identifying methods with low fault risk. We conduct an empirical study to assess our approach with six Java open-source projects containing precise fault data at the method level. Results. Our results show that inverse defect prediction can identify approx. 32-44% of the methods of a project to have a low fault risk; on average, they are about six times less likely to contain a fault than other methods. In cross-project predictions with larger, more diversified training sets, identified methods are even eleven times less likely to contain a fault. Conclusions. Inverse defect prediction supports the efficient allocation of test resources by identifying methods that can be treated with less priority in testing activities and is well applicable in cross-project prediction scenarios.Comment: Submitted to PeerJ C