Reliability and validity in comparative studies of software prediction models

Empirical studies on software prediction models do not converge with respect to the question "which prediction model is best?" The reason for this lack of convergence is poorly understood. In this simulation study, we have examined a frequently used research procedure comprising three main ingredients: a single data sample, an accuracy indicator, and cross validation. Typically, these empirical studies compare a machine learning model with a regression model. In our study, we use simulation and compare a machine learning and a regression model. The results suggest that it is the research procedure itself that is unreliable. This lack of reliability may strongly contribute to the lack of convergence. Our findings thus cast some doubt on the conclusions of any study of competing software prediction models that used this research procedure as a basis of model comparison. Thus, we need to develop more reliable research procedures before we can have confidence in the conclusions of comparative studies of software prediction models

An empirical study of evolution of inheritance in Java OSS

Previous studies of Object-Oriented (OO) software have reported avoidance of the inheritance mechanism and cast doubt on the wisdom of ‘deep’ inheritance levels. From an evolutionary perspective, the picture is unclear - we still know relatively little about how, over time, changes tend to be applied by developers. Our conjecture is that an inheritance hierarchy will tend to grow ‘breadth-wise’ rather than ‘depth-wise’. This claim is made on the basis that developers will avoid extending depth in favour of breadth because of the inherent complexity of having to understand the functionality of superclasses. Thus the goal of our study is to investigate this empirically. We conduct an empirical study of seven Java Open-Source Systems (OSSs) over a series of releases to observe the nature and location of changes within the inheritance hierarchies. Results show a strong tendency for classes to be added at levels one and two of the hierarchy (rather than anywhere else). Over 96% of classes added over the course of the versions of all systems were at level 1 or level 2. The results suggest that changes cluster in the shallow levels of a hierarchy; this is relevant for developers since it indicates where remedial activities such as refactoring should be focused

How reliable are systematic reviews in empirical software engineering?

BACKGROUND – the systematic review is becoming a more commonly employed research instrument in empirical software engineering. Before undue reliance is placed on the outcomes of such reviews it would seem useful to consider the robustness of the approach in this particular research context. OBJECTIVE – the aim of this study is to assess the reliability of systematic reviews as a research instrument. In particular we wish to investigate the consistency of process and the stability of outcomes. METHOD – we compare the results of two independent reviews under taken with a common research question. RESULTS – the two reviews find similar answers to the research question, although the means of arriving at those answers vary. CONCLUSIONS – in addressing a well-bounded research question, groups of researchers with similar domain experience can arrive at the same review outcomes, even though they may do so in different ways. This provides evidence that, in this context at least, the systematic review is a robust research method

Class movement and re-location: An empirical study of Java inheritance evolution

This is the post-print version of the final paper published in Journal of Systems and Software. The published article is available from the link below. Changes resulting from the publishing process, such as peer review, editing, corrections, structural formatting, and other quality control mechanisms may not be reflected in this document. Changes may have been made to this work since it was submitted for publication. Copyright @ 2009 Elsevier B.V.Inheritance is a fundamental feature of the Object-Oriented (OO) paradigm. It is used to promote extensibility and reuse in OO systems. Understanding how systems evolve, and specifically, trends in the movement and re-location of classes in OO hierarchies can help us understand and predict future maintenance effort. In this paper, we explore how and where new classes were added as well as where existing classes were deleted or moved across inheritance hierarchies from multiple versions of four Java systems. We observed first, that in one of the studied systems the same set of classes was continuously moved across the inheritance hierarchy. Second, in the same system, the most frequent changes were restricted to just one sub-part of the overall system. Third, that a maximum of three levels may be a threshold when using inheritance in a system; beyond this level very little activity was observed, supporting earlier theories that, beyond three levels, complexity becomes overwhelming. We also found evidence of ‘collapsing’ hierarchies to bring classes up to shallower levels. Finally, we found that larger classes and highly coupled classes were more frequently moved than smaller and less coupled classes. Statistical evidence supported the view that larger classes and highly coupled classes were less cohesive than smaller classes and lowly coupled classes and were thus more suitable candidates for being moved (within an hierarchy)

Software defect prediction: do different classifiers find the same defects?

Open Access: This article is distributed under the terms of the Creative Commons Attribution 4.0 International License CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.During the last 10 years, hundreds of different defect prediction models have been published. The performance of the classifiers used in these models is reported to be similar with models rarely performing above the predictive performance ceiling of about 80% recall. We investigate the individual defects that four classifiers predict and analyse the level of prediction uncertainty produced by these classifiers. We perform a sensitivity analysis to compare the performance of Random Forest, Naïve Bayes, RPart and SVM classifiers when predicting defects in NASA, open source and commercial datasets. The defect predictions that each classifier makes is captured in a confusion matrix and the prediction uncertainty of each classifier is compared. Despite similar predictive performance values for these four classifiers, each detects different sets of defects. Some classifiers are more consistent in predicting defects than others. Our results confirm that a unique subset of defects can be detected by specific classifiers. However, while some classifiers are consistent in the predictions they make, other classifiers vary in their predictions. Given our results, we conclude that classifier ensembles with decision-making strategies not based on majority voting are likely to perform best in defect prediction.Peer reviewedFinal Published versio

Symmetry breaking in commensurate graphene rotational stacking; a comparison of theory and experiment

Graphene stacked in a Bernal configuration (60 degrees relative rotations between sheets) differs electronically from isolated graphene due to the broken symmetry introduced by interlayer bonds forming between only one of the two graphene unit cell atoms. A variety of experiments have shown that non-Bernal rotations restore this broken symmetry; consequently, these stacking varieties have been the subject of intensive theoretical interest. Most theories predict substantial changes in the band structure ranging from the development of a Van Hove singularity and an angle dependent electron localization that causes the Fermi velocity to go to zero as the relative rotation angle between sheets goes to zero. In this work we show by direct measurement that non-Bernal rotations preserve the graphene symmetry with only a small perturbation due to weak effective interlayer coupling. We detect neither a Van Hove singularity nor any significant change in the Fermi velocity. These results suggest significant problems in our current theoretical understanding of the origins of the band structure of this material.Comment: 7 pages, 6 figures, submitted to PR

A wide band gap metal-semiconductor-metal nanostructure made entirely from graphene

A blueprint for producing scalable digital graphene electronics has remained elusive. Current methods to produce semiconducting-metallic graphene networks all suffer from either stringent lithographic demands that prevent reproducibility, process-induced disorder in the graphene, or scalability issues. Using angle resolved photoemission, we have discovered a unique one dimensional metallic-semiconducting-metallic junction made entirely from graphene, and produced without chemical functionalization or finite size patterning. The junction is produced by taking advantage of the inherent, atomically ordered, substrate-graphene interaction when it is grown on SiC, in this case when graphene is forced to grow over patterned SiC steps. This scalable bottomup approach allows us to produce a semiconducting graphene strip whose width is precisely defined within a few graphene lattice constants, a level of precision entirely outside modern lithographic limits. The architecture demonstrated in this work is so robust that variations in the average electronic band structure of thousands of these patterned ribbons have little variation over length scales tens of microns long. The semiconducting graphene has a topologically defined few nanometer wide region with an energy gap greater than 0.5 eV in an otherwise continuous metallic graphene sheet. This work demonstrates how the graphene-substrate interaction can be used as a powerful tool to scalably modify graphene's electronic structure and opens a new direction in graphene electronics research.Comment: 11 pages, 7 figure

A Comprehensive Guide to Fuels Treatment Practices for Ponderosa Pine in the Black Hills, Colorado Front Range, and Southwest

The objective of this paper is to present recommendations for fuels treatments in ponderosa pine forests in the Southwest, Colorado Front Range, and Black Hills of South Dakota. We have synthesized existing knowledge from the peer-reviewed literature and administrative studies and acquired local knowledge through a series of discussions with fuels treatment practitioners. We describe specific treatments, the circumstances under which they can be applied, and treatment effects. We provide recommendations related to where, how, and how often fuels treatments may be prescribed to achieve desired outcomes. Desired outcomes address social, political, economic, and ecological factors

A Comprehensive Guide to Fuels Treatment Practices for Ponderosa Pine in the Black Hills, Colorado Front Range, and Southwest

The objective of this paper is to present recommendations for fuels treatments in ponderosa pine forests in the Southwest, Colorado Front Range, and Black Hills of South Dakota. We have synthesized existing knowledge from the peer-reviewed literature and administrative studies and acquired local knowledge through a series of discussions with fuels treatment practitioners. We describe specific treatments, the circumstances under which they can be applied, and treatment effects. We provide recommendations related to where, how, and how often fuels treatments may be prescribed to achieve desired outcomes. Desired outcomes address social, political, economic, and ecological factors

Silicon intercalation into the graphene-SiC interface

In this work we use LEEM, XPEEM and XPS to study how the excess Si at the graphene-vacuum interface reorders itself at high temperatures. We show that silicon deposited at room temperature onto multilayer graphene films grown on the SiC(000[`1]) rapidly diffuses to the graphene-SiC interface when heated to temperatures above 1020. In a sequence of depositions, we have been able to intercalate ~ 6 ML of Si into the graphene-SiC interface.Comment: 6 pages, 8 figures, submitted to PR