The Nature of evidence in empirical software engineering

In this paper, we argue that the gap between empirical software engineering and software engineering practice might be lessened if more attention were paid to two important aspects of evidence. The first is that evidence from case or field studies of actual software engineering practice is essential in order to understand and inform that practice. The second is that the nature of evidence should fit the purpose to which the evidence is going to be put. One type of evidence is not per se better than another. For example, the evidence required to persuade a manager to change an aspect of practice might be totally different in nature from that required to deepen the academic community's understanding of such practice

Why project size matters for contract choice in software development outsourcing

The contractual mechanism of software development outsourcing, typically either fixed-price (FP) or time-and-materials (T&M), determines the nature of incentives, risk sharing, and coordination between client and vendor. While software engineering considers project size as crucial for project planning and success, neither economic nor organizational theory considers size per se among the determinants of contract choice. In this paper, we address the gap between the centrality of project size in the software engineering literature and the attention it receives in software contracting research by modeling and testing the association between project size and contract choice. Existing empirical evidence indicates that FP contracts are appropriate for small development efforts whereas T&M contracts are suitable for larger projects, based on the reasoning that cost and schedule are difficult to estimate in larger projects. This prediction that size is directly associated with contract choice is the basis upon which two models are developed. The first model draws on the contracting efficiency approach to hypothesize that the effect of project size on contract choice is mediated by project detail. The second model draws on the contingency approach to software development risk management to hypothesize that the effect of project size on contract choice is moderated by project detail and vendor familiarity. We test these models using a large portfolio of software development contracts entered into by a leading European bank, and the results confirm that both mediation and moderation are at play

Evolution of statistical analysis in empirical software engineering research: Current state and steps forward

Software engineering research is evolving and papers are increasingly based on empirical data from a multitude of sources, using statistical tests to determine if and to what degree empirical evidence supports their hypotheses. To investigate the practices and trends of statistical analysis in empirical software engineering (ESE), this paper presents a review of a large pool of papers from top-ranked software engineering journals. First, we manually reviewed 161 papers and in the second phase of our method, we conducted a more extensive semi-automatic classification of papers spanning the years 2001--2015 and 5,196 papers. Results from both review steps was used to: i) identify and analyze the predominant practices in ESE (e.g., using t-test or ANOVA), as well as relevant trends in usage of specific statistical methods (e.g., nonparametric tests and effect size measures) and, ii) develop a conceptual model for a statistical analysis workflow with suggestions on how to apply different statistical methods as well as guidelines to avoid pitfalls. Lastly, we confirm existing claims that current ESE practices lack a standard to report practical significance of results. We illustrate how practical significance can be discussed in terms of both the statistical analysis and in the practitioner's context.Comment: journal submission, 34 pages, 8 figure

A secondary analyses of Bradac et al. s prototype process-monitoring experiment

We report on the secondary analyses of some conjectures and empirical evidence presented in Bradac et al. s prototype process-monitoring experiment, published previously in IEEE Transactions on Software Engineering. We identify 13 conjectures in the original paper, and re-analyse six of these conjectures using the original evidence. Rather than rejecting any of the original conjectures, we identify assumptions underlying those conjectures, identify alternative interpretations of the conjectures, and also propose a number of new conjectures. Bradac et al. s study focused on reducing the project schedule interval. Some of our re-analysis has--considered improving software quality. We note that our analyses were only possible because of the quality and quantity of evidence presented in the original paper. Reflecting on our analyses leads us to speculate about the value of descriptive papers --that seek to present empirical material (together with an explicit statement of goals, assumptions and constraints) separate from the analyses that proceeds from that material. Such descriptive papers could improve the public scrutiny of software engineering research and may respond, in part, to some researchers criticisms concerning the small amount of software engineering research that is actually--evaluated. We also consider opportunities for further research, in particular opportunities for relating individual actions to project outcomes

How do software architects consider non-functional requirements: an exploratory study

© 2012 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Dealing with non-functional requirements (NFRs) has posed a challenge onto software engineers for many years. Over the years, many methods and techniques have been proposed to improve their elicitation, documentation, and validation. Knowing more about the state of the practice on these topics may benefit both practitioners' and researchers' daily work. A few empirical studies have been conducted in the past, but none under the perspective of software architects, in spite of the great influence that NFRs have on daily architects' practices. This paper presents some of the findings of an empirical study based on 13 interviews with software architects. It addresses questions such as: who decides the NFRs, what types of NFRs matter to architects, how are NFRs documented, and how are NFRs validated. The results are contextualized with existing previous work.Peer ReviewedPostprint (author’s final draft

Exploring the eradication of code smells: An empirical and theoretical perspective

This article has been made available through the Brunel Open Access Publishing Fund - Copyright @ 2010 Hindawi Publishing CorporationCode smells reflect code decay, and, as such, developers should seek to eradicate such smells through application of “deodorant” in the form of one or more refactorings. However, a relative lack of studies exploring code smells either theoretically or empirically when compared with literature on refactoring suggests that there are reasons why smell eradication is neither being applied in anger, nor the subject of significant research. In this paper, we present three studies as supporting evidence for this stance. The first is an analysis of a set of five, open-source Java systems in which we show very little tendency for smells to be eradicated by developers; the second is an empirical study of a subsystem of a proprietary, C# web-based application where practical problems arise in smell identification and the third, a theoretical enumeration of smell-related refactorings to suggest why smells may be left alone from an effort perspective. Key findings of the study were that first, smells requiring application of simple refactorings were eradicated in favour of smells requiring more complex refactorings; second, a wide range of conflicts and anomalies soon emerged when trying to identify smelly code; an interesting result with respect to comment lines was also observed. Finally, perceived (estimated) effort to eradicate a smell may be a key factor in explaining why smell eradication is avoided by developers. The study thus highlights the need for a clearer research strategy on the issue of code smells and all aspects of their identification and measurement.The research in this paper was supported by a grant from the UK Engineering and Physical Sciences Research Council (EPSRC) (Grant no: EP/G031126/1

A literature review of expert problem solving using analogy

We consider software project cost estimation from a problem solving perspective. Taking a cognitive psychological approach, we argue that the algorithmic basis for CBR tools is not representative of human problem solving and this mismatch could account for inconsistent results. We describe the fundamentals of problem solving, focusing on experts solving ill-defined problems. This is supplemented by a systematic literature review of empirical studies of expert problem solving of non-trivial problems. We identified twelve studies. These studies suggest that analogical reasoning plays an important role in problem solving, but that CBR tools do not model this in a biologically plausible way. For example, the ability to induce structure and therefore find deeper analogies is widely seen as the hallmark of an expert. However, CBR tools fail to provide support for this type of reasoning for prediction. We conclude this mismatch between experts’ cognitive processes and software tools contributes to the erratic performance of analogy-based prediction