Conformance Factor in Test-driven Development: Initial Results from an Enhanced Replication

Test-driven development (TDD) is an iterative software development technique where unit-tests are defined before production code. The proponents of TDD claim that it improves both external quality and developers’ productivity. In particular, Erdogmus et al. (i.e., original study) proposed a two-stage model to investigate these claims regarding TDD’s effects. Our aim is to enhance the model proposed in the original study by investigating an additional factor: TDD process conformance. We conducted a close, external replication of the original study accompanied by a correlation analysis to check whether process conformance is related to improvements for the subjects using TDD. We partially confirmed the results of the original study. Moreover, we observed a correlation between process conformance and quality, but not productivity. We found no evidence to support the claim that external quality and productivity are improved by the adoption of TDD compared to test-last development. Finally, conformance to TDD process improves the quality and does not affect productivity. We conclude that the role of process conformance is relevant in studying the quality and productivity-related effects of TDD

Impact of process conformance on the effects of test-driven development

Context: One limitation of the empirical studies about test-driven development (TDD) is knowing whether the developers followed the advocated test-code-refactor cycle. Research dealt with the issue of process conformance only in terms of internal validity, while investigating the role of other confounding variables that might explain the controversial effects of TDD. None of the research included process conformance as a fundamental part of the analysis. Goal: We aim to examine the impact of process conformance on the claimed effects of TDD on external quality, developers’ productivity and test quality. Method: We used data collected during a previous study to create regression models in which the level of process conformance was used to predict external quality, productivity, and tests thoroughness. Result: Based on our analysis of the available data (n = 22), we observe that neither quality (p – value = 0.21), productivity (p – value = 0.80), number of tests (p – value = 0.39) nor coverage (p – value = 0.09) was correlated with the level of TDD process conformance. Conclusion: While based on a small sample, we raise concerns about how TDD is interpreted. We also question whether the cost of strictly following TDD will pay-off in terms of external quality, productivity, and tests thoroughness.This research has been supported in part by the Academy of Finland with decision/grant no: 278354

On the effects of programming and testing skills on external quality and productivity in a test-driven development context

Background: In previous studies, a model was proposed that investigated how the developers’ unit testing effort impacted their productivity as well as the external quality of the software they developed. Goal: The aim of this study is to enhance the proposed model by considering two additional factors related to the expertise of developers: programming and unit testing skills. The possibility of including such skills in a model that represents the relationship that testing effort has with the developer’s productivity and the product’s external quality was investigated. Method: Data collected from a test-first development task in academic setting was used in order to gauge the relationship between testing effort, external quality, and productivity. Furthermore, Analysis of Covariance (ANCOVA) was utilized to check the impact of developers’ skills on productivity and quality. Result: The results obtained in previous studies were confirmed: there exists a positive effect of testing effort on productivity, but not on quality. Moreover, the developers’ skills have an impact on productivity but none on external quality. Conclusion: Productivity improves with testing effort, a result consistent across previous, similar studies. The role of existing skills is a relevant factor in studying the effects of developers’ unit testing effort on productivity. Nevertheless, more investigations are needed regarding the relationship between unit testing effort and external quality.This research is partially supported by the Academy of Finland with decision no: 278354. The rst author would like to acknowledge ISACA Finland Chapter for the support provided to complete this work

An external replication on the effects of test-driven development using a multi-site blind analysis approach

Context: Test-driven development (TDD) is an agile practice claimed to improve the quality of a software product, as well as the productivity of its developers. A previous study (i.e., baseline experiment) at the University of Oulu (Finland) compared TDD to a test-last development (TLD) approach through a randomized controlled trial. The results failed to support the claims. Goal: We want to validate the original study results by replicating it at the University of Basilicata (Italy), using a different design. Method: We replicated the baseline experiment, using a crossover design, with 21 graduate students. We kept the settings and context as close as possible to the baseline experiment. In order to limit researchers bias, we involved two other sites (UPM, Spain, and Brunel, UK) to conduct blind analysis of the data. Results: The Kruskal-Wallis tests did not show any significant difference between TDD and TLD in terms of testing effort (p-value = .27), external code quality (p-value = .82), and developers' productivity (p-value = .83). Nevertheless, our data revealed a difference based on the order in which TDD and TLD were applied, though no carry over effect. Conclusions: We verify the baseline study results, yet our results raises concerns regarding the selection of experimental objects, particularly with respect to their interaction with the order in which of treatments are applied. We recommend future studies to survey the tasks used in experiments evaluating TDD. Finally, to lower the cost of replication studies and reduce researchers' bias, we encourage other research groups to adopt similar multi-site blind analysis approach described in this paper.This research is supported in part by the Academy of Finland Project 278354

Towards an operationalization of test-driven development skills: An industrial empirical study

Abstract Context: The majority of the empirical studies on Test-driven development (TDD) are concerned with verifying or refuting the effectiveness of the technique over a traditional approach, and they tend to neglect whether the subjects possess the necessary skills to apply TDD, though they argue such skills are necessary. Objective: We evaluate a set of minimal, a priori and in process skills necessary to apply TDD. We determine whether variations in external quality (i.e., number of defects) and productivity (i.e., number of features implemented) can be associated with different clusters of the TDD skills? set. Method: We executed a quasi-experiment involving 30 practitioners from industry. We first grouped the participants according to their TDD skills? set (consisting of a priori experience on programming and testing as well as in-process TDD conformance) into three levels (Low-Medium-High) using k-means clustering. We then applied ANOVA to compare the clusters in terms of external quality and productivity, and conducted post-hoc pairwise analysis. Results: We did not observe a statistically significant difference between the clusters either for external software quality ( F ( 2 , 27 = 1.44 , p = . 260 ), or productivity ( F ( 2 , 27 ) = 3.02 , p = . 065 ). However, the analysis of the effect sizes and their confidence intervals shows that the TDD skills? set is a factor that could account for up to 28% of the external quality, and 38% for productivity. Conclusion: We have reason to conclude that focusing on the improvement of TDD skills? set investigated in this study could benefit software developers in improving their baseline productivity and the external quality of the code they produce. However, replications are needed to overcome the issues related with the statistical power of this study. We suggest practical insights for future work to investigate the phenomenon further.hisresearchispartiallysupportedbytheAcademyofFinlandwithdecisionno.:278354,andbyFinnishDistinguishedProfessor(Fi.Di.Pro.) programme, ESEIL

Naming the pain in requirements engineering : Contemporary problems, causes, and effects in practice

Requirements Engineering (RE) has received much attention in research and practice due to its importance to software project success. Its interdisciplinary nature, the dependency to the customer, and its inherent uncertainty still render the discipline difficult to investigate. This results in a lack of empirical data. These are necessary, however, to demonstrate which practically relevant RE problems exist and to what extent they matter. Motivated by this situation, we initiated the Naming the Pain in Requirements Engineering (NaPiRE) initiative which constitutes a globally distributed, bi-yearly replicated family of surveys on the status quo and problems in practical RE. In this article, we report on the qualitative analysis of data obtained from 228 companies working in 10 countries in various domains and we reveal which contemporary problems practitioners encounter. To this end, we analyse 21 problems derived from the literature with respect to their relevance and criticality in dependency to their context, and we complement this picture with a cause-effect analysis showing the causes and effects surrounding the most critical problems. Our results give us a better understanding of which problems exist and how they manifest themselves in practical environments. Thus, we provide a first step to ground contributions to RE on empirical observations which, until now, were dominated by conventional wisdom only.Peer reviewe

An industry experiment on the effects of test-driven development on external quality and productivity

Existing empirical studies on test-driven development (TDD) report different conclusions about its effects on quality and productivity. Very few of those studies are experiments conducted with software professionals in industry. We aim to analyse the effects of TDD on the external quality of the work done and the productivity of developers in an industrial setting. We conducted an experiment with 24 professionals from three different sites of a software organization. We chose a repeated-measures design, and asked subjects to implement TDD and incremental test last development (ITLD) in two simple tasks and a realistic application close to real-life complexity. To analyse our findings, we applied a repeated-measures general linear model procedure and a linear mixed effects procedure. We did not observe a statistical difference between the quality of the work done by subjects in both treatments. We observed that the subjects are more productive when they implement TDD on a simple task compared to ITLD, but the productivity drops significantly when applying TDD to a complex brownfield task. So, the task complexity significantly obscured the effect of TDD. Further evidence is necessary to conclude whether TDD is better or worse than ITLD in terms of external quality and productivity in an industrial setting. We found that experimental factors such as selection of tasks could dominate the findings in TDD studies.This research has been partly funded by Spanish Ministry of Science and Innovation projects TIN2011-23216, the Distinguished Professor Program of Tekes, and the Academy of Finland (Grant Decision No. 260871)

Inferencing into the void: problems with implicit populations Comments on `Empirical software engineering experts on the use of students and professionals in experiments'

I welcome the contribution from Falessi et al. [1] hereafter referred to as F++ , and the ensuing debate. Experimentation is an important tool within empirical software engineering, so how we select participants is clearly a relevant question. Moreover as F++ point out, the question is considerably more nuanced than the simple dichotomy it might appear to be at first sight. This commentary is structured as follows. In Section 2 I briefly summarise the arguments of F++ and comment on their approach. Next, in Section 3, I take a step back to consider the nature of representativeness in inferential arguments and the need for careful definition. Then I give three examples of using different types of participant to consider impact. I conclude by arguing, largely in agreement with F++, that the question of whether student participants are representative or not depends on the target population. However, we need to give careful consideration to defining that population and, in particular, not to overlook the representativeness of tasks and environment. This is facilitated by explicit description of the target populations.Comment: This is a commentary on the article in Empirical Software Engineering entitled `Empirical software engineering experts on the use of students and professionals in experiments' by Falessi et al. It will form part of an accepted an article for EMSE (Dec 2018) comprising commentary and a response from the original author

Software Startups -- A Research Agenda

Software startup companies develop innovative, software-intensive products within limited time frames and with few resources, searching for sustainable and scalable business models. Software startups are quite distinct from traditional mature software companies, but also from micro-, small-, and medium-sized enterprises, introducing new challenges relevant for software engineering research. This paper's research agenda focuses on software engineering in startups, identifying, in particular, 70+ research questions in the areas of supporting startup engineering activities, startup evolution models and patterns, ecosystems and innovation hubs, human aspects in software startups, applying startup concepts in non-startup environments, and methodologies and theories for startup research. We connect and motivate this research agenda with past studies in software startup research, while pointing out possible future directions. While all authors of this research agenda have their main background in Software Engineering or Computer Science, their interest in software startups broadens the perspective to the challenges, but also to the opportunities that emerge from multi-disciplinary research. Our audience is therefore primarily software engineering researchers, even though we aim at stimulating collaborations and research that crosses disciplinary boundaries. We believe that with this research agenda we cover a wide spectrum of the software startup industry current needs