Results from an ethnographically-informed study in the context of test driven development

Background: Test-driven development (TDD) is an iterative software development technique where unit tests are defined before production code. Previous studies fail to analyze the values, beliefs, and assumptions that inform and shape TDD. Aim: We designed and conducted a qualitative study to understand the values, beliefs, and assumptions of TDD. In particular, we sought to understand how novice and professional software developers, arranged in pairs (a driver and a pointer), perceive and apply TDD. Method: 14 novice software developers, i.e., graduate students in Computer Science at the University of Basilicata, and six professional software developers (with one to 10 years work experience) participated in our ethnographicallyinformed study. We asked the participants to implement a new feature for an existing software written in Java. We immersed ourselves in the context of the study, and collected data by means of contemporaneous field notes, audio recordings, and other artifacts. Results: A number of insights emerge from our analysis of the collected data, the main ones being: (i) refactoring (one of the phases of TDD) is not performed as often as the process requires and it is considered less important than other phases, (ii) the most important phase is implementation, (iii) unit tests are almost never up-to-date, (iv) participants first build a sort of mental model of the source code to be implemented and only then write test cases on the basis of this model; and (v) apart from minor differences, professional developers and students applied TDD in a similar fashion. Conclusions: Developers write quick-and-dirty production code to pass the tests and ignore refactoring.Copyright is held by the owner/auther(s)

Impact of process conformance on the effects of test-driven development

Context: One limitation of the empirical studies about test-driven development (TDD) is knowing whether the developers followed the advocated test-code-refactor cycle. Research dealt with the issue of process conformance only in terms of internal validity, while investigating the role of other confounding variables that might explain the controversial effects of TDD. None of the research included process conformance as a fundamental part of the analysis. Goal: We aim to examine the impact of process conformance on the claimed effects of TDD on external quality, developers’ productivity and test quality. Method: We used data collected during a previous study to create regression models in which the level of process conformance was used to predict external quality, productivity, and tests thoroughness. Result: Based on our analysis of the available data (n = 22), we observe that neither quality (p – value = 0.21), productivity (p – value = 0.80), number of tests (p – value = 0.39) nor coverage (p – value = 0.09) was correlated with the level of TDD process conformance. Conclusion: While based on a small sample, we raise concerns about how TDD is interpreted. We also question whether the cost of strictly following TDD will pay-off in terms of external quality, productivity, and tests thoroughness.This research has been supported in part by the Academy of Finland with decision/grant no: 278354

Conformance Factor in Test-driven Development: Initial Results from an Enhanced Replication

Test-driven development (TDD) is an iterative software development technique where unit-tests are defined before production code. The proponents of TDD claim that it improves both external quality and developers’ productivity. In particular, Erdogmus et al. (i.e., original study) proposed a two-stage model to investigate these claims regarding TDD’s effects. Our aim is to enhance the model proposed in the original study by investigating an additional factor: TDD process conformance. We conducted a close, external replication of the original study accompanied by a correlation analysis to check whether process conformance is related to improvements for the subjects using TDD. We partially confirmed the results of the original study. Moreover, we observed a correlation between process conformance and quality, but not productivity. We found no evidence to support the claim that external quality and productivity are improved by the adoption of TDD compared to test-last development. Finally, conformance to TDD process improves the quality and does not affect productivity. We conclude that the role of process conformance is relevant in studying the quality and productivity-related effects of TDD

On the effects of programming and testing skills on external quality and productivity in a test-driven development context

Background: In previous studies, a model was proposed that investigated how the developers’ unit testing effort impacted their productivity as well as the external quality of the software they developed. Goal: The aim of this study is to enhance the proposed model by considering two additional factors related to the expertise of developers: programming and unit testing skills. The possibility of including such skills in a model that represents the relationship that testing effort has with the developer’s productivity and the product’s external quality was investigated. Method: Data collected from a test-first development task in academic setting was used in order to gauge the relationship between testing effort, external quality, and productivity. Furthermore, Analysis of Covariance (ANCOVA) was utilized to check the impact of developers’ skills on productivity and quality. Result: The results obtained in previous studies were confirmed: there exists a positive effect of testing effort on productivity, but not on quality. Moreover, the developers’ skills have an impact on productivity but none on external quality. Conclusion: Productivity improves with testing effort, a result consistent across previous, similar studies. The role of existing skills is a relevant factor in studying the effects of developers’ unit testing effort on productivity. Nevertheless, more investigations are needed regarding the relationship between unit testing effort and external quality.This research is partially supported by the Academy of Finland with decision no: 278354. The rst author would like to acknowledge ISACA Finland Chapter for the support provided to complete this work

An external replication on the effects of test-driven development using a multi-site blind analysis approach

Context: Test-driven development (TDD) is an agile practice claimed to improve the quality of a software product, as well as the productivity of its developers. A previous study (i.e., baseline experiment) at the University of Oulu (Finland) compared TDD to a test-last development (TLD) approach through a randomized controlled trial. The results failed to support the claims. Goal: We want to validate the original study results by replicating it at the University of Basilicata (Italy), using a different design. Method: We replicated the baseline experiment, using a crossover design, with 21 graduate students. We kept the settings and context as close as possible to the baseline experiment. In order to limit researchers bias, we involved two other sites (UPM, Spain, and Brunel, UK) to conduct blind analysis of the data. Results: The Kruskal-Wallis tests did not show any significant difference between TDD and TLD in terms of testing effort (p-value = .27), external code quality (p-value = .82), and developers' productivity (p-value = .83). Nevertheless, our data revealed a difference based on the order in which TDD and TLD were applied, though no carry over effect. Conclusions: We verify the baseline study results, yet our results raises concerns regarding the selection of experimental objects, particularly with respect to their interaction with the order in which of treatments are applied. We recommend future studies to survey the tasks used in experiments evaluating TDD. Finally, to lower the cost of replication studies and reduce researchers' bias, we encourage other research groups to adopt similar multi-site blind analysis approach described in this paper.This research is supported in part by the Academy of Finland Project 278354

Results of the fifth international spectroradiometer comparison for improved solar spectral irradiance measurements and related impact on reference solar cell calibration

This paper reports on the results of the fifth spectral irradiance measurement intercomparison and the impact these results have on the spread of spectral mismatch calculations in the outdoor characterization of reference solar cell and photovoltaic (PV) devices. Ten laboratories and commercial partners with their own instruments were involved in the comparison. Solar spectral irradiance in clear sky condition was measured with both fast fixed and slow rotating grating spectroradiometers. This paper describes the intercomparison campaign, describes different statistical analysis used on acquired data, reports on the results, and analyzes the impact these results would have on the primary calibration of a c-Si PV reference cell under natural sunlight

Students’ and Professionals’ Perceptions of Test-driven Development: A Focus Group Study

We have conducted a qualitative investigation on test-driven development (TDD) with focus groups to develop insights on the opinions of developers using TDD regarding the unintuitive process involved, its claimed effects, as well as the context factors that can facilitate (or hinder) its application. In particular, we conducted two focus group sessions: one with professionals and another with Master students in Computer Science. We used thematic analysis template (TAT) method for identifying patterns, themes, and interpretations in gathered data. We obtained a number of results that can be summarized as follows: (i) applying TDD without knowing advanced unit testing techniques can be difficult; (ii) refactoring (one of the phases of TDD) is not done as often as the process requires; (iii) there is a need for live feedback to let developers understand if TDD is being applied correctly; and (iv) the usefulness of TDD hinges on task and domain to which it is applied to