Why Research on Test-Driven Development is Inconclusive?

[Background] Recent investigations into the effects of Test-Driven Development (TDD) have been contradictory and inconclusive. This hinders development teams to use research results as the basis for deciding whether and how to apply TDD. [Aim] To support researchers when designing a new study and to increase the applicability of TDD research in the decision-making process in the industrial context, we aim at identifying the reasons behind the inconclusive research results in TDD. [Method] We studied the state of the art in TDD research published in top venues in the past decade, and analyzed the way these studies were set up. [Results] We identified five categories of factors that directly impact the outcome of studies on TDD. [Conclusions] This work can help researchers to conduct more reliable studies, and inform practitioners of risks they need to consider when consulting research on TDD.Comment: ESEM '20: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), October 8--9, 2020, Bari, Ital

Effectiveness of test driven development and continuous integration - a case study

In this article we describe the implementation of hybrid agile practices, namely Test Driven Development (TDD) and Continuous Integration (CI) at a Dutch SME. The quality and productivity outcomes of the case study were compared to a performance baseline set by a reference case, a preceding development project of similar context, size, complexity and team. We observed that on applying TDD and CI, a higher number of defects were discovered compared to the baseline case. The team members at the Dutch SME perceived an increase in the focus on quality and test applications, while considering customer acceptance. As a result of the case study, the Dutch SME now has an infrastructure in place to further evaluate software process improvement (SPI) initiatives

Topic selection in industry experiments

This paper shares our experience with initial negotiation and topic elicitation process for conducting industry experiments in six software development organizations in Finland. The process involved interaction with company representatives in the form of both multiple group discussions and separate face-to-face meetings. Fitness criteria developed by researchers were applied to the list of generated topics to decide on a common topic. The challenges we faced include diversity of proposed topics, communication gaps, skepticism about research methods, initial disconnect between research and industry needs, and lack of prior work relationship. Lessons learned include having enough time to establish trust with partners, importance of leveraging the benefits of training and skill development that are inherent in the experimental approach, uniquely positioning the experimental approach within the landscape of other validation approaches more familiar to industrial partners, and introducing the fitness criteria early in the process

On the Effectiveness of Unit Tests in Test-driven Development

Background: Writing unit tests is one of the primary activities in test-driven development. Yet, the existing reviews report few evidence supporting or refuting the effect of this development approach on test case quality. Lack of ability and skills of developers to produce sufficiently good test cases are also reported as limitations of applying test-driven development in industrial practice. Objective: We investigate the impact of test-driven development on the effectiveness of unit test cases compared to an incremental test last development in an industrial context. Method: We conducted an experiment in an industrial setting with 24 professionals. Professionals followed the two development approaches to implement the tasks. We measure unit test effectiveness in terms of mutation score. We also measure branch and method coverage of test suites to compare our results with the literature. Results: In terms of mutation score, we have found that the test cases written for a test-driven development task have a higher defect detection ability than test cases written for an incremental test-last development task. Subjects wrote test cases that cover more branches on a test-driven development task compared to the other task. However, test cases written for an incremental test-last development task cover more methods than those written for the second task. Conclusion: Our findings are different from previous studies conducted at academic settings. Professionals were able to perform more effective unit testing with test-driven development. Furthermore, we observe that the coverage measure preferred in academic studies reveal different aspects of a development approach. Our results need to be validated in larger industrial contexts.Istanbul Technical University Scientific Research Projects (MGA-2017-40712), and the Academy of Finland (Decision No. 278354)

A Longitudinal Cohort Study on the Retainment of Test-Driven Development

Background: Test-Driven Development (TDD) is an agile software development practice, which is claimed to boost both external quality of software products and developers' productivity. Aims: We want to study (i) the TDD effects on the external quality of software products as well as the developers' productivity, and (ii) the retainment of TDD over a period of five months. Method: We conducted a (quantitative) longitudinal cohort study with 30 third year undergraduate students in Computer Science at the University of Bari in Italy. Results: The use of TDD has a statistically significant effect neither on the external quality of software products nor on the developers' productivity. However, we observed that participants using TDD produced significantly more tests than those applying a non-TDD development process and that the retainment of TDD is particularly noticeable in the amount of tests written. Conclusions: Our results should encourage software companies to adopt TDD because who practices TDD tends to write more tests---having more tests can come in handy when testing software systems or localizing faults---and it seems that novice developers retain TDD.Comment: ESEM, October 2018, Oulu, Finlan

Conformance Factor in Test-driven Development: Initial Results from an Enhanced Replication

Test-driven development (TDD) is an iterative software development technique where unit-tests are defined before production code. The proponents of TDD claim that it improves both external quality and developers’ productivity. In particular, Erdogmus et al. (i.e., original study) proposed a two-stage model to investigate these claims regarding TDD’s effects. Our aim is to enhance the model proposed in the original study by investigating an additional factor: TDD process conformance. We conducted a close, external replication of the original study accompanied by a correlation analysis to check whether process conformance is related to improvements for the subjects using TDD. We partially confirmed the results of the original study. Moreover, we observed a correlation between process conformance and quality, but not productivity. We found no evidence to support the claim that external quality and productivity are improved by the adoption of TDD compared to test-last development. Finally, conformance to TDD process improves the quality and does not affect productivity. We conclude that the role of process conformance is relevant in studying the quality and productivity-related effects of TDD

Qué factores personales afectan a la calidad y productividad de TDD?: un experimento con profesionales

Contexto: Test-Driven Developement (TDD) es una técnica de desarrollo de software ágil que es ampliamente utilizada en la industria, aunque su efectividad ha generado incertidumbre si se compara con técnicas de desarrollo tradicional. Objetivo: Estudiar la efectividad de TDD considerando el grado de influencia de distintos factores humanos. Metodología: Experimento aleatorizado (crossover 2x2) realizado con sujetos profesionales en un ámbito académico. Resultados: La calidad y productividad al aplicar TDD es algo superior a lo obtenido con el desarrollo iterativo incremental (ITLD). La edad de los participantes, la función que desempeñaban en su trabajo y el conocimiento previo de la técnica de TDD ejercen influencia sobre las variables respuesta.Sociedad Argentina de Informática e Investigación Operativ

Efectividad del Test-Driven Development: Un Experimento Replicado

Los métodos ágiles y sus prácticas asociadas, e.g.: Test-Driven Developement (TDD), son ampliamente utilizadas en la industria y han sido repetidamente sometidas a estudios empíricos. Antecedentes: Se han realizado diversos experimentos en empresas y academia acerca de TDD. En general, los experimentos no muestran un efecto positivo de TDD en la calidad del código o la productividad de los programadores. Objetivo: Replicar el experimento UPM 2014 efectuado por N. Juristo y su equipo, para reproducir sus resultados y secundariamente, estudiar el efecto de la experiencia del desarrollador en la efectividad de TDD. Método: Replicación experimental manteniendo similares el training y los materiales del experimento original. La replicación fue llevada a cabo en la Universidad de las Fuerzas Armadas ESPE sede Latacunga (ESPEL). Los sujetos experimentales fueron 17 estudiantes del Master en Ingeniería de Software. Resultados: Los resultados de la replicación confirman los efectos observados en UPM 2014. La efectividad de TDD ha resultado menor que ITL, aunque las diferencias no son significativas. La productividad y calidad del código producido por los estudiantes ESPEL cuando utilizan ITL y TDD es comparable a la de los estudiantes UPM, aunque menor en valores absolutos. Conclusiones: TDD no produce beneficios en calidad o productividad, o al menos no de forma inmediata. Parece necesario que los sujetos experimentales reciban training intensivo para que los efectos de TDD sean evidentes

Avaliação da ferramenta de testes Selenium no desenvolvimento guiado por teste de uma aplicação web

Trabalho de Conclusão de Curso (graduação)—Universidade de Brasília, Instituto de Ciências Exatas, Departamento de Ciência da Computação, 2019.O teste de software é um processo importante no desenvolvimento de software. Em algumas situações, a automação desse processo pode representar benefícios importantes para o controle de qualidade do software. Nesse contexto, existem diversas ferramentas de automação de testes que buscam facilitar a atividade de automação. O presente trabalho procura desenvolver um arcabouço para avaliação do Selenium, uma ferramenta de automação de testes de aplicações web. Para atingir esse objetivo, inicialmente alguns critérios de avaliação de ferramentas foram definidos para que, após o desenvolvimento de uma aplicação web por meio do processo de desenvolvimento guiado por testes (TDD), o uso da ferramenta de automação de testes fosse então avaliado.Software testing is an important process of software development. In some situations, automating this process may produce some important benefits for software quality control. Therefore, there are several test automation tools that aims to facilitate automation of software testing. This work aims to evaluate Selenium, a test automation tool for web application. To achieve this goal, some evaluation criteria was defined to evaluate the automation testing tool after the development of a web application in which the Test Driven Development (TDD) process was used