Why Research on Test-Driven Development is Inconclusive?

[Background] Recent investigations into the effects of Test-Driven Development (TDD) have been contradictory and inconclusive. This hinders development teams to use research results as the basis for deciding whether and how to apply TDD. [Aim] To support researchers when designing a new study and to increase the applicability of TDD research in the decision-making process in the industrial context, we aim at identifying the reasons behind the inconclusive research results in TDD. [Method] We studied the state of the art in TDD research published in top venues in the past decade, and analyzed the way these studies were set up. [Results] We identified five categories of factors that directly impact the outcome of studies on TDD. [Conclusions] This work can help researchers to conduct more reliable studies, and inform practitioners of risks they need to consider when consulting research on TDD.Comment: ESEM '20: ACM / IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), October 8--9, 2020, Bari, Ital

Lessons learned from a partial replication of an experiment in the context of a software engineering course

Replications are an integral component of experimentation through which the validity and reliability of the observed outcome in a previous experiment can be probed. In a strict replication, the experiment is executed in the same conditions as the original by following the same protocol and thus the evidence is strengthened statistically by means of increased sample size. Another objective for running replications is generalizing the experimental results beyond the limitations of one study and its context. For this purpose, certain elements of the original experiment, such as experimenters, experimental objects, and construct operationalization are altered and their impact is investigated. This paper presents lessons learned from a replication that was conducted as a part of an undergraduate university course in Serbia. The focus of the experiment was investigating the effectiveness of writing tests during the development process. The original experiment investigated the effectiveness of test-first programming and was conducted in Italy (Politecnico di Torino) with thirdyear computer science students during an intensive Java course. Lessons learned from this partial replication are that the given task descriptions and structure has an impact on the experiment outcome and that variations in metrics collection can occur when multiple researchers analyse the data, which requires metrics consolidation

Revisión sistemática de la calidad del software en prácticas ágiles

El desarrollo de software ágil representa un alejamiento importante de los enfoques tradicionales basados en una detallada planificación. Una pregunta fundamental para la mayoría de las organizaciones es saber cuáles prácticas ágiles ayudan, en menor o mayor grado, a mejorar la calidad del producto software. En este trabajo se muestra el resultado de una revisión sistemática de la literatura que intenta presentar los resultados de estudios empíricos relativos a la evaluación de la calidad en prácticas ágiles publicadas hasta el año 2014. Los estudios encontrados se analizaron siguiendo los requerimientos de calidad definidos en estándares como el ISO/IEC 25010, se catalogaron en cinco grupos: programación en pares, desarrollos guiados por pruebas, extreme programming, scrum y otras prácticas ágiles; finalmente los hallazgos se compararon e interpretaron. Los resultados de la mayoría de los estudios sugieren que las prácticas ágiles pueden ayudar a mejorar la calidad del producto si son aplicadas correctamente. Los hallazgos significativos de este estudio pueden ser usados como directrices para los interesados en sus propios escenarios.Tesi

Una propuesta para reportar replicaciones de experimentos en Ingeniería del Software

En esta memoria, se presenta el trabajo de investigación llevado a cabo para aumentar el beneficio que se obtiene mediante la replicación de experimentos, desarrollando una propuesta que facilite la especificación sistemática de los cambios de las replicaciones y su documentación de forma homogénea. La importancia de especificar bien los cambios en las replicaciones viene dada por un doble motivo; por un lado, por el propio autor de la replicación; para tener una guía de cómo reportarla. Parte de los cambios que se propongan permitirán mitigar algunas limitaciones identificadas en el experimento base, o bien son consecuencia de adaptar el experimento al nuevo entorno y es necesario analizar su influencia en la validez del experimento. Por otro lado, a un experimentador externo que realiza una replicación, le será de gran ayuda conocer cómo ha evolucionado un experimento original dentro de una familia, porque se han realizado cambios y de qué tipo son. De este modo, se facilita el diseño de experimentos, sin repetir fallos ya identificados y adaptar el ˜ experimento al nuevo entorno con mas éxito. Partiendo de la información identificada en la descripción del cambio, se ha propuesto un metamodelo visualizado mediante una plantilla que presenta los datos de forma fija y sirve de guía para evitar que falte información relevante. La plantilla se completa con patrones lingüísticos que facilitan la redacción de los cambios. Basada en la plantilla propuesta, se presenta la herramienta CÆSAR que permite a los experimentadores la definición de los cambios, que pasan a formar parte del repositorio de información, proporcionando una visión global de la familia de experimentos. Para validar la plantilla, se ha instanciado mediante un estudio de caso múltiple que abarca las áreas de Ingeniería del Software, Ciencias y Experimentos automáticos. La utilización comparativa de la plantilla nos ha permitido establecer diferencias entre las áreas debidas, a la distinta terminología y conceptos utilizados al llevar a cabo un experimento.In this dissertation, we report on our contribution to increase the benefit obtained through the replication of experiments, developing a proposal that facilitates the systematic specification of replication changes and their documentation in a homogeneous way. The importance of specifying replication changes is twofold; on the one hand, so that the author has a guide on how to report the replication. Some of the proposed changes will mitigate some limitations identified in the base experiment or are a consequence of adapting the experiment to the new environment and it is necessary to analyze their influence on the validity of the experiment. On the other hand, an external experimenter who carries out a replication needs to know the evolution of the original experiment within a family, as well as the type and reason for the changes. This facilitates the design of experiments without repeating failures already identified and adapting the experiment to the new environment successfully. Based on the information identified in the change description, a metamodel has been proposed which is displayed using a template that shows the data in a fixed form and serves as a guide to avoid the loss of relevant information. The template is completed with linguistic patterns that facilitate the writing of changes. In order to facilitate the definition of changes for experimenters, the CÆSAR tool based on the proposed template is presented. The changes are part of the information repository providing an overview of the family of experiments. To validate the template, it has been instantiated by means of a multi-case study that covers the areas of Software Engineering, Science and Automatic Experiments. The comparative use of the template has allowed us to establish differences between the areas due to the different terminology and concepts used when carrying out an experiment