Design Patterns in Software Maintenance: An Experiment Replication at UPM - Experiences with the RESER'11 Joint Replication Project

Replication of software engineering experiments is crucial for dealing with validity threats to experiments in this area. Even though the empirical software engineering community is aware of the importance of replication, the replication rate is still very low. The RESER'11 Joint Replication Project aims to tackle this problem by simultaneously running a series of several replications of the same experiment. In this article, we report the results of the replication run at the Universidad Politécnica de Madrid. Our results are inconsistent with the original experiment. However, we have identified possible causes for them. We also discuss our experiences (in terms of pros and cons) during the replication

Understanding replication of experiments in software engineering: a classification

Context: Replication plays an important role in experimental disciplines. There are still many uncertain- ties about how to proceed with replications of SE experiments. Should replicators reuse the baseline experiment materials? How much liaison should there be among the original and replicating experiment- ers, if any? What elements of the experimental configuration can be changed for the experiment to be considered a replication rather than a new experiment? Objective: To improve our understanding of SE experiment replication, in this work we propose a classi- fication which is intend to provide experimenters with guidance about what types of replication they can perform. Method: The research approach followed is structured according to the following activities: (1) a litera- ture review of experiment replication in SE and in other disciplines, (2) identification of typical elements that compose an experimental configuration, (3) identification of different replications purposes and (4) development of a classification of experiment replications for SE. Results: We propose a classification of replications which provides experimenters in SE with guidance about what changes can they make in a replication and, based on these, what verification purposes such a replication can serve. The proposed classification helped to accommodate opposing views within a broader framework, it is capable of accounting for less similar replications to more similar ones regarding the baseline experiment. Conclusion: The aim of replication is to verify results, but different types of replication serve special ver- ification purposes and afford different degrees of change. Each replication type helps to discover partic- ular experimental conditions that might influence the results. The proposed classification can be used to identify changes in a replication and, based on these, understand the level of verification

Variation Factors in the Design and Analysis of Replicated Controlled Experiments - Three (Dis)similar Studies on Inspections versus Unit Testing

Background. In formal experiments on software engineering, the number of factors that may impact an outcome is very high. Some factors are controlled and change by design, while others are are either unforeseen or due to chance. Aims. This paper aims to explore how context factors change in a series of for- mal experiments and to identify implications for experimentation and replication practices to enable learning from experimentation. Method. We analyze three experiments on code inspections and structural unit testing. The first two experiments use the same experimental design and instrumentation (replication), while the third, conducted by different researchers, replaces the programs and adapts defect detection methods accordingly (reproduction). Experimental procedures and location also differ between the experiments. Results. Contrary to expectations, there are significant differences between the original experiment and the replication, as well as compared to the reproduction. Some of the differences are due to factors other than the ones designed to vary between experiments, indicating the sensitivity to context factors in software engineering experimentation. Conclusions. In aggregate, the analysis indicates that reducing the complexity of software engineering experiments should be considered by researchers who want to obtain reliable and repeatable empirical measures

Definition of a support infrastructure for replicating and aggregating families of software engineering experiments

Experimental software engineering includes several processes, the most representative being run experiments, run replications and synthesize the results of multiple replications. Of these processes, only the first is relatively well established in software engineering. Problems of information management and communication among researchers are one of the obstacles to progress in the replication and synthesis processes. Software engineering experimentation has expanded considerably over the last few years. This has brought with it the invention of experimental process support proposals. However, few of these proposals provide integral support, including replication and synthesis processes. Most of the proposals focus on experiment execution. This paper proposes an infrastructure providing integral support for the experimental research process, specializing in the replication and synthesis of a family of experiments. The research has been divided into stages or phases, whose transition milestones are marked by the attainment of their goals. Each goal exactly matches an artifact or product. Within each stage, we will adopt cycles of successive approximations (generateand- test cycles), where each approximation includes a diferent viewpoint or input. Each cycle will end with the product approval

Configuration management and product lines to enhance the replication process in software engineering

This research is concerned with the experimental software engineering area, specifically experiment replication. Replication has traditionally been viewed as a complex task in software engineering. This is possibly due to the present immaturity of the experimental paradigm applied to software development. Researchers usually use replication packages to replicate an experiment. However, replication packages are not the solution to all the information management problems that crop up when successive replications of an experiment accumulate. This research borrows ideas from the software configuration management and software product line paradigms to support the replication process. We believe that configuration management can help to manage and administer information from one replication to another: hypotheses, designs, data analysis, etc. The software product line paradigm can help to organize and manage any changes introduced into the experiment by each replication. We expect the union of the two paradigms in replication to improve the planning, design and execution of further replications and their alignment with existing replications. Additionally, this research work will contribute a web support environment for archiving information related to different experiment replications. Additionally, it will provide flexible enough information management support for running replications with different numbers and types of changes. Finally, it will afford massive storage of data from different replications. Experimenters working collaboratively on the same experiment must all have access to the different experiments

Replication studies considered harmful

Context: There is growing interest in establishing software engineering as an evidence-based discipline. To that end, replication is often used to gain confidence in empirical findings, as opposed to reproduction where the goal is showing the correctness, or validity of the published results. Objective: To consider what is required for a replication study to confirm the original experiment and apply this understanding in software engineering. Method: Simulation is used to demonstrate why the prediction interval for confirmation can be surprisingly wide. This analysis is applied to three recent replications. Results: It is shown that because the prediction intervals are wide, almost all replications are confirmatory, so in that sense there is no ‘replication crisis’, however, the contributions to knowledge are negligible. Conclusion: Replicating empirical software engineering experiments, particularly if they are under-powered or under-reported, is a waste of scientific resources. By contrast, meta-analysis is strongly advocated so that all relevant experiments are combined to estimate the population effect

Using configuration management and product line software paradigms to support the experimentation process in software engineering.

There is no empirical evidence whatsoever to support most of the beliefs on which software construction is based. We do not yet know the adequacy, limits, qualities, costs and risks of the technologies used to develop software. Experimentation helps to check and convert beliefs and opinions into facts. This research is concerned with the replication area. Replication is a key component for gathering empirical evidence on software development that can be used in industry to build better software more efficiently. Replication has not been an easy thing to do in software engineering (SE) because the experimental paradigm applied to software development is still immature. Nowadays, a replication is executed mostly using a traditional replication package. But traditional replication packages do not appear, for some reason, to have been as effective as expected for transferring information among researchers in SE experimentation. The trouble spot appears to be the replication setup, caused by version management problems with materials, instruments, documents, etc. This has proved to be an obstacle to obtaining enough details about the experiment to be able to reproduce it as exactly as possible. We address the problem of information exchange among experimenters by developing a schema to characterize replications. We will adapt configuration management and product line ideas to support the experimentation process. This will enable researchers to make systematic decisions based on explicit knowledge rather than assumptions about replications. This research will output a replication support web environment. This environment will not only archive but also manage experimental materials flexibly enough to allow both similar and differentiated replications with massive experimental data storage. The platform should be accessible to several research groups working together on the same families of experiments

Propuesta de paquete de laboratorio para experimentos de ingeniería de software

En la ingeniería de software empírica se aplican diversos métodos para promover la generación y validación de conocimiento. La realización de experimentos es un paso necesario del método científico. Para que el conocimiento empírico pueda madurar se requiere que los experimentos sean replicados. El propósito de la replicación es aumentar la confianza sobre los resultados y profundizar en los detalles del fenómeno estudiado. Este trabajo pretende facilitar la replicación de experimentos de ingeniería de software. Para satisfacer este objetivo, se realiza una propuesta de paquete de laboratorio para experimentos de ingeniería de software. El paquete de laboratorio es el contenedor del conocimiento relativo a un experimento. Esto incluye las instrucciones y materiales del experimento con el objetivo de replicarlo. La propuesta está compuesta por una estructura y guías para la elaboración del contenido de un paquete de laboratorio en el contexto de la investigación experimental en ingeniería de software. Para distintos experimentos de ingeniería de software se puede instanciar un paquete de laboratorio de acuerdo a la propuesta que facilite la replicación. Para obtener la propuesta de paquete de laboratorio se sigue un proceso empírico e inductivo. En primer lugar, se elabora un modelo de evaluación de replicaciones. El modelo define un conjunto de aspectos de evaluación que permiten comparar los resultados de replicaciones que usan distintos tipos de instrumentos e identificar los incidentes ocurridos. Mediante el análisis de los incidentes, el estudio de la usabilidad de la documentación y las propuestas de los usuarios, se desarrolla una propuesta genérica de paquete de laboratorio. La validación de la propuesta se realiza mediante la instanciación de paquetes de laboratorio para experimentos concretos y usándolos en replicaciones. La instanciación de paquetes de laboratorio para varios de experimentos muestra la viabilidad y flexibilidad de la propuesta. Las evaluaciones de replicaciones que usan paquetes de laboratorio estructurados de acuerdo a la propuesta muestran mejores resultados en los aspectos: completitud, usabilidad, eficacia, eficiencia y satisfacción del investigador participante. Abstract Several methods are used in empirical software engineering to generate and validate knowledge. Performing experiments is a necessary step of the scientific method. Replication of experiments is required to mature the empirical knowledge. The objective of replication is to gain trust on the results and dig into the details of the studied phenomenon. The goal of this work is to facilitate replication of software engineering experiments. In order to satisfy this objective, a laboratory package proposal for software engineering experiments is made. The laboratory package is the container of the knowledge about an experiment. This includes experiment instructions and materials with the objective of replication. The proposal is composed by a structure and content guidelines in the context of experimental software engineering research. For different experiments a laboratory package can be instantiated according to the proposal to facilitate replication. An empirical and inductive process is used to obtain the laboratory package proposal. First, a replication evaluation model is created. The model defines a set of evaluation aspects to allow the comparison of replications which use different types of instruments and to identify incidents. A generic laboratory package proposal is developed using the incident analysis, the study of document usability and the users’ requests. The proposal is validated with the instantiation of laboratory packages for concrete experiments and using them in replications. The instantiation of laboratory packages for different experiments shows the viability and flexibility of the proposal. The evaluation of replications which use laboratory packages structured according to the proposal shows better results in the aspects: completeness, usability, efficacy, efficiency and satisfaction of the participating researcher