A process for managing interaction between experimenters to get useful similar replications

Context: A replication is the repetition of an experiment. Several efforts have been made to adopt replication as a common practice in software engineering. There are different types of replications, depending on their purpose. Similar replications keep the experimental conditions as alike as possible to the original ones. External similar replications, where the replicating experimenters are not the same people as the original experimenters, have been a stumbling block. Several attempts at combining the results of replications have resulted in failure. Software engineering does not appear to be well suited to such replications, because it works with complex experimentally immature contexts. Software engineering settings have a large number of variables, and the role that many of them play is unknown. A successful (or useful) similar replication helps to better understand the phenomenon under study by verifying results and/or identifying contextual variables that could influence (or not) the results, through the combination of experimental results. Objective: To be able to get successful similar replications, there needs to be interaction between original and replicating experimenters. In this paper, we propose an interaction process for achieving successful similar replications. Method: This process consists of: an adaptation meeting, where experimenters tailor the experiment to the new setting; querying, to settle occasional inquiries while the experiment is being run; and a combination meeting, where experimenters meet to discuss the combination of replication outcomes with previous results. To check its effectiveness, the process has been tested on three different replications of the same experiment. Results: The proposed interaction process has helped to identify new contextual variables that could potentially influence (or not) the experimental results in the three replications run. Additionally, the interaction process has helped to uncover certain problems and deviations that occurred during some of the replications that we would have not been aware of otherwise. Conclusions: There are signs that suggest that it is possible to get successful similar replications in software engineering experimentation, when there is appropriate interaction among experimenters.This work has been performed under research Grant TIN2011-23216 of the Spanish Ministry of Science and Innovation.Juristo, N.; Vegas, S.; Solari, M.; Abrahao Gonzales, SM.; Ramos, I. (2013). A process for managing interaction between experimenters to get useful similar replications. Information and Software Technology. 55(2):215-225. https://doi.org/10.1016/j.infsof.2012.07.016S21522555

Understanding replication of experiments in software engineering: a classification

Context: Replication plays an important role in experimental disciplines. There are still many uncertain- ties about how to proceed with replications of SE experiments. Should replicators reuse the baseline experiment materials? How much liaison should there be among the original and replicating experiment- ers, if any? What elements of the experimental configuration can be changed for the experiment to be considered a replication rather than a new experiment? Objective: To improve our understanding of SE experiment replication, in this work we propose a classi- fication which is intend to provide experimenters with guidance about what types of replication they can perform. Method: The research approach followed is structured according to the following activities: (1) a litera- ture review of experiment replication in SE and in other disciplines, (2) identification of typical elements that compose an experimental configuration, (3) identification of different replications purposes and (4) development of a classification of experiment replications for SE. Results: We propose a classification of replications which provides experimenters in SE with guidance about what changes can they make in a replication and, based on these, what verification purposes such a replication can serve. The proposed classification helped to accommodate opposing views within a broader framework, it is capable of accounting for less similar replications to more similar ones regarding the baseline experiment. Conclusion: The aim of replication is to verify results, but different types of replication serve special ver- ification purposes and afford different degrees of change. Each replication type helps to discover partic- ular experimental conditions that might influence the results. The proposed classification can be used to identify changes in a replication and, based on these, understand the level of verification

Investigation of individual factors impacting the effectiveness of requirements inspections: a replicated experiment

Abstract This paper presents a replication of an empirical study regarding the impact of individual factors on the effectiveness of requirements inspections. Experimental replications are important for verifying results and investigating the generality of empirical studies. We utilized the lab package and procedures from the original study, with some changes and additions, to conduct the replication with 69 professional developers in three different companies in Turkey. In general the results of the replication were consistent with those of the original study. The main result from the original study, which is supported in the replication, was that inspectors whose degree is in a field related to software engineering are less effective during a requirements inspection than inspectors whose degrees are in other fields. In addition, we found that Company, Experience, and English Proficiency impacted inspection effectiveness

Identification and Evaluation of Predictors for Learning Success and of Models for Teaching Computer Programming in Contemporary Contexts

Introductory undergraduate computer programming courses are renowned for higher than average failure and withdrawal rates when compared to other subject areas. The closer partnership between higher education and the rapidly expanding digital technology industry, as demonstrated by the establishment of new Degree Apprenticeships in computer science and digital technologies, requires efficient and effective means for teaching programming skills. This research, therefore, aimed to identify reliable predictors of success in learning programming or vulnerability to failure. The research also aimed to evaluate teaching methods and remedial interventions towards recommending a teaching model that supported and engaged learners in contemporary contexts that were relevant to the workplace. Investigation of qualifications designed to prepare students for undergraduate computer science courses revealed that A-level entrants achieved significantly higher programming grades than BTEC students. However, there was little difference between the grades of those with and those without previous qualifications in computing or ICT subjects. Analysis of engagement metrics revealed a strong correlation between extent of co-operation and programming grade, in contrast to a weak correlation between programming grade and code understanding. Further analysis of video recordings, interviews and observational records distinguished between the type of communication that helped peers comprehend tasks and concepts, and other forms of communication that were only concerned with completing tasks. Following the introduction of periodic assessment, essentially converting a single final assessment to three staged summative assessment points, it was found that failing students often pass only one of the three assignment parts. Furthermore, only 10% of those who failed overall had attempted all three assignments. Reasons for failure were attributed to ‘surface’ motivations (such as regulating efforts to achieve a minimum pass of 40%), ineffective working habits or stressful personal circumstances rather than any fundamental difficulty encountered with subject material. A key contribution to pedagogical practice made by this research is to propose an ‘incremental’ teaching model. This model is informed by educational theory and empirical evidence and comprises short cycles of three activities: presenting new topic information, tasking students with a relevant exercise and then demonstrating and discussing the exercise solution. The effectiveness of this model is evidenced by increased engagement, increased quiz scores at the end of each teaching session and increased retention of code knowledge at the end of the course

Tipología de Replicaciones para la Síntesis de Experimentos en Ingeniería del Software

La Ingenieria del Software Experimental (ISE) traslada a la Ingenieria del Software (IS) el paradigma experimental que se ha aplicado con exito en diversas disciplinas cientificas. El objetivo de la ISE es hacer de la construccion del software una actividad predecible gracias al conocimiento de las relaciones entre los procesos de produccion del software y los productos que se obtienen. Para avanzar en el paradigma experimental en IS no es suficiente aplicar las tecnicas de diseno experimental y el analisis estadistico de datos, sino que es necesario construir una metodologia (bien desde cero o adaptada de otras disciplinas) basada en los principios generales del experimentalismo. La motivacion principal de esta investigacion es trabajar en la adaptacion de un aspecto particular del paradigma experimental a la experimentacion en IS: la replicación. En ISE se han realizado varias replicaciones de experimentos, sin embargo, aun existe discusion sobre el modo mas adecuado de llevarlas a cabo. Algunas preguntas que surgen de esta discusion son: .se deben reutilizar los materiales del experimento base?, .la replicacion debe realizarse de forma independiente, o puede existir algun tipo de comunicacion entre experimentadores y replicadores?, .que elementos de la estructura del experimento a replicar pueden variarse y aun considerarse una replicacion? En esta investigacion se estudia el concepto de replicacion desde una perspectiva teorico-practica para su incorporacion a la ISE. En concreto, se persiguen los siguientes objetivos: 1) estudio del concepto de replicacion en distintas disciplinas cientificas para tener mayor comprension de su importacion a la ISE, 2) desarrollo de una tipologia de replicaciones que ayude a comprender tanto los diferentes tipos de replicacion que pueden llevarse a cabo en ISE, asi como el papel que cada uno de estos tipos desempena en la verificacion de resultados experimentales y 3) desarrollo de un marco conceptual con ideas clave para comparar conjuntos de replicaciones y obtener conocimiento de ellas que sea de utilidad tanto para el profesional como para el investigador. Para la evaluacion de las propuestas de esta tesis se usa un conjunto de 20 replicaciones de diversos autores donde entre otros aspectos se evalua la efectividad de tres tecnicas de evaluacion de software