Definition of a support infrastructure for replicating and aggregating families of software engineering experiments

Experimental software engineering includes several processes, the most representative being run experiments, run replications and synthesize the results of multiple replications. Of these processes, only the first is relatively well established in software engineering. Problems of information management and communication among researchers are one of the obstacles to progress in the replication and synthesis processes. Software engineering experimentation has expanded considerably over the last few years. This has brought with it the invention of experimental process support proposals. However, few of these proposals provide integral support, including replication and synthesis processes. Most of the proposals focus on experiment execution. This paper proposes an infrastructure providing integral support for the experimental research process, specializing in the replication and synthesis of a family of experiments. The research has been divided into stages or phases, whose transition milestones are marked by the attainment of their goals. Each goal exactly matches an artifact or product. Within each stage, we will adopt cycles of successive approximations (generateand- test cycles), where each approximation includes a diferent viewpoint or input. Each cycle will end with the product approval

The background information on subjects in program comprehension studies

Program comprehension is a very important skill a software engineer need. Many researchers conduct experiments on program comprehension in order to improve tools, documentation, and maintenance guidelines supporting program comprehension. Individual programmers’ productivity might vary significantly even though they have similar background. Thus, the subjects’ background is very important when conducting and analyzing experiments on program comprehension. The survey presented in this short Master thesis identifies subjects background information reported in software experiments on program comprehension. The background information reported in 24 articles was systematically analyzed in order to answer what kind of background information is reported and how the background information was used in the analysis. The articles reports many different background variables, but the overall impression of the background information reported in program comprehension experiments is that it is rather arbitrary and small. The analysis shows that there is a need for standards and guidelines of how to collect and report subjects’ background information. The survey shows also that almost no background information of the subjects is used in the experiments’ analysis. The articles in this survey provide so little information about the subjects’ background that it is difficult to perform replications and meta-analysis. This thesis aims to make researchers more aware of the subjects’ background in their experiments and reports. On the basis of the results of the analysis I have suggested background variables that should be collected in comprehension studies and proposed a background questionnaire. The questionnaire was used in an experiment with 24 subjects from the industry. I report here experiences with the questionnaire

An Environment of Conducting Families of Software Engineering Experiments

The classroom is a valuable resource for conducting software engineering experiments. However, coordinating a family of experiments in classroom environments presents a number of challenges to researchers. This paper describes an environment that simplifies the process of collecting, managing and sanitizing data from classroom experiments, while minimizing disruption to natural subject behavior. We have successfully used this environment to study the impact of parallel programming languages on programmer productivity at multiple universities across the United States

Ferramentas de apoio a experimentos : um mapeamento sistemático

Dissertação (mestrado)—Universidade de Brasília, Instituto de Ciências Exatas, Departamento de Ciência da Computação, 2018.Pesquisadores que desejam obter maior rigor em seus estudos e apresentar maior confiabilidade a seus resultados podem utilizar diversos métodos de estudos empíricos, dentre eles os experimentos controlados. Para isso existem guias que definem fases para um projeto de experimentação, e ferramentas são desenvolvidas para apoiar na realização das atividades. Nesse contexto, este trabalho tem como objetivo analisar estudos que apresentam ferramentas de apoio a experimentação de projetos em engenharia de software, e obter uma relação dessas ferramentas indicando quais técnicas, abstrações de linguagem, automação e garantias são oferecidos por elas em cada fase de um projeto de experimentos. O resultado será sintetizado para identificar os pontos fortes e fracos e quais os problemas mais comumente identificados entre elas. Em relação ao método, será realizado um mapeamento sistemático baseado em questões de pesquisa e um protocolo de pesquisa para extrair informações dos estudos primários. Em termos de resultados esperados, obtemos um guia para ajudar o pesquisador em engenharia de software a escolher uma ferramenta ao realizar experimentos.Researchers who wish to be more rigorous in their studies and present greater reliability to their results can use several methods of empirical studies, among them controlled experiments. There are guides that define phases for a project of experimentation, and tools are developed to support the accomplishment of the activities. In this context, this work aims to analyze studies that present tools to support the experimentation of projects in software engineering, and to obtain a list of these tools indicating which techniques, language abstractions, automation and guarantees are offered by them in each phase of a design of experiments. The result will be synthesized to identify the strengths and weaknesses and which are the most commonly identified problems among them. Regarding method, a systematic mapping will be performed based on research questions and a research protocol to extract information from the primary studies. In terms of expected results, we obtain a guide to help software engineering researcher choose a tool when conducting experiments