Um algoritmo para o cálculo de cobertura de estados

Dissertação (mestrado) - Universidade Federal de Santa Catarina, Centro Tecnológico, Programa de Pós-Graduação em Ciência da Computação, Florianópolis, 2015.Cobertura de estados é um critério de adequação de testes de software que mede a quantidade de modificações de estados feitas durante a execução dos testes que foram verificadas através de asserções. O presente trabalho propõe um algoritmo para o cálculo de cobertura de estados baseado em construções comuns a linguagens orientadas a objetos, como atribuições, retorno de métodos e chamadas de funções. O algoritmo identifica modificações de estados cobertas por asserções através de um novo cálculo de influências de atributos em métodos de uma classe baseado na extração de dependências entre identificadores existentes no código. São apresentados ainda uma extensão à definição de estado modificado levando em consideração a distinção entre atributos simples e compostos (estruturas de dados) e uma implementação do algoritmo através da instrumentação de bytecode Java. Experimentos feitos em sete projetos de código aberto mostraram a utilidade do algoritmo na identificação de atributos não verificados por asserções mas executados pelo teste. Ao inserir erros propositais nestes atributos os testes não falharam. Em uma situação real, estes erros estariam imperceptíveis pelos desenvolvedores. Os experimentos ainda mostraram que a execução do algoritmo adicionou um overhead de 20% a 33% no tempo de execução dos testes unitários, um valor abaixo dos trabalhos existentes.Abstract : State coverage is a test adequacy criterion that measures the quantity of state modifications made by a test execution that were verified by assertions. This work proposes an algorithm for state coverage based on common constructions of object-oriented languages, such as assignments, method returns and function calls. The algorithm identifies influences of an instance attribute on assertions applying a new method to compute influences of attributes on methods. This work also presents an extension to the modified state definition by applying a distinction between simple and compound types (data-structures) and an implementation of the algorithm based on instrumentation of Java bytecode. Experiments made on seven open source projects showed the validity of the algorithm in identifying attributes not verified by assertions that were executed by tests. Bugs inserted in these attributes were not captured by the tests. The results also showed that the algorithm adds an overhead of 20% and 33% at the test execution. These values are below of existent works

Redefining and Evaluating Coverage Criteria Based on the Testing Scope

Test coverage information can help testers in deciding when to stop testing and in augmenting their test suites when the measured coverage is not deemed sufficient. Since the notion of a test criterion was introduced in the 70’s, research on coverage testing has been very active with much effort dedicated to the definition of new, more cost-effective, coverage criteria or to the adaptation of existing ones to a different domain. All these studies share the premise that after defining the entity to be covered (e.g., branches), one cannot consider a program to be adequately tested if some of its entities have never been exercised by any input data. However, it is not the case that all entities are of interest in every context. This is particularly true for several paradigms that emerged in the last decade (e.g., component-based development, service-oriented architecture). In such cases, traditional coverage metrics might not always provide meaningful information. In this thesis we address such situation and we redefine coverage criteria so to focus on the program parts that are relevant to the testing scope. We instantiate this general notion of scope-based coverage by introducing three coverage criteria and we demonstrate how they could be applied to different testing contexts. When applied to the context of software reuse, our approach proved to be useful for supporting test case prioritization, selection and minimization. Our studies showed that for prioritization we can improve the average rate of faults detected. For test case selection and minimization, we can considerably reduce the test suite size with small to no extra impact on fault detection effectiveness. When the source code is not available, such as in the service-oriented architecture paradigm, we propose an approach that customizes coverage, measured on invocations at service interface, based on data from similar users. We applied this approach to a real world application and, in our study, we were able to predict the entities that would be of interest for a given user with high precision. Finally, we introduce the first of its kind coverage criterion for operational profile based testing that exploits program spectra obtained from usage traces. Our study showed that it is better correlated than traditional coverage with the probability that the next test input will fail, which implies that our approach can provide a better stopping rule. Promising results were also observed for test case selection. Our redefinition of coverage criteria approaches the topic of coverage testing from a completely different angle. Such a novel perspective paves the way for new avenues of research towards improving the cost-effectiveness of testing, yet all to be explored