Test adequacy assessment using test-defect coverage analytic model

Software testing is an essential activity in software development process that has been widely used as a means of achieving software reliability and quality. The emergence of incremental development in its various forms required a different approach to determining the readiness of the software for release. This approach needs to determine how reliable the software is likely to be based on planned tests, not defect growth and decline as typically shown in reliability growth models. A combination of information from a number of sources into an easily understood dashboard is expected to provide both qualitative and quantitative analyses of test and defect coverage properties. Hence, Test-Defect Coverage Analytic Model (TDCAM) is proposed which combines test and defect coverage information presented in a dashboard to help deciding whether there are enough tests planned. A case study has been conducted to demonstrate the usage of the proposed model. The visual representations and results gained from the case study show the benefits of TDCAM in assisting practitioners making informed test adequacy-related decisions

Goal-Oriented Mutation Testing with Focal Methods

Mutation testing is the state-of-the-art technique for assessing the fault-detection capacity of a test suite. Unfortunately, mutation testing consumes enormous computing resources because it runs the whole test suite for each and every injected mutant. In this paper we explore fine-grained traceability links at method level (named focal methods), to reduce the execution time of mutation testing and to verify the quality of the test cases for each individual method, instead of the usually verified overall test suite quality. Validation of our approach on the open source Apache Ant project shows a speed-up of 573.5x for the mutants located in focal methods with a quality score of 80%.Comment: A-TEST 201

Trade-off between automated and manual testing: A production possibility curve cost model

Testing is always important for Software Quality Assurance (SQA) activities and key cost multiplier in software development. The decision to automate or not to automate a test case is critical. In this paper we discuss the possibility of test automation and in relation to the trade-off between manual and automated test cases. We purpose a Production cost frontier based technique to distinguish the point of automation and manual test within the cost constraints. Our objective is to identify the facts that up to what extant a testing process can be automated. In this paper a cost model is proposed for deciding the proportion of automated and manual testing. The objective is to find best possible combination of these two and production possibility in one type by eliminating the other type of testin

Predicting Test Suite Effectiveness for Java Programs

The coverage of a test suite is often used as a proxy for its effectiveness. However, previous studies that investigated the influence of code coverage on test suite effectiveness have failed to reach a consensus about the nature and strength of the relationship between these test suite characteristics. Moreover, many of the studies were done with small or synthetic programs, making it unclear that their results generalize to larger programs. In addition, some of the studies did not account for the confounding influence of test suite size. We have extended these studies by evaluating the relationship between test suite size, block coverage, and effectiveness for large Java programs. Our test subjects were four Java programs from different application domains: Apache POI, HSQLDB, JFreeChart, and Joda Time. All four are actively developed open source programs; they range from 80,000 to 284,000 source lines of code. For each test subject, we generated between 5,000 and 7,000 test suites by randomly selecting test methods from the program's entire test suite. The suites ranged in size from 3 to 3,000 methods. We used the coverage tool Emma to measure the block coverage of each suite and the mutation testing tool Javalanche to evaluate the effectiveness of each suite. We found that there is a low correlation between block coverage and effectiveness when the number of tests in the suite is controlled for. This suggests that block coverage, while useful for identifying under-tested parts of a program, should not be used as a quality target because it is not a good indicator of test suite effectiveness

Um algoritmo para o cálculo de cobertura de estados

Dissertação (mestrado) - Universidade Federal de Santa Catarina, Centro Tecnológico, Programa de Pós-Graduação em Ciência da Computação, Florianópolis, 2015.Cobertura de estados é um critério de adequação de testes de software que mede a quantidade de modificações de estados feitas durante a execução dos testes que foram verificadas através de asserções. O presente trabalho propõe um algoritmo para o cálculo de cobertura de estados baseado em construções comuns a linguagens orientadas a objetos, como atribuições, retorno de métodos e chamadas de funções. O algoritmo identifica modificações de estados cobertas por asserções através de um novo cálculo de influências de atributos em métodos de uma classe baseado na extração de dependências entre identificadores existentes no código. São apresentados ainda uma extensão à definição de estado modificado levando em consideração a distinção entre atributos simples e compostos (estruturas de dados) e uma implementação do algoritmo através da instrumentação de bytecode Java. Experimentos feitos em sete projetos de código aberto mostraram a utilidade do algoritmo na identificação de atributos não verificados por asserções mas executados pelo teste. Ao inserir erros propositais nestes atributos os testes não falharam. Em uma situação real, estes erros estariam imperceptíveis pelos desenvolvedores. Os experimentos ainda mostraram que a execução do algoritmo adicionou um overhead de 20% a 33% no tempo de execução dos testes unitários, um valor abaixo dos trabalhos existentes.Abstract : State coverage is a test adequacy criterion that measures the quantity of state modifications made by a test execution that were verified by assertions. This work proposes an algorithm for state coverage based on common constructions of object-oriented languages, such as assignments, method returns and function calls. The algorithm identifies influences of an instance attribute on assertions applying a new method to compute influences of attributes on methods. This work also presents an extension to the modified state definition by applying a distinction between simple and compound types (data-structures) and an implementation of the algorithm based on instrumentation of Java bytecode. Experiments made on seven open source projects showed the validity of the algorithm in identifying attributes not verified by assertions that were executed by tests. Bugs inserted in these attributes were not captured by the tests. The results also showed that the algorithm adds an overhead of 20% and 33% at the test execution. These values are below of existent works

Eine kunden- und lebenszyklusorientierte Produktfamilienabsicherung für die Automobilindustrie

In der vorliegenden Arbeit wird eine automotive-geeignete Absicherungsstrategie entwickelt, die erstmals den kompletten Variantenumfang eines massengefertigten eingebetteten Systems inklusive der Versionierung dessen Einzelkomponenten im Lebenszyklus systematisch betrachten und empirisch bewerten kann. Die Realisierung der Strategie bietet statistisch insbesondere Vorteile durch die Steigerung der vom Kunden wahrgenommenen Qualität aufgrund einer optimierten Absicherung von Systemvarianten