O Teste de Mutação apoiado pelo Algoritmo Genético Coevolucionário com Classificação Genética Controlada

Este artigo situa-se no campo dos algoritmos genéticos coevolucionários que objetivam a seleção de bons subconjuntos de casos de teste e mutantes, no contexto do Teste de Mutação. Desse campo de estudo, selecionou-se e avaliou-se duas abordagens existentes. Tal avaliação, subsidiou o desenvolvimento de um novo Algoritmo Coevolucionário com Classificação Genética Controlada (AGC − CGC). Para analisar a abordagem, 164 experimentos foram realizados comparando os resultados do algoritmo proposto com outros três métodos aplicados em quatro benchmarks reais. Os resultados revelam uma melhora significativa do AGC − CGC sobre as outras abordagens quando se considera o aumento do escore de mutação sem aumentar acentuadamente o tempo de execução

Operator-based and random mutant selection: Better together

Abstract—Mutation testing is a powerful methodology for evaluating the quality of a test suite. However, the methodology is also very costly, as the test suite may have to be executed for each mutant. Selective mutation testing is a well-studied technique to reduce this cost by selecting a subset of all mutants, which would otherwise have to be considered in their entirety. Two common approaches are operator-based mutant selection, which only generates mutants using a subset of mutation operators, and random mutant selection, which selects a subset of mutants generated using all mutation operators. While each of the two approaches provides some reduction in the number of mutants to execute, applying either of the two to medium-sized, real-world programs can still generate a huge number of mutants, which makes their execution too expensive. This paper presents eight random sampling strategies defined on top of operator-based mutant selection, and empirically validates that operator-based selection and random selection can be applied in tandem to further reduce the cost of mutation testing. The experimental results show that even sampling only 5 % of mutants generated by operator-based selection can still provide precise mutation testing results, while reducing the average mutation testing time to 6.54 % (i.e., on average less than 5 minutes for this study). I

Higher Order Mutation Testing

Mutation testing is a fault-based software testing technique that has been studied widely for over three decades. To date, work in this field has focused largely on first order mutants because it is believed that higher order mutation testing is too computationally expensive to be practical. This thesis argues that some higher order mutants are potentially better able to simulate real world faults and to reveal insights into programming bugs than the restricted class of first order mutants. This thesis proposes a higher order mutation testing paradigm which combines valuable higher order mutants and non-trivial first order mutants together for mutation testing. To overcome the exponential increase in the number of higher order mutants a search process that seeks fit mutants (both first and higher order) from the space of all possible mutants is proposed. A fault-based higher order mutant classification scheme is introduced. Based on different types of fault interactions, this approach classifies higher order mutants into four categories: expected, worsening, fault masking and fault shifting. A search-based approach is then proposed for locating subsuming and strongly subsuming higher order mutants. These mutants are a subset of fault mask and fault shift classes of higher order mutants that are more difficult to kill than their constituent first order mutants. Finally, a hybrid test data generation approach is introduced, which combines the dynamic symbolic execution and search based software testing approaches to generate strongly adequate test data to kill first and higher order mutants

Parallel, Cross-Platform Unit Testing for Real-Time Embedded Systems

Embedded systems are used in a wide variety of applications (e.g., automotive, agricultural, home security, industrial, medical, military, and aerospace) due to their small size, low-energy consumption, and the ability to control real-time peripheral devices precisely. These systems, however, are different from each other in many aspects: processors, memory size, develop applications/OS, hardware interfaces, and software loading methods. Unit testing is a fundamental part of software development and the lowest level of software testing, as it tests individual or groups of functions, methods, and classes, to increase confidence that the developed software satisfies both software specifications and user requirements. Although hundreds of unit testing frameworks exist, none of them address the diverse properties of real-time embedded platforms. This inspires us to introduce XEUnit, a cross-platform unit testing framework for real-time embedded systems. XEUnit provides scalability to the framework by supporting parallel execution on multiple embedded platforms simultaneously. To address the time constraints in real-time embedded systems, we evaluate the impact of runtime overhead from traditional instrumentation through a case study of time-sensitive algorithms. Then, we introduce iterative instrumentation, which is a code coverage technique without runtime overhead, along with a case study demonstrating the effectiveness of this technique. Although iterative instrumentation can measure code coverage effectively in time-sensitive applications, the total execution cost of this approach is much higher than traditional instrumentation due to the execution of multiple variants of the system under test. This leads to scalability and performance issues especially in large applications. To solve these issues, there are two approaches we use: reducing the number of variants and executing them simultaneously. To reduce the number of variants, we present cluster iterative instrumentation, a graph clustering technique that can reduce the number of nodes in a control flow graph resulting in lower execution time. We also provide a case study of node coverage of control software to show the efficiency of cluster iterative instrumentation compared to iterative instrumentation. In addition to reducing the number of variants, the other method is to execute multiple variants at the same time. Because all executions are independent from each other, we can use parallel execution on multiple embedded platforms. Thus, we design and implement a parallel unit testing framework for real-time embedded system along with a case study comparing the execution times from different numbers of embedded platforms (executing nodes). Our final contribution is a cross-platform unit testing framework using the concepts of runtime adapters and a runtime protocol that enables testers to run code across different embedded platforms. We also demonstrate the effectiveness of this framework by testing black-box test cases on seven different embedded platforms. Overall, our results indicate that cluster iterative instrumentation with parallel unit testing can address the scalability and performance issues, and the case studies demonstrate that XEUnit can effectively test the same code on a variety of embedded platforms