Mutation testing on an object-oriented framework: An experience report

This is the preprint version of the article - Copyright @ 2011 ElsevierContext The increasing presence of Object-Oriented (OO) programs in industrial systems is progressively drawing the attention of mutation researchers toward this paradigm. However, while the number of research contributions in this topic is plentiful, the number of empirical results is still marginal and mostly provided by researchers rather than practitioners. Objective This article reports our experience using mutation testing to measure the effectiveness of an automated test data generator from a user perspective. Method In our study, we applied both traditional and class-level mutation operators to FaMa, an open source Java framework currently being used for research and commercial purposes. We also compared and contrasted our results with the data obtained from some motivating faults found in the literature and two real tools for the analysis of feature models, FaMa and SPLOT. Results Our results are summarized in a number of lessons learned supporting previous isolated results as well as new findings that hopefully will motivate further research in the field. Conclusion We conclude that mutation testing is an effective and affordable technique to measure the effectiveness of test mechanisms in OO systems. We found, however, several practical limitations in current tool support that should be addressed to facilitate the work of testers. We also missed specific techniques and tools to apply mutation testing at the system level.This work has been partially supported by the European Commission (FEDER) and Spanish Government under CICYT Project SETI (TIN2009-07366) and the Andalusian Government Projects ISABEL (TIC-2533) and THEOS (TIC-5906)

Object-Oriented Mutation Applied in Common Intermediate Language Programs Originated from C#

Abstract-Application of object-oriented mutation operators in C# programs using a parser-based tool can be precise but requires compilation of mutants. Mutations can be introduced faster directly to the Common Intermediate Language of .NET. It can be simple for traditional mutation operators but more complicated for the object-oriented ones. We propose the reconstruction of complex object-oriented faults on the intermediate language level. The approach was tested in the ILMutator tool implementing few object-oriented mutation operators in the intermediate code derived from compiled C# programs. Exemplary mutation and performance results are given and compared to results of the parser-based mutation tool CREAM

Uma abordagem baseada em hiper-heurística e otimização multi-objetivo para o teste de mutação de ordem superior

Orientador : Prof. Dr. Silvia Regina VergilioDissertação (mestrado) - Universidade Federal do Paraná, Setor de Ciências Exatas, Programa de Pós-Graduação em Informática. Defesa: Curitiba, 23/02/2017Inclui referências : f. 90-98Área de concentração: Ciência da ComputaçãoResumo: Determinar um conjunto de casos de teste que possua alta probabilidade de revelar defeitos em um software é um dos principais objetivos da área de teste de software. Dentre os vários critérios propostos na literatura destaca-se a Análise de Mutantes, uma abordagem promissora devido a sua capacidade em revelar defeitos, embora possua um custo computacional relativamente alto. Com o intuito de reduzir o custo da Análise de Mutantes, estudos empregam a utilização de mutação de ordem superior (Higher Order Mutants, HOMs). O uso de HOMs tem se destacado por reduzir o número de mutantes equivalentes, reduzir o esforço do teste e simular defeitos próximos dos defeitos reais. Entretanto, a geração dos melhores HOMs é uma tarefa complexa, devido ao grande número de mutantes que podem existir e a outros fatores que influenciam a geração, tais como a eficácia dos HOMs gerados. Trabalhos têm aplicado com sucesso técnicas da área da Engenharia de Software baseada em busca por meio da utilização de técnicas de otimização para solucionar esse problema. Entretanto, há ainda a necessidade de possuir um conhecimento sobre o comportamento do problema, de modo a determinar a melhor estratégia a ser utilizada, como projetar e configurar os algoritmos, escolhendo os diferentes operadores de busca e definindo seus parâmetros, para assim melhorar o direcionamento da busca. Neste sentido, o uso de hiper-heurística possibilita uma abordagem mais flexível para automatizar estas tarefas. Além disso, o uso de uma hiper-heurística de seleção de diferentes estratégias existentes para geração de HOMs pode ser útil para reduzir o esforço do testador. Diante disso, este trabalho propõe uma abordagem multi-objetivo que utiliza o conceito de hiper-heurística para gerar conjuntos de HOMs, denominada Hyper-Heuristic for Generation of Higher Order Mutants (HG4HOM). O objetivo é selecionar a menor quantidade de HOMs, os mais difíceis de serem mortos e assim melhorar a eficácia do teste, além de também possibilitar que ao matar um HOM seus FOMs (First Order Mutants) constituintes também possam ser mortos. Para isso, a abordagem é implementada e avaliada com dois algoritmos multi-objetivos: NSGA-II e SPEA2, e três métodos de seleção: Choice-Function (CF), Fitness-Rate-Rank based Multi-Armed Bandit (FRR-MAB) e a seleção aleatória (Random). O algoritmo SPEA2 utilizando o conceito de hiper-heurística juntamente com o método de seleção CF obteve os melhores resultados. Quando comparado com as estratégias tradicionais, a abordagem obteve resultados próximos em relação ao escore de mutação e valor equivalente ao melhor em relação ao tamanho do conjunto de casos de teste adequado. A abordagem obteve as soluções com melhores valores de Euclidean Distance considerando os objetivos relacionados a encontrar a menor quantidade de HOMs, os mais difíceis de serem mortos e capazes de substituírem seus FOMs constituintes. Palavras-chave: Teste de Software, Análise de Mutantes, Mutação de Ordem Superior, Algoritmos Evolutivos Multi-Objetivos, Hiper-Heurística.Abstract: One of the main testing goals is to determine test sets with a high probability of revealing faults. Mutant Analysis is a promising criterion due to its ability to reveal faults, although with a high computational cost. In order to decrease the mutation testing cost, studies employ the use of Higher Order Mutants (HOMs). The use of HOMs can contribute to decrease the number of equivalent mutants, decrease the test effort and simulate faults close to the real ones. However, the generation of the best HOMs is a complex task, due to the large number of mutants that may exist, and to other factors that influence the generation, such as the efficacy of the generated HOMs. To solve such a problem, some works have successfully applied Search-based Software Engineering techniques through the use of optimization techniques. However, it is still needed to have knowledge about the problem behavior, to determine the best strategy to be applied, and to know how to design and configure the algorithms by choosing the different search operators and defining their parameters in order to improve the search. In this sense, the use of hyper-heuristics allows a more flexible approach to automating these tasks. Also, the use of a hyper-heuristic for selection of different existing strategies to generate HOMs can be useful to reduce the tester's effort. Considering all these facts, this work proposes a multi-objective approach, called Hyper-Heuristic for Generation of Higher Order Mutants (HG4HOM), which uses the hyper-heuristic concept to generate sets of HOMs. The goal is to select a small number of HOMs which are difficult to kill, and that contribute to improve the test efficacy, that is, it is desired the test cases that kill the selects HOMs are also capable of killing their corresponding FOMs (First Order Mutants). The approach is implemented and evaluated with two multi-objective algorithms: NSGA-II and SPEA2, and three selection methods: Choice-Function (CF), Fitness-Rate-Rank based Multi-Armed Bandit (FRR-MAB ), and random selection (Random). The SPEA2 algorithm using the hyper-heuristic concept together with the CF selection method obtained the best results. In comparison with respect to the traditional strategies, the approach achieved similar results related to the mutation score and statically equivalent values to the best strategy considering the size of the adequate test case sets. The approach obtained the best results when considering the Euclidean Distance values of the solutions with respect to the goals proposed. Keywords: Software Testing, Mutation Analysis, Higher Order Mutation, Multi-objective Evolutionary Algorithms, Hyper-heuristic

Automatic testing of software with structurally complex inputs

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2005.Includes bibliographical references (p. 123-132).Modern software pervasively uses structurally complex data such as linked data structures. The standard approach to generating test suites for such software, manual generation of the inputs in the suite, is tedious and error-prone. This dissertation proposes a new approach for specifying properties of structurally complex test inputs; presents a technique that automates generation of such inputs; describes the Korat tool that implements this technique for Java; and evaluates the effectiveness of Korat in testing a set of data-structure implementations. Our approach allows the developer to describe the properties of valid test inputs using a familiar implementation language such as Java. Specifically, the user provides an imperative predicate--a piece of code that returns a truth value--that returns true if the input satisfies the required property and false otherwise. Korat implements our technique for solving imperative predicates: given a predicate and a bound on the size of the predicate's inputs, Korat automatically generates the bounded-exhaustive test suite that consists of all inputs, within the given bound, that satisfy the property identified by the predicate. To generate these inputs, Korat systematically searches the bounded input space by executing the predicate on the candidate inputs. Korat does this efficiently by pruning the search based on the predicate's executions and by generating only nonisomorphic inputs. Bounded-exhaustive testing is a methodology for testing the code on all inputs within the given small bound.(cont.) Our experiments on a set of ten linked and array- based data structures show that Korat can efficiently generate bounded-exhaustive test suites from imperative predicates even for very large input spaces. Further, these test suites can achieve high statement, branch, and mutation coverage. The use of our technique for generating structurally complex test inputs also enabled testers in industry to detect faults in real, production-quality applications.by Darko Marinov.Ph.D

Caractérisation de la sûreté de fonctionnement de systèmes à base d'intergiciel

Les systèmes critiques sont soumis, comme le reste de l'industrie informatique, à des contraintes de coût de plus en plus sévères. Cette pression pousse les développeurs à privilégier la réutilisation de logiciels, plutôt que de procéder à des développements spécifiques à chaque projet. Cette tendance à l'utilisation de composants logiciels "sur étagère", souvent développés par des tiers, est renforcée par des besoins technologiques de plus en plus complexes, en particulier l'intégration des systèmes dans des réseaux de communication. L'utilisation de composants sur étagère permet aux industriels de se concentrer sur leur domaine de compétence, sans gaspiller de l'effort à redévelopper des fonctions qui ont déjà été implantées dans d'autres secteurs. Cette tendance à la réutilisation, ainsi que l'interconnexion croissante des systèmes, a favorisé l'émergence de standards d'interface, qui permettent l'interopérabilité de systèmes, même lorsqu'ils sont développés par des groupes différents. L'un des standards d'interface pour l'intégration de systèmes est les intergiciels de communication tels que la plate-forme CORBA. Ces intergiciels facilitent l'interaction entre des applications disparates, s'exécutant sur des plates-formes matérielles et logicielles hétérogènes. Pour les intégrateurs de systèmes distribués, ces technologies sont attractives pour plusieurs raisons, autant technologiques qu'économiques : elles constituent un moyen rapide d'intégration de nouvelles technologies, d'augmentation de la souplesse et l'ouverture vers d'autres systèmes. Toutefois, cette attractivité est conditionnée par des craintes concernant la robustesse des composants intergiciels, qui n'ont pas bénéficié de la rigueur du processus de développement utilisé dans le contexte de systèmes critiques. Les intégrateurs de systèmes distribués semi-critiques souhaitent avoir des assurances sur la qualité et la robustesse des composants qu'ils intègrent au sein de leurs systèmes. Ils souhaitent des informations sur les modes de défaillance de l'intergiciel, sur les canaux de propagation d'erreur qu'il introduit. Ils souhaitent avoir des informations quantitatives leur permettant de comparer différentes implémentations candidates du point de vue de la sûreté de fonctionnement, afin de sélectionner le candidat qui est le mieux adapté à leurs besoins. La problématique que nous venons d'énoncer peut se résumer en deux points : - obtenir une meilleure connaissance des types de fautes et d'erreurs qui existent dans les systèmes à base d'intergiciel de communication ; - développer une méthode permettant de caractériser expérimentalement la robustesse de candidats intergiciels. Il existe actuellement très peu de travaux qui permettent de répondre à ces interrogations. L'objectif de cette thèse est de proposer une méthode de caractérisation qui puisse être appliquée à des intergiciels cibles, et de répondre ainsi aux problèmes des intégrateurs de systèmes semi-critiques, ainsi qu'aux développeurs de composants intergiciel. Notre contribution est de proposer une méthodologie pour l'analyse de la sûreté de fonctionnement d'un intergiciel. Notre méthode est basée sur une analyse structurelle des intergiciels de communication, sur l'élaboration d'un modèle de fautes, une classification des modes de défaillance, et le développement d'un ensemble de techniques d'injection de faute adaptées à l'intergiciel. Nous avons validé notre approche en menant des campagnes d'injection de faute ciblant plusieurs implémentations de la norme CORBA. ABSTRACT : We propose a method for the dependability assessment and failure mode characterization of communications middleware. The method is based on the structural analysis of communications-oriented middleware, the identification of a fault model, a failure modes classification, and the development of a number of fault injection techniques that can be used to target middleware implementations. We have applied our method by carrying out fault injection campaigns targeting a number of CORBA implementations, and obtained quantitative measures of the robustness of the different candidates. Our work allows integrators of dependable distributed systems to obtain assurances on the robustness of the software components they place at the heart of their systems, and provides information to middleware vendors regarding robustness failings in their products

Higher Order Mutation Testing

Mutation testing is a fault-based software testing technique that has been studied widely for over three decades. To date, work in this field has focused largely on first order mutants because it is believed that higher order mutation testing is too computationally expensive to be practical. This thesis argues that some higher order mutants are potentially better able to simulate real world faults and to reveal insights into programming bugs than the restricted class of first order mutants. This thesis proposes a higher order mutation testing paradigm which combines valuable higher order mutants and non-trivial first order mutants together for mutation testing. To overcome the exponential increase in the number of higher order mutants a search process that seeks fit mutants (both first and higher order) from the space of all possible mutants is proposed. A fault-based higher order mutant classification scheme is introduced. Based on different types of fault interactions, this approach classifies higher order mutants into four categories: expected, worsening, fault masking and fault shifting. A search-based approach is then proposed for locating subsuming and strongly subsuming higher order mutants. These mutants are a subset of fault mask and fault shift classes of higher order mutants that are more difficult to kill than their constituent first order mutants. Finally, a hybrid test data generation approach is introduced, which combines the dynamic symbolic execution and search based software testing approaches to generate strongly adequate test data to kill first and higher order mutants

Resilience-Building Technologies: State of Knowledge -- ReSIST NoE Deliverable D12

This document is the first product of work package WP2, "Resilience-building and -scaling technologies", in the programme of jointly executed research (JER) of the ReSIST Network of Excellenc