Assessing the influence of multiple test case selection on mutation experiments

Mutation testing is widely used in experiments.\ud Some papers experiment with mutation directly, while others\ud use it to introduce faults to measure the effectiveness of tests\ud created by other methods. There is some random variation in the\ud mutation score depending on the specific test values used. When\ud generating tests to use in experiments, a common, although not\ud universal practice, is to generate multiple sets of tests to satisfy\ud the same criterion or according to the same procedure, and then\ud to compute their average performance. Averaging over multiple\ud test sets is thought to reduce the variation in the mutation score.\ud This practice is extremely expensive when tests are generated by\ud hand (as is common) and as the number of programs increase (a\ud current positive trend in software engineering experimentation).\ud The research reported in this short paper asks a simple\ud and direct question: do we need to generate multiple sets of\ud test cases? That is, how do different test sets influence the\ud cost and effectiveness results? In a controlled experiment, we\ud generated 10 different test cases to be adequate for the Statement\ud Deletion (SSDL) mutation operator for 39 small programs and\ud functions, and then evaluated how they differ in terms of cost and\ud effectiveness. We found that averaging over multiple programs\ud was effective in reducing the variance in the mutation scores\ud introduced by specific testsFAPESP (número processo 2012/16950-5

Establishing theoretical minimal sets of mutants

Mutation analysis generates tests that distinguish\ud variations, or mutants, of an artifact from the original. Mutation\ud analysis is widely considered to be a powerful approach to testing,\ud and hence is often used to evaluate other test criteria in terms of\ud mutation score, which is the fraction of mutants that are killed\ud by a test set. But mutation analysis is also known to provide\ud large numbers of redundant mutants, and these mutants can\ud inflate the mutation score. While mutation approaches broadly\ud characterized as reduced mutation try to eliminate redundant\ud mutants, the literature lacks a theoretical result that articulates\ud just how many mutants are needed in any given situation. Hence,\ud there is, at present, no way to characterize the contribution\ud of, for example, a particular approach to reduced mutation\ud with respect to any theoretical minimal set of mutants. This\ud paper’s contribution is to provide such a theoretical foundation\ud for mutant set minimization. The central theoretical result of the\ud paper shows how to minimize efficiently mutant sets with respect\ud to a set of test cases. We evaluate our method with a widely-used\ud benchmark.FAPESP (número processo 2012/16950-5

Designing deletion mutation operators

Mutation analysis modifies a program by applying\ud syntactic rules, called mutation operators, systematically to create\ud many versions of the program (mutants) that differ in small\ud ways. Testers then design tests to cause the mutants to behave\ud differently from the original program. Mutation testing is widely\ud considered to result in very effective tests, however, it is also quite\ud costly. Cost comes from the many mutants that are created, the\ud number of tests that are needed to kill the mutants, and the\ud difficulty of deciding whether mutants behave equivalently to\ud the original program. One-op mutation theorizes that cost can be\ud reduced by using a single, very powerful, mutation operator that\ud leads to tests that are almost as effective as if all operators are\ud used. Previous research proposed the statement deletion operator\ud (SDL) and found promising results. This paper investigates the\ud use of SDL-mutation in a new context, the language C, and poses\ud additional empirical questions, including whether other operators\ud can be used. We carried out a controlled experiment in which\ud cost and effectiveness of each individual C mutation operator\ud were collected for 39 different subject programs. Experimental\ud data are used to define a cost-effectiveness metric to choose the\ud best single operator for one-op mutation.FAPESP (número processo 2012/16950-5

Test Case Selection Using CBIR and Clustering

Choosing test cases for the optimization process of information systems testing is crucial, because it helps to eliminate unnecessary and redundant testing data. However, its use in systems that address complex domains (e.g. images) is still underexplored. This paper presents a new approach that uses Content-Based Image Retrieval (CBIR), similarity functions and clustering techniques to select test cases from an image-based test suite. Two experiments performed on an image processing system show that our approach, when compared with random tests, can significantly enhance the performance of tests execution by reducing the test cases required to find a fault. The results also show the potential use of CBIR for information abstraction, as well as the effectiveness of similarity functions and clustering for test case selection

Mutant subsumption graphs

Mutation testing researchers have long known that\ud many generated mutants are not needed. This paper develops a\ud graph model to describe redundancy among mutations. We define\ud “true” subsumption, a relation that practicing test engineers\ud would like to have, but cannot due to issues of computability.\ud We also define dynamic subsumption and static subsumption as\ud approximations of “true” subsumption. We explore the properties\ud of the approximate subsumption relations in the context of a small\ud example. We suggest possible uses for subsumption graphs.FAPESP (número processo 2012/16950-5

Experimental evaluation of SDL and One-Op mutation for C

Mutation analysis modifies a program by applying\ud syntactic rules, called mutation operators, systematically to create\ud many versions of the program (mutants) that differ in small\ud ways. Testers then design tests to cause the mutants to behave\ud differently from the original program. Mutation testing is widely\ud considered to result in very effective tests, however, it is also quite\ud costly. Cost comes from the many mutants that are created, the\ud number of tests that are needed to kill the mutants, and the\ud difficulty of deciding whether mutants behave equivalently to\ud the original program. One-op mutation theorizes that cost can be\ud reduced by using a single, very powerful, mutation operator that\ud leads to tests that are almost as effective as if all operators are\ud used. Previous research proposed the statement deletion operator\ud (SDL) and found promising results. This paper investigates the\ud use of SDL-mutation in a new context, the language C, and poses\ud additional empirical questions, including whether other operators\ud can be used. We carried out a controlled experiment in which\ud cost and effectiveness of each individual C mutation operator\ud were collected for 39 different subject programs. Experimental\ud data are used to define a cost-effectiveness metric to choose the\ud best single operator for one-op mutation.FAPESP (número processo 2012/16950-5

Extratores para oráculos de teste de sistemas texto-fala utilizando recuperação de áudio baseada em conteúdo

Ferramentas automatizadas para teste de sistemas computacionais têm sido desenvolvidas para aplicação em sistemas com saídas tradicionais. Sistemas com saídas complexas, como imagens e sons, ainda constituem um desafio. Este trabalho aplica técnicas de recuperação de áudio baseada em conteúdo no desenvolvimento de extratores de características visando a apoiar o teste de sistemas texto-fala. Os extratores desenvolvidos constituem complementos de um framework para apoiar a criação de oráculos de teste para sistemas com saídas complexas. Os extratores foram validados com sistemas reais e os resultados mostram que a abordagem é promissora para automatização de oráculos de teste para sistemas com saída sonora

Interface mutation: an interprocedural adequacy criterion for integration testing

Um dos pontos fundamentais na atividade de teste de software é o projeto de casos de teste. Diversos critérios de adequação têm sido propostos com o objetivo de fornecer meios que permitam que a avaliação e elaboração de casos de teste sejam feitas de maneira sistemática e fundamentadas teoricamente. Infelizmente, a maioria dos critérios de adequação de casos de teste definidos tem seu uso restrito ao teste de unidade. Para fases posteriores da atividade de teste, em particular para o teste de integração, nota-se a ausência de critérios de adequação, principalmente porque os critérios propostos definem requisitos de teste que se restringem aos limites de uma única unidade, não exercitando de maneira efetiva as interações entre as unidades, que devem ser alvo principal no teste de integração. Com exceção de alguns poucos trabalhos que procuram estender critérios estruturais para o nível interprocedimental, tem-se utilizado nessa fase de teste, quase que exclusivamente, critérios funcionais. Dada essa ausência de critérios e salientando ainda o caráter complementar entre as diferentes técnicas de teste, esta tese apresenta um critério de teste interprocedimental baseado em defeitos chamado de Mutação de Interface. Esse critério busca exercitar as interações entre as unidades através da seleção de casos de teste que distingam mutantes criados pela introdução de defeitos típicos e que, de acordo com um modo definido, caracterizamos erros de integração. Definiu-se um conjunto de operadores de Mutação de Interface que concentram sua aplicação em pontos do programa relacionados com as interações entre as unidades, como, por exemplo, chamadas de subprogramas e seus parâmetros. Dados o alto custo de aplicação, inerente de critérios baseados em mutação, e pelas próprias características do conjunto de operadores de Mutação de Interface, torna-se necessário definirem-se abordagens para reduzir esse custo. Assim, foram estabelecidas maneiras de se parametrizar a aplicação dos operadores de mutação, definindo-se critérios de Mutação de Interface alternativos, estendendo-se abordagem sutilizadas no teste de mutação convencional como mutação restrita. A aplicação de um critério de teste está fortemente condicionada à sua automatização. A definição de um critério de teste sem que pelo menos se apontem soluções para sua automatização tem pouca utilidade prática. Por isso, especificou-se e implementou-se a ferramenta PROTEUM/IM para apoiar a aplicação do critério Mutação de Interface. Essa ferramenta torna-se essencial neste trabalho à medida que permite que estudos empíricos possam ser realizados, avaliando o critério proposto. Dois estudos de caso são apresentados. Esses estudos aplicam o critério Mutação de interface em programas reais e buscam avaliar seu custo e sua eficácia em revelar erros. Estes estudos aplicam ainda critérios alternativos, mostrando que a Mutação de interface é bastante efetiva em revelar erros o de ter custo de aplicação bastante reduzido, quando aplicada de maneira incremental, utilizando-se as parametrizações que os operadores de mutação oferecem.The project of test cases is one of the most important topics in the software testing activity. Several criteria have been proposed aiming at allowing the evaluation and selection of test cases in a systematic and theoretically well founded way. Unfortunately, the use of most of these criteria is restricted to the unit testing phase. For other testing phases, in particular for integration testing. there is a lack of such criteria, mainly because the existing criteria define test requirements only in the scope of a single unit. They arc not able to effectively exercise the interactions between units, what should be the focus of integration testing. Excepting some few works that extend structural criteria to the interprocedural level, only functional testing has been used at integration testing phase. Given this lack of criteria and the complementary characteristics of different testing techniques, this thesis presents an interprocedural fault based criterion named Interface Mutation. This criterion exercises the interactions between units through the selection of test cases that distinguish mutants created by introducing typical faults that characterize integration errors. A set of Interface Mutation operators was defined. The focus of these operators are the points of the program related to the unit interactions, for instance, subprogram calls and their parameters. Given the high cost associated to mutation testing in general and particularly to the Interface Mutation operators, it is necessary to define some approaches to reduce its application cost. Thus, some parameterizations were defined to the mutation operators, allowing to establish alternative Interface Mutation criteria, extending approaches already used in conventional mutation testing, as random mutation and constrained mutation. The application of any testing criterion strongly depends on its automatization. The definition of a criterion, without pointing out ways to its implementation has little practical utility. So, a tool named PROTEUM/IM was specified and implemented to support the application of Interface Mutation. This tool is an essential point in the present work because it allows the conduction of empirical studies aiming at evaluating the proposed criterion. Two case studies arc presented. In these studies the criterion Interface Mutation is applied to real programs and the cost of its application as well as its errors revealing effectiveness are evaluated. Alternative criteria are also used. Showing that Interface Mutation is very effective to reveal errors and can be applied with a reduced cost if used in an incremental way, taking advantage of the parameterization characteristics provided by the Interface Mutation operators set

Not available

Com o avanço da tecnologia de hardware, a Análise de Mutantes um dos critérios de teste baseados em erros tem sido mais recentemente investigada por diversos pesquisadores e se mostrado um critério atrativo e factível para o uso na produção de software. A relevância e necessidade de ferramentas de teste são amplamente reconhecidas pela comunidade e esforços têm sido feitos para o desenvolvimento de ferramentas de apoio a este critério. Nesta dissertação são apresentados os principais aspectos da especificação e implementação de um ambiente de teste multilinguagem, denominado Proteum (Program Testing Using Mutants), que apóia a aplicação do critério Análise de Mutantes para programas em C. Esta ferramenta, além de características comuns a outras ferramentas de teste, procura incorporar algumas características próprias como facilidades para a realização de experimentos empíricos. A Análise de Mutantes, bem como os principais trabalhos relacionados com este critério, são revistos, procurando fornecer subsídios para a caracterização do Proteum. Aspectos funcionais da ferramenta são mostrados, principalmente através de um exemplo que ilustra a operação da ferramenta. A arquitetura e os principais aspectos de implementação desta ferramenta são apresentados. Uma avaliação da ferramenta é feita testando-se alguns programas extraídos da literatura.With the advance in hardware technology, Mutation Analysis an error based testing criteria has been recently investigated by many researchers, and evidences have been found that it is an atractive and practical testing criterion for software production. The necessity and relevance of testing tools are widely recognized and efforts have been carried out for developing tools to support application of this criterion. In this thesis, the mam aspects of specification and implementation of a multilanguage testing tool, named Proteum (Program Testing Using Mutants), which supports aplication of Mutation Analysis criterion for testing C programs, are presented. In addition to common features available in similar tools, Proteum offers facilities for conducting experimental studies. Mutation Analysis, as well as related works, are reviewed, aiming at specifying functional and operational requirements of the Proteum tool. A complete example is given to illustrate the implemented characteristics. An evaluation of the tool, based on testing a set of programs extracted from the literature, is presented

Interface mutation: an interprocedural adequacy criterion for integration testing

Um dos pontos fundamentais na atividade de teste de software é o projeto de casos de teste. Diversos critérios de adequação têm sido propostos com o objetivo de fornecer meios que permitam que a avaliação e elaboração de casos de teste sejam feitas de maneira sistemática e fundamentadas teoricamente. Infelizmente, a maioria dos critérios de adequação de casos de teste definidos tem seu uso restrito ao teste de unidade. Para fases posteriores da atividade de teste, em particular para o teste de integração, nota-se a ausência de critérios de adequação, principalmente porque os critérios propostos definem requisitos de teste que se restringem aos limites de uma única unidade, não exercitando de maneira efetiva as interações entre as unidades, que devem ser alvo principal no teste de integração. Com exceção de alguns poucos trabalhos que procuram estender critérios estruturais para o nível interprocedimental, tem-se utilizado nessa fase de teste, quase que exclusivamente, critérios funcionais. Dada essa ausência de critérios e salientando ainda o caráter complementar entre as diferentes técnicas de teste, esta tese apresenta um critério de teste interprocedimental baseado em defeitos chamado de Mutação de Interface. Esse critério busca exercitar as interações entre as unidades através da seleção de casos de teste que distingam mutantes criados pela introdução de defeitos típicos e que, de acordo com um modo definido, caracterizamos erros de integração. Definiu-se um conjunto de operadores de Mutação de Interface que concentram sua aplicação em pontos do programa relacionados com as interações entre as unidades, como, por exemplo, chamadas de subprogramas e seus parâmetros. Dados o alto custo de aplicação, inerente de critérios baseados em mutação, e pelas próprias características do conjunto de operadores de Mutação de Interface, torna-se necessário definirem-se abordagens para reduzir esse custo. Assim, foram estabelecidas maneiras de se parametrizar a aplicação dos operadores de mutação, definindo-se critérios de Mutação de Interface alternativos, estendendo-se abordagem sutilizadas no teste de mutação convencional como mutação restrita. A aplicação de um critério de teste está fortemente condicionada à sua automatização. A definição de um critério de teste sem que pelo menos se apontem soluções para sua automatização tem pouca utilidade prática. Por isso, especificou-se e implementou-se a ferramenta PROTEUM/IM para apoiar a aplicação do critério Mutação de Interface. Essa ferramenta torna-se essencial neste trabalho à medida que permite que estudos empíricos possam ser realizados, avaliando o critério proposto. Dois estudos de caso são apresentados. Esses estudos aplicam o critério Mutação de interface em programas reais e buscam avaliar seu custo e sua eficácia em revelar erros. Estes estudos aplicam ainda critérios alternativos, mostrando que a Mutação de interface é bastante efetiva em revelar erros o de ter custo de aplicação bastante reduzido, quando aplicada de maneira incremental, utilizando-se as parametrizações que os operadores de mutação oferecem.The project of test cases is one of the most important topics in the software testing activity. Several criteria have been proposed aiming at allowing the evaluation and selection of test cases in a systematic and theoretically well founded way. Unfortunately, the use of most of these criteria is restricted to the unit testing phase. For other testing phases, in particular for integration testing. there is a lack of such criteria, mainly because the existing criteria define test requirements only in the scope of a single unit. They arc not able to effectively exercise the interactions between units, what should be the focus of integration testing. Excepting some few works that extend structural criteria to the interprocedural level, only functional testing has been used at integration testing phase. Given this lack of criteria and the complementary characteristics of different testing techniques, this thesis presents an interprocedural fault based criterion named Interface Mutation. This criterion exercises the interactions between units through the selection of test cases that distinguish mutants created by introducing typical faults that characterize integration errors. A set of Interface Mutation operators was defined. The focus of these operators are the points of the program related to the unit interactions, for instance, subprogram calls and their parameters. Given the high cost associated to mutation testing in general and particularly to the Interface Mutation operators, it is necessary to define some approaches to reduce its application cost. Thus, some parameterizations were defined to the mutation operators, allowing to establish alternative Interface Mutation criteria, extending approaches already used in conventional mutation testing, as random mutation and constrained mutation. The application of any testing criterion strongly depends on its automatization. The definition of a criterion, without pointing out ways to its implementation has little practical utility. So, a tool named PROTEUM/IM was specified and implemented to support the application of Interface Mutation. This tool is an essential point in the present work because it allows the conduction of empirical studies aiming at evaluating the proposed criterion. Two case studies arc presented. In these studies the criterion Interface Mutation is applied to real programs and the cost of its application as well as its errors revealing effectiveness are evaluated. Alternative criteria are also used. Showing that Interface Mutation is very effective to reveal errors and can be applied with a reduced cost if used in an incremental way, taking advantage of the parameterization characteristics provided by the Interface Mutation operators set