Study of trivial compiler equivalence on C++ object-oriented mutation operators

Trivial Compiler Equivalence (TCE) has been recently proposed as an effective technique to detect equivalences between programs, where two or more programs are equivalent if the compiler produces the same binary code. Mutation testing can greatly benefit from TCE as a way to reveal some equivalent and duplicate mutants, which traditionally hinder the applicability of the technique. For instance, previous research has shown that about 28% of the mutants generated by traditional mutation operators in C programs can be removed using TCE. However, the effectiveness of TCE has not been assessed with class-level operators, where the percentage of equivalent mutants is known to be higher than when using traditional ones. In this paper, we present an empirical study on the effectiveness of TCE at identifying equivalent and duplicate mutants using C++ class operators. The results show that TCE is helpful to discard equivalent and duplicate mutants: 241 out of 1,987 (12%) in our study, including 189 out of 684 (27.6%) manually-identified equivalent mutants. Large differences were observed among the different case studies, especially in the detection rate of equivalent mutants, which ranged from 4% to 45%

An experimental and practical study on the equivalent mutant connection: An evolutionary approach

Context: Mutation testing is considered to be a powerful approach to assess and improve the quality of test suites. However, this technique is expensive mainly because some mutants are semantically equivalent to the original program; in general, equivalent mutants require manual revision to differentiate them from useful ones, which is known as the Equivalent Mutant Problem (EMP). Objective: In the past, several authors have proposed different techniques to individually identify certain equivalent mutants, with notable advances in the last years. In our work, by contrast, we address the EMP from a global perspective. Namely, we wonder the extent to which equivalent mutants are connected (i.e., whether they share mutation operators and code areas) as well as the extent to which the knowledge of that connection can benefit the mutant selection process. Such a study could allow going beyond the implicit limit in the traditional individual detection of equivalent mutants. Method: We use an evolutionary algorithm to select the mutants, an approach called Evolutionary Mutation Testing (EMT). We propose a new derived version, Equivalence-Aware EMT (EA-EMT), which penalizes the fitness of known equivalent mutants so that they do not transfer their features to the next generations of mutants. Results: In our experiments applying EMT to well-known C++ programs, we found that (i) equivalent mutants often originate from other equivalent mutants (over 60% on average); (ii) EA-EMT’s approach of penalizing known equivalent mutants provides better results than the original EMT in most of the cases (notably, the more equivalent mutants are detected, the better); and (iii) we can combine EA-EMT with Trivial Compiler Equivalence as a way to automatically identify equivalent mutants in a real situation, reaching a more stable version of EMT. Conclusions: This novel approach opens the way for improvement in other related areas that deal with equivalent versions.This work is partially funded by the European Commission (FEDER), the Spanish Ministry of Science, Innovation and Universities (RTI2018-093608-B-C33), the Spanish Ministry of Innovation and Competitiveness (TIN2017-88213-R), and the University of Malaga (Exhauro project)

Exploring and Assessing the Trivial Compiler Equivalence

Mutation testing is the state-of-the-art technique to assess the fault-detection capability of a test suite. However, its adoption in industry is deterred by few of its inherent limitations including the equivalent mutants. Since the equivalent mutants are functionally similar to the original program, the test suite cannot kill them, hence they produce false alarms for the developers and reduce the mutation score. Although to automatically verify whether the mutant is equivalent to the original program is undecidable, yet there exist heuristics such as trivial compiler equivalence to automatically eliminate sufficient equivalent mutants. In this paper, we explore the use of compiler optimizations at assembly level code to detect equivalent mutants and find that it can indeed detect equivalent mutant

Mutation-inspired symbolic execution for software testing

Software testing is a complex and costly stage during the software development lifecycle. Nowadays, there is a wide variety of solutions to reduce testing costs and improve test quality. Focussing on test case generation, Dynamic Symbolic Execution (DSE) is used to generate tests with good structural coverage. Regarding test suite evaluation, Mutation Testing (MT) assesses the detection capability of the test cases by introducing minor localised changes that resemble real faults. DSE is however known to produce tests that do not have good mutation detection capabilities: in this paper, the authors set out to solve this by combining DSE and MT into a new family of approaches that the authors call Mutation-Inspired Symbolic Execution (MISE). First, this known result on a set of open source programs is confirmed: DSE by itself is not good at killing mutants, detecting only 59.9% out of all mutants. The authors show that a direct combination of DSE and MT (naive MISE) can produce better results, detecting up to 16% more mutants depending on the programme, though at a high computational cost. To reduce these costs, the authors set out a roadmap for more efficient versions of MISE, gaining its advantages while avoiding a large part of its additional costs

Mutation-based genetic improvement of software

Genetic Improvement (GI) of software is a recent field that has drawn much attention from Software Engineering researchers. It aims to use search techniques to automatically modify and improve existing software. The drawback in previous GI approaches is scalability of these approaches, due to the large search space formed by the code base in real-world systems. To overcome the scalability challenge, more recent studies have confined the granularity of code modification at the statement level and applied a prior sensitivity analysis to further reduce the search space. However, some software improvements may require code changes at a finer level of granularity. This thesis demonstrates that, by combining with Mutation Testing techniques, GI can operate at this finer granularity while preserving scalability. The thesis applies Mutation Operators to automatically modify the source code of the target software. After a prior sensitivity analysis on First Order Mutants, "deep" (previously unavailable) parameters are exposed from the most sensitive locations, followed by a bi-objective optimisation process to fine tune them together with existing ("shallow") parameters. The objective is to improve both time and memory resources required by the computation. Since this approach relies on the selection of Mutation Operators and traditional Mutation Operators are not concerned with memory performance, the thesis proposes and evaluates Memory Mutation Operators in the Mutation Testing context. Using both traditional and Memory Mutation Operators, the thesis further seeks to improve the target software by searching for Higher Order Mutants (HOMs). The thesis presents the result of a code analysis study, which reveals that, among all the code modifications that contribute to the improvement, more than half of them require a finer control of the code, which our approach is better at than previous GI approaches

Técnicas de prueba avanzadas para la generación de casos de prueba

Software testing is a crucial phase in software development, particularly in contexts such as critical systems, where even minor errors can have severe consequences. The advent of Industry 4.0 brings new challenges, with software present in almost all industrial systems. Overcoming technical limitations, as well as limited development times and budgets, is a major challenge that software testing faces nowadays. Such limitations can result in insufficient attention being paid to it. The Bay of Cadiz’s industrial sector is known for its world-leading technological projects, with facilities and staff fully committed to innovation. The close relationship between these companies and the University of Cadiz allows for a constant exchange between industry and academia. This PhD thesis aims to identify the most important elements of software testing in Industry 4.0, based on close industrial experience and the latest state-of-the-art work. This allows us to break down the software testing process in a context where large teams work on large-scale, changing projects with numerous dependencies. It also allows us to estimate the percentage benefit that a solution could provide to test engineers throughout the process. Our results indicate a need for non-commercial, flexible, and adaptable solutions for the automation of software testing, capable of meeting the constantly changing needs of industry projects. This work provides a comprehensive study on the industry’s needs and motivates the development of two new solutions using state-of-the-art technologies, which are rarely present in industrial work. These results include a tool, ASkeleTon, which implements a procedure for generating test harnesses based on the Abstract Syntax Tree (AST) and a study examining the ability of the Dynamic Symbolic Execution (DSE) testing technique to generate test data capable of detecting potential faults in software. This study leads to the creation of a novel family of testing techniques, called mutationinspired symbolic execution (MISE), which combines DSE with mutation testing (MT) to produce test data capable of detecting more potential faults than DSE alone. The findings of this work can serve as a reference for future research on software testing in Industry 4.0. The solutions developed in this PhD thesis are able to automate essential tasks in software testing, resulting in significant potential benefits. These benefits are not only for the industry, but the creation of the new family of testing techniques also represents a promising line of research for the scientific community, benefiting all software projects regardless of their field of application.La prueba del software es una de las etapas más importantes durante el desarrollo de software, especialmente en determinados tipos de contextos como el de los sistemas críticos, donde el más mínimo fallo puede conllevar la más grave de las consecuencias. Nuevos paradigmas tecnológicos como la Industria 4.0 conllevan desafíos que nunca antes se habían planteado, donde el software está presente en prácticamente todos los sistemas industriales. Uno de los desafíos más importantes a los que se enfrenta la prueba del software consiste en superar las limitaciones técnicas además de los tiempos de desarrollo y presupuestos limitados, que provocan que en ocasiones no se le preste la atención que merece. El tejido industrial de la Bahía de Cádiz es conocido por sacar adelante proyectos tecnológicos punteros a nivel mundial, con unas instalaciones y un personal totalmente implicado con la innovación. Las buenas relaciones de este conjunto de empresas con la Universidad de Cádiz, sumadas a la cercanía geográfica, permiten que haya una conversación constante entre la industria y la academia. Este trabajo de tesis persigue identificar los elementos más importantes del desarrollo de la prueba del software en la Industria 4.0 en base a una experiencia industrial cercana, además de a los últimos trabajos del estado del arte. Esto permite identificar cada etapa en la que se desglosa la prueba del software en un contexto donde trabajan equipos muy grandes con proyectos de gran envergadura, cambiantes y con multitud de dependencias. Esto permite, además, estimar el porcentaje de beneficio que podría suponer una solución que ayude a los ingenieros de prueba durante todo el proceso. Gracias a los resultados de esta experiencia descubrimos que existe la necesidad de soluciones para la automatización de la prueba del software que sean no comerciales, flexibles y adaptables a las constantes necesidades cambiantes entre los proyectos de la industria. Este trabajo aporta un estudio completo sobre las necesidades de la industria en relación a la prueba del software. Los resultados motivan el desarrollo de dos nuevas soluciones que utilizan tecnologías del estado del arte, ampliamente usadas en trabajos académicos, pero raramente presentes en trabajos industriales. En este sentido, se presentan dos resultados principales que incluyen una herramienta que implementa un procedimiento para la generación de arneses de prueba basada en el Árbol de Sintaxis Abstracta (AST) a la que llamamos ASkeleTon y un estudio donde se comprueba la capacidad de la técnica de pruebas Ejecución Simbólica Dinámica (DSE, por sus siglas en inglés) para generar datos de prueba capaces de detectar fallos potenciales en el software. Este estudio deriva en la creación de una novedosa familia de técnicas de prueba a la que llamamos mutation-inspired symbolic execution (MISE) que combina DSE con la prueba de mutaciones (MT, por sus siglas en inglés) para conseguir un conjunto de datos de prueba capaz de detectar más fallos potenciales que DSE por sí sola. Las soluciones desarrolladas en este trabajo de tesis son capaces de automatizar parte de la prueba del software, resultando en unos beneficios potenciales importantes. No solo se aportan beneficios a la industria, sino que la creación de la nueva familia de técnicas de prueba supone una línea de investigación prometedora para la comunidad científica, siendo beneficiados todos los proyectos software independientemente de su ámbito de aplicación

Mutation‐inspired symbolic execution for software testing

Software testing is a complex and costly stage during the software development lifecycle. Nowadays, there is a wide variety of solutions to reduce testing costs and improve test quality. Focussing on test case generation, Dynamic Symbolic Execution (DSE) is used to generate tests with good structural coverage. Regarding test suite evaluation, Mutation Testing (MT) assesses the detection capability of the test cases by introducing minor localised changes that resemble real faults. DSE is however known to produce tests that do not have good mutation detection capabilities: in this paper, the authors set out to solve this by combining DSE and MT into a new family of approaches that the authors call Mutation‐Inspired Symbolic Execution (MISE). First, this known result on a set of open source programs is confirmed: DSE by itself is not good at killing mutants, detecting only 59.9% out of all mutants. The authors show that a direct combination of DSE and MT (naive MISE) can produce better results, detecting up to 16% more mutants depending on the programme, though at a high computational cost. To reduce these costs, the authors set out a roadmap for more efficient versions of MISE, gaining its advantages while avoiding a large part of its additional costs