Design of Mutation Operators for Testing Geographic Information Systems

[Abstract] In this article, we propose the definition of specific mutation operators for testing Geographic Information Systems. We describe the process for applying the operators and generating mutants, and present a case study where these mutation operators are applied to two real-world applications.Xunta de Galicia; ED431G/01Xunta de Galicia; ED431C 2017/58Ministerio de Economía y Competitividad; TIN2016-78011-c4-1-RMinisterio de Ciencia, Innovación e Universidades; BIZDEVOPS-GLOBAL; RTI2018-098309-B-C3

Object-Oriented Mutation Applied in Common Intermediate Language Programs Originated from C#

Abstract-Application of object-oriented mutation operators in C# programs using a parser-based tool can be precise but requires compilation of mutants. Mutations can be introduced faster directly to the Common Intermediate Language of .NET. It can be simple for traditional mutation operators but more complicated for the object-oriented ones. We propose the reconstruction of complex object-oriented faults on the intermediate language level. The approach was tested in the ILMutator tool implementing few object-oriented mutation operators in the intermediate code derived from compiled C# programs. Exemplary mutation and performance results are given and compared to results of the parser-based mutation tool CREAM

Testing database applications using coverage analysis and mutation analysis

Database applications are built using two different programming language constructs: one that controls the behavior of the application, also referred to as the host language; and the other that allows the application to access/retrieve information from the back-end database, also referred to as the query language. The interplay between these two languages makes testing of database applications a challenging process. Independent approaches have been developed to evaluate test case quality for host languages and query languages. Typically, the quality of test cases for the host language (e.g., Java) is evaluated on the basis of the number of lines, statements and blocks covered by the test cases. High quality test cases for host languages can be automatically generated using recently developed concolic testing techniques, which rely on manipulating and guiding the search of test cases based on carefully comparing the concrete and symbolic execution of the program written in the host language. Query language test case quality (e.g., SQL), on the other hand, is evaluated using mutation analysis, which is considered to be a stronger criterion for assessing quality. In this case, several mutants or variants of the original SQL query are generated and the quality is measured using a metric called mutation score. The score indicates the percentage of mutants that can be identified in terms of their results using the given test cases. Higher mutation score indicates higher quality for the test cases. In this thesis we present novel testing strategy which guides concolic testing using mutation analysis for test case (which includes both program input and synthetic data) generation for database applications. The novelty of this work is that it ensures that the test cases are of high quality not only in terms of coverage of code written in the host language, but also in terms of mutant detection of the queries written in the query language

A Novel Approach to Mutation Operator Design for MDE Languages

Due to the increasing reliance on the software of systems, such as enterprise systems, a wide array of approaches has been found to facilitate the development of software for such systems. The growth of system complexity, however, has provoked concerns about the quality of the software. One approach that copes with complexity is model driven engineering that uses models containing only essential domain concepts, in order to represent complex systems. With some level of automation, models are then maintained by programs that are prone to error, as they are man-made. In order to find errors in programs, software engineers use mutation testing to build strong test inputs by introducing faults into the tested software using mutation operators. They then study if the introduced faults are detected by the test inputs. There have been few attempts to design mutation operators for model driven languages, which have common metamodeling language models, compared with traditional programming languages. This thesis presents an effective language-agnostic approach for mutation operator design for the rapidly emerging model driven engineering languages by considering metamodeling languages. The approach produces generic operators that can be instantiated to generate concrete ones for a given language model, which can be used to mutate program models that conform to the language model. The approach and generic operators are evaluated using empirical mutation analysis experiments over programs written in the ATL and EOL languages. The results show that the generic operators generated from the approach are instantiatable over ATL and EOL metamodels and have produced low proportions of invalid and equivalent mutants that can impact negatively on any mutation testing process. Also, the generic operators have produced useful mutants such as live and not trivially detected kinds of mutants

Implementación de operadores de mutación para WS-BPEL 2.0

En este proyecto fin de carrera se completa una herramienta real denominada GAmera desarrollada en el grupo de investigación SPI&FM. GAmera es una herramienta de generación de mutantes para WS-BPEL que en vez de generarlos todos genera sólo un subconjunto de ellos. Dicha herramienta se divide en dos partes: un algoritmo genético que dirige todo el proceso y un entorno que compara los mutantes con el proceso original, para comprobar si su comportamiento varía o no. Este PFC se enmarca dentro de esta segunda parte

Higher Order Mutation Testing

Mutation testing is a fault-based software testing technique that has been studied widely for over three decades. To date, work in this field has focused largely on first order mutants because it is believed that higher order mutation testing is too computationally expensive to be practical. This thesis argues that some higher order mutants are potentially better able to simulate real world faults and to reveal insights into programming bugs than the restricted class of first order mutants. This thesis proposes a higher order mutation testing paradigm which combines valuable higher order mutants and non-trivial first order mutants together for mutation testing. To overcome the exponential increase in the number of higher order mutants a search process that seeks fit mutants (both first and higher order) from the space of all possible mutants is proposed. A fault-based higher order mutant classification scheme is introduced. Based on different types of fault interactions, this approach classifies higher order mutants into four categories: expected, worsening, fault masking and fault shifting. A search-based approach is then proposed for locating subsuming and strongly subsuming higher order mutants. These mutants are a subset of fault mask and fault shift classes of higher order mutants that are more difficult to kill than their constituent first order mutants. Finally, a hybrid test data generation approach is introduced, which combines the dynamic symbolic execution and search based software testing approaches to generate strongly adequate test data to kill first and higher order mutants

Evolutionary Mutation Testing in Object-Oriented Environments

La prueba de mutaciones es reconocida como un potente método para evaluar la fortaleza de un conjunto de casos de prueba en la detección de posibles fallos en el código. No obstante, la aplicación de esta técnica es costosa, lo cual ha supuesto normalmente un obstáculo para una mayor acogida de la misma por parte de la industria. Varias técnicas han mostrado ser capaces de reducir ampliamente su coste sin mucha pérdida de efectividad, pero también es cierto que estas técnicas solo han sido evaluadas en determinados contextos, especialmente en el ámbito de los operadores de mutación tradicionales para programas procedurales. Por ejemplo, la Prueba de Mutación Evolutiva ha sido aplicada únicamente a composiciones WS-BPEL, a pesar de que se obtuvo un resultado positivo al seleccionar un subconjunto de mutantes a través de un algoritmo evolutivo a fin de mejorar el conjunto de casos de prueba. Como resultado, se desconoce a día de hoy si los mismos beneficios pueden extrapolarse a otros niveles y dominios. En particular, en esta tesis nos preguntamos hasta qué punto la Prueba de Mutación Evolutiva es también útil para reducir el número de mutantes en sistemas orientados a objetos. Más específicamente, nos enfocamos en el lenguaje de programación C++, ya que la prueba de mutaciones casi no se ha desarrollado respecto a este popular lenguaje a juzgar por la falta de artículos de investigación en este campo que se dirigen este lenguaje. Dado que C++ ha sido apenas abordado en cuanto a investigación y en cuanto a la práctica, en esta tesis nos ocupamos de todas las fases de la prueba de mutaciones: desde la definición e implementación de operadores de mutación en un sistema de mutaciones, hasta la evaluación de esos operadores y la aplicación de la Prueba de Mutación Evolutiva entre otras técnicas de reducción del coste. En esta tesis definimos e implementamos un conjunto de operadores de mutación de clase para C++ en MuCPP, herramienta de mutaciones que nos permite llevar a cabo experimentos con programas reales gracias a las características incorporadas a la misma. Estos operadores de mutación son automatizados siguiendo un conjunto de reglas para que produzcan los mutantes que se esperan de los mismos. En términos generales, los operadores de clase generan bastantes menos mutantes que los operadores tradicionales, un porcentaje mayor de mutantes equivalentes y se aplican con diversa frecuencia dependiendo de las características del programa analizado. El desarrollo de reglas de mejora en la implementación de los operadores permite reducir incluso más el número de mutantes, evitando generar mutantes que no son interesantes para el propósito de la prueba de mutaciones. Otro descubrimiento interesante es que el conjunto de mutantes de clase y el de mutantes tradicionales se complementan, ayudando a diseñar un conjunto de casos de prueba más efectivo. También desarrollamos GiGAn, un nuevo sistema para conectar MuCPP y un algoritmo genético para aplicar la Prueba de Mutación Evolutiva a sistemas orientados a objetos en C++. El algoritmo genético permite reducir el número de mutantes que sería generado por MuCPP ya que guía la búsqueda a la selección de aquellos mutantes que pueden inducir a la generación de nuevos casos de prueba (mutantes fuertes). El rendimiento de esta técnica se muestra mejor que el de un algoritmo aleatorio, tanto cuando se buscan diferentes porcentajes de mutantes fuertes como cuando se simula el refinamiento del conjunto de casos de prueba mediante los mutantes seleccionados por ambas técnicas. La estabilidad de la Prueba de Mutación Evolutiva en los diferentes programas analizados y los buenos resultados en aquellos programas de los que se deriva un mayor número de mutantes son observaciones adicionales. Finalmente, realizamos experimentos para evaluar de forma individual a estos operadores de mutación desde una doble perspectiva: cómo de útiles son para la evaluación (TSE) y para la mejora (TSR) de un conjunto de casos de prueba. Para ello clasificamos a los operadores usando dos métricas distintas: el grado de redundancia (TSE) y la calidad para guiar a la generación de casos de prueba de alta calidad (TSR). Siguiendo estas clasificaciones, ponemos en práctica un estudio selectivo teniendo en cuenta que los operadores menos valiosos están en las últimas posiciones. Este enfoque selectivo revela que los operadores no son necesariamente igual de útiles para TSE y TSR, y que estas clasificaciones son apropiadas para llevar a cabo una estrategia selectiva cuando lo comparamos con la aplicación de otras clasificaciones de operadores o la selección aleatoria de mutantes. Sin embargo, favorecer la generación de mutantes individuales a partir de los operadores mejor valorados es mucha mejor opción que descartar operadores al completo debido a que cada uno de estos operadores se centra en una característica concreta del paradigma de orientación a objetos. En conjunto, todas estas evaluaciones en torno a estos operadores de clase sugieren que la naturaleza de los mismos puede limitar los beneficios de aplicar cualquier técnica de reducción del coste.Mutation testing is acknowledged as a powerful method to evaluate the strength of test suites in detecting possible faults in the code. However, its application is expensive, which has traditionally been an obstacle for a broader use in the industry. While it is true that several techniques have shown to greatly reduce the cost without losing much effectiveness, it is also true that those techniques have been evaluated in limited contexts, especially in the scope of traditional operators for procedural programs. To illustrate this fact, Evolutionary Mutation Testing has only been applied to WS-BPEL compositions, despite the positive outcome when selecting a subset of mutants through an evolutionary algorithm with the aim of improving a test suite. As a result, it is unknown whether the same benefits can be extrapolated to other levels and domains. In particular, we wonder in this thesis to what extent Evolutionary Mutation Testing is also useful to reduce the number of mutants generated by class mutation operators in object-oriented systems. More specifically, we focus on the C++ programming language, since the development of mutation testing with regard to this widely-used language is clearly immature judging from the lack of papers in the literature tackling this language. Given that C++ has been hardly addressed in research and practice, we deal with all the phases of mutation testing: from the definition and implementation of mutation operators in a mutation system to the evaluation of those operators and the application of Evolutionary Mutation Testing among other cost reduction techniques. We define a set of class mutation operators for C++ and implement them in MuCPP, which allows us to perform experiments with real programs thanks to the facilities incorporated into this mutation tool. These mutation operators are automated following a set of guidelines so that they produce the expected mutations. In general, class-level operators generate far fewer mutants than traditional operators, a higher equivalence percentage and they are applied with varying frequency depending on the features of the tested program. Developing improvement rules in the implementation of several mutation operators help further reduce the number of mutants, avoiding the creation of uninteresting mutants. Another interesting finding is that the set of class mutants and the set of traditional mutants complement each other to help the tester design more effective test suites. We also develop GiGAn, a new system to connect the mutation tool MuCPP and a genetic algorithm to apply Evolutionary Mutation Testing to C++ object-oriented systems. The genetic algorithm allows reducing the number of mutants that would be generated by MuCPP as it guides to the selection of those mutants that can induce the generation of new test cases (strong mutants). The performance of this technique shows to be better than the application of a random algorithm, both when trying to find different percentages of strong mutants and also when simulating the refinement of the test suite through the mutants selected by each of these techniques. The stability of EMT among different case studies and the good results of the simulation in the programs that lead to the largest set of mutants are additional observations. Finally, we conduct an experiment to assess individually these mutation operators from a double perspective: how useful they are for the evaluation of the test suite (TSE) and its refinement (TSR). To that end, we rank the operators using two different metrics: degree of redundancy (TSE) and quality to guide on the generation of high-quality test cases (TSR). Based on these rankings, we perform a selective study taking into account that the less valuable operators are at the bottom of the classification. This selective approach reveals that an operator is not necessarily as useful for TSE as for TSR, and that these rankings are appropriate for a selective strategy when compared to other rankings or the selection of mutants randomly. However, favouring the generation of individual mutants from the best-valued operators is much better than discarding operators completely because each of the operators targets a particular object-oriented feature. Altogether, these evaluations about class operators suggest that their nature can limit the benefits of any cost reduction technique.Este trabajo fue financiado por la beca de investigación PU-EPIF-FPI-PPI-BC 2012- 037 de la Universidad de Cádiz, por el proyecto DArDOS (TIN2015-65845-C3-3-R) del Programa Estatal de Investigación, Desarrollo e Innovación Orientada a los Retos de la Sociedad del Ministerio de Economía y Competitividad, y por la Red de Excelencia SEBASENET (TIN2015-71841-REDT) del Programa Estatal de Fomento de la Investigación Científica y Técnica de Excelencia del Ministerio de Economía y Competitividad.Número de páginas: 23

Automatic Generation of Cases in Event Processing using EPL

https://prezi.com/ikn8ub6nth1u/generacion-automatica-de-casos-en-procesamiento-de-eventos-con-epl/ (presentación)La aplicación en diversas áreas de Internet de las Cosas (IoT) ha ido en aumento en los últimos años. Uno de los principales inconvenientes que tienen los sistemas IoT es la cantidad de información que tienen que manejar. Esta información llega en forma de eventos, cuyo receptor ha de tomar las decisiones correctas, en tiempo real, según los datos recibidos. A raíz de estos nuevos sistemas, surgen nuevas herramientas, dispositivos, lenguajes y mecanismos para obtener, procesar y transmitir esta información. Entre estas novedades se destacan los lenguajes de programación "Event Processing Language" (EPL), que se desarrollaron para detectar las situaciones de interés sobre un dominio concreto en tiempo real. Estos lenguajes utilizan patrones para describir las situaciones en las que se quiere filtrar la información, para realizar las actuaciones correctas. Los lenguajes EPL procesan y analizan grandes cantidades de datos (eventos) en tiempo real, por lo que cualquier fallo de programación podría afectar gravemente en la toma de decisiones. Viendo la relevancia que tiene el procesado de esta información, resulta fundamental analizar y probar programas escritos en estos lenguajes de programación, detectando los posibles fallos más comunes que los programadores pueden cometer. Para poder hacer cualquier tipo de prueba en sistemas que utilicen EPL, se necesita una gran cantidad de eventos con estructuras y valores específicos. Conseguir estos eventos de forma manual puede ser una tarea muy costosa y propensa a errores. Esta tesis aborda el problema de la generación automática de los casos de prueba proponiendo y empleando un método para la generación automática de eventos, el cual se comparará con los sistemas de generación de eventos disponibles. El método incluye una propuesta de especificación para la definición de tipos de eventos y un generador de eventos basado en la especificación anterior. Con este método se obtienen casos de prueba, que van a permitir evaluar situaciones críticas de las aplicaciones, ya que los eventos pueden ser generados con valores concretos que ayuden a simularlas. Adicionalmente, para validar el método propuesto, se escoge el lenguaje de consulta EPL de la empresa EsperTech y se aplica la prueba de mutaciones en programas que lanzan este tipo de consultas. Entre los motivos de la elección del lenguaje EPL de EsperTech se destaca que es de código abierto y que su sintaxis se aproxima bastante al lenguaje SQL (ampliamente conocido). Por otro lado, se escoge la prueba de mutaciones ya que esta ha sido aplicada con éxito a una gran variedad de lenguajes de programación y mostramos que es una técnica de prueba adecuada para verificar y validar estos programas. Se generan multitud de variantes del programa a probar que representan los fallos máscomunes de programación. Para poder aplicar la prueba de mutaciones se requiere de una gran cantidad de casos de prueba. Dado que la prueba de mutaciones no había sido aplicada anteriormente a este lenguaje, esta tesis afronta el proceso completo de aplicación de esta técnica a un lenguaje: definición de operadores de mutación, desarrollo de una herramienta de generación y ejecución automática de los mutantes y evaluación de los operadores definidos (comprobando su frecuencia de aplicación en casos de estudio).Internet of Things (IoT) has increasingly become popular in diferent areas. One of the main drawbacks of the IoT systems is the amount of information they have to handle. This information arrives as events that need to be processed in real-time in order to make correct decisions. As a consequence, new ways (tools, devices, mechanisms...) of obtaining, processing and transmitting information have to be put into action. It is worth mentioning the "Event Processing Languages" (EPL), which were created to detect, in real-time, interesting situations in a particular domain. These languages use patterns to filter the information (good knowledge of the EPL languages is needed). A huge amount of data is processed and analysed by EPLs, so any programmer error could seriously affect the outcome because of a poor decision-making system. Given that processing the data is crucial, testing and analysing programs that run any EPL language is required. The most common mistakes that programmers could make have to be detected. A large number of events with specific values and structures are needed to apply any kind of testing in programs which use EPL. As this is a very hard task and very prone to error if done by hand, this dissertation addresses the automated generation of test cases; a method to automatically generate events has been proposed and used. Moreover, it will be compared to existing event generation systems. This method includes a general definition of what is an event type and its representation is proposed. Additionally, an event generator is developed based on this definition. Test cases to evaluate critical situations the EPL programs may suffer are obtained following this method. The proposed method allows the generation of events with specific values and structures, which will simulate these critical situations. In order to validate the proposed method, the EPL of EsperTech company is selected to apply mutation testing to programs which run the EsperTech EPL queries. EsperTech EPL language has been selected because it is open source and its syntax is very similar to SQL language (very popular programming language). Mutation testing is the chosen testing technique because it has been successfully applied to various programming languages. Mutation testing is a suitable testing technique to verify and validate these programs. Several variations of the original program are generated, which include "typical" programming errors. In order to apply mutation testing, a huge amount of data is required. Due to mutation testing has not been applied to EsperTech EPL language before, this dissertation faces the complete process of applying the mutation testing technique toa programming language: mutation operators definition, development of a tool which automatically generates and executes mutants and evaluation of the defined operators (checking their application frequency in studies cases).Documento PDF de 328 páginas https://dialnet.unirioja.es/servlet/tesis?codigo=71530 https://www.researchgate.net/profile/L_Gutierrez-Madronal/publication/318394697_Generacion_Automatica_de_Casos_en_Procesamiento_de_Eventos_con_EPL_Automatic_Generation_of_Cases_in_Event_Processing_using_EPL/links/59675223458515e9af9e9e55/Generacion-Automatica-de-Casos-en-Procesamiento-de-Eventos-con-EPL-Automatic-Generation-of-Cases-in-Event-Processing-using-EPL.pd