Semantic Mutation Testing for Multi-Agent Systems

This paper introduces semantic mutation testing (SMT) into multiagent systems. SMT is a test assessment technique that makes changes to the interpretation of a program and then examines whether a given test set has the ability to detect each change to the original interpretation. These changes represent possible misunderstandings of how the program is interpreted. SMT is also a technique for assessing the robustness of a program to semantic changes. This paper applies SMT to three rule-based agent programming languages, namely Jason, GOAL and 2APL, provides several contexts in which SMT for these languages is useful, and proposes three sets of semantic mutation operators (i.e., rules to make semantic changes) for these languages respectively, and a set of semantic mutation operator classes for rule-based agent languages. This paper then shows, through preliminary evaluation of our semantic mutation operators for Jason, that SMT has some potential to assess tests and program robustness

Quantitative metrics for mutation testing

Program mutation is the process of generating versions of a base program by applying elementary syntactic modifications; this technique has been used in program testing in a variety of applications, most notably to assess the quality of a test data set. A good test set will discover the difference between the original program and mutant except if the mutant is semantically equivalent to the original program, despite being syntactically distinct. Equivalent mutants are a major nuisance in the practice of mutation testing, because they introduce a significant amount of bias and uncertainty in the analysis of test results; indeed, mutants are useful only to the extent that they define distinct functions from the base program. Yet, despite several decades of research, the identification of equivalent mutants remains a tedious, inefficient, ineffective and error prone process. The approach that is adopted in this dissertation is to turn away from the goal of identifying individual mutants which are semantically equivalent to the base program, in favor of an approach that merely focuses on estimating their number. To this effect, the following question is considered: what makes a base program P prone to produce equivalent mutants? The position taken in this work is that what makes a program prone to generate equivalent mutants is the same property that makes a program fault tolerant, since fault tolerance is by definition the ability to maintain correct behavior despite the presence and sensitization of faults; whether these faults stem from poor design or from mutation operators does not matter. Hence if we could only quantify the redundancy of a program, we should be able to use the redundancy metrics to estimate the ratio of equivalent mutants (REM for short) of a program. Using redundancy metrics that were previously defined to reflect the state redundancy of a program, its functional redundancy, its non injectivity and its non-determinacy, this dissertation makes the following contributions: The design and implementation of a Java compiler, using compiler generation technology, to analyze Java code and compute its redundancy metrics. An empirical study on standard mutation testing benchmarks to analyze the statistical relationships between the REM of a program and its redundancy metrics. The derivation of regression models to estimate the REM of a program from its compiler generated redundancy metrics, for a variety of mutation policies. The use of the REM to address a number of mutation related issues, including: estimating the level of redundancy between non-equivalent mutants; redefining the mutation score of a test data set to take into account the possibility that mutants may be semantically equivalent to each other; using the REM to derive a minimal set of mutants without having to analyze all the pairs of mutants for equivalence. The main conclusions of this work are the following: The REM plays a very important role in the mutation analysis of a program, as it gives many useful insights into the properties of its mutants. All the attributes that can be computed from the REM of a program are very sensitive to the exact value of the REM; Hence the REM must be estimated with great precision. Consequently, the focus of future research is to revisit the Java compiler and enhance the precision of its estimation of redundancy metrics, and to revisit the regression models accordingly

Mutation for Multi-Agent Systems

Although much progress has been made in engineering multi-agent systems (MAS), many issues remain to be resolved. One issue is that there is a lack of techniques that can adequately evaluate the effectiveness (fault detection ability) of tests or testing techniques for MAS. Another is that there are no systematic approaches to evaluating the impact of possible semantic changes (changes in the interpretation of agent programs) on agents' behaviour and performance. This thesis introduces syntactic and semantic mutation to address these two issues. Syntactic mutation is a technique that systematically generates variants ("syntactic mutants") of a description (usually a program) following a set of rules ("syntactic mutation operators"). Each mutant is expected to simulate a real description fault, therefore, the effectiveness of a test set can be evaluated by checking whether it can detect each simulated fault, in other words, distinguish the original description from each mutant. Although syntactic mutation is widely considered very effective, only limited work has been done to introduce it into MAS. This thesis extends syntactic mutation for MAS by proposing a set of syntactic mutation operators for the Jason agent language and showing that they can be used to generate real faults in Jason agent programs. By contrast, semantic mutation systematically generates variant interpretations ("semantic mutants") of a description following a set of rules ("semantic mutation operators"). Semantic mutation has two uses: to evaluate the effectiveness of a test set by simulating faults caused by misunderstandings of how the description is interpreted, and to evaluate the impact of possible semantic changes on agents' behaviour and performance. This thesis, for the first time, proposes semantic mutation for MAS, more specifically, for three logic based agent languages, namely Jason, GOAL and 2APL. It proposes semantic mutation operators for these languages, shows that the operators for Jason can represent real misunderstandings and are practically useful

Automatic Generation of Cases in Event Processing using EPL

https://prezi.com/ikn8ub6nth1u/generacion-automatica-de-casos-en-procesamiento-de-eventos-con-epl/ (presentación)La aplicación en diversas áreas de Internet de las Cosas (IoT) ha ido en aumento en los últimos años. Uno de los principales inconvenientes que tienen los sistemas IoT es la cantidad de información que tienen que manejar. Esta información llega en forma de eventos, cuyo receptor ha de tomar las decisiones correctas, en tiempo real, según los datos recibidos. A raíz de estos nuevos sistemas, surgen nuevas herramientas, dispositivos, lenguajes y mecanismos para obtener, procesar y transmitir esta información. Entre estas novedades se destacan los lenguajes de programación "Event Processing Language" (EPL), que se desarrollaron para detectar las situaciones de interés sobre un dominio concreto en tiempo real. Estos lenguajes utilizan patrones para describir las situaciones en las que se quiere filtrar la información, para realizar las actuaciones correctas. Los lenguajes EPL procesan y analizan grandes cantidades de datos (eventos) en tiempo real, por lo que cualquier fallo de programación podría afectar gravemente en la toma de decisiones. Viendo la relevancia que tiene el procesado de esta información, resulta fundamental analizar y probar programas escritos en estos lenguajes de programación, detectando los posibles fallos más comunes que los programadores pueden cometer. Para poder hacer cualquier tipo de prueba en sistemas que utilicen EPL, se necesita una gran cantidad de eventos con estructuras y valores específicos. Conseguir estos eventos de forma manual puede ser una tarea muy costosa y propensa a errores. Esta tesis aborda el problema de la generación automática de los casos de prueba proponiendo y empleando un método para la generación automática de eventos, el cual se comparará con los sistemas de generación de eventos disponibles. El método incluye una propuesta de especificación para la definición de tipos de eventos y un generador de eventos basado en la especificación anterior. Con este método se obtienen casos de prueba, que van a permitir evaluar situaciones críticas de las aplicaciones, ya que los eventos pueden ser generados con valores concretos que ayuden a simularlas. Adicionalmente, para validar el método propuesto, se escoge el lenguaje de consulta EPL de la empresa EsperTech y se aplica la prueba de mutaciones en programas que lanzan este tipo de consultas. Entre los motivos de la elección del lenguaje EPL de EsperTech se destaca que es de código abierto y que su sintaxis se aproxima bastante al lenguaje SQL (ampliamente conocido). Por otro lado, se escoge la prueba de mutaciones ya que esta ha sido aplicada con éxito a una gran variedad de lenguajes de programación y mostramos que es una técnica de prueba adecuada para verificar y validar estos programas. Se generan multitud de variantes del programa a probar que representan los fallos máscomunes de programación. Para poder aplicar la prueba de mutaciones se requiere de una gran cantidad de casos de prueba. Dado que la prueba de mutaciones no había sido aplicada anteriormente a este lenguaje, esta tesis afronta el proceso completo de aplicación de esta técnica a un lenguaje: definición de operadores de mutación, desarrollo de una herramienta de generación y ejecución automática de los mutantes y evaluación de los operadores definidos (comprobando su frecuencia de aplicación en casos de estudio).Internet of Things (IoT) has increasingly become popular in diferent areas. One of the main drawbacks of the IoT systems is the amount of information they have to handle. This information arrives as events that need to be processed in real-time in order to make correct decisions. As a consequence, new ways (tools, devices, mechanisms...) of obtaining, processing and transmitting information have to be put into action. It is worth mentioning the "Event Processing Languages" (EPL), which were created to detect, in real-time, interesting situations in a particular domain. These languages use patterns to filter the information (good knowledge of the EPL languages is needed). A huge amount of data is processed and analysed by EPLs, so any programmer error could seriously affect the outcome because of a poor decision-making system. Given that processing the data is crucial, testing and analysing programs that run any EPL language is required. The most common mistakes that programmers could make have to be detected. A large number of events with specific values and structures are needed to apply any kind of testing in programs which use EPL. As this is a very hard task and very prone to error if done by hand, this dissertation addresses the automated generation of test cases; a method to automatically generate events has been proposed and used. Moreover, it will be compared to existing event generation systems. This method includes a general definition of what is an event type and its representation is proposed. Additionally, an event generator is developed based on this definition. Test cases to evaluate critical situations the EPL programs may suffer are obtained following this method. The proposed method allows the generation of events with specific values and structures, which will simulate these critical situations. In order to validate the proposed method, the EPL of EsperTech company is selected to apply mutation testing to programs which run the EsperTech EPL queries. EsperTech EPL language has been selected because it is open source and its syntax is very similar to SQL language (very popular programming language). Mutation testing is the chosen testing technique because it has been successfully applied to various programming languages. Mutation testing is a suitable testing technique to verify and validate these programs. Several variations of the original program are generated, which include "typical" programming errors. In order to apply mutation testing, a huge amount of data is required. Due to mutation testing has not been applied to EsperTech EPL language before, this dissertation faces the complete process of applying the mutation testing technique toa programming language: mutation operators definition, development of a tool which automatically generates and executes mutants and evaluation of the defined operators (checking their application frequency in studies cases).Documento PDF de 328 páginas https://dialnet.unirioja.es/servlet/tesis?codigo=71530 https://www.researchgate.net/profile/L_Gutierrez-Madronal/publication/318394697_Generacion_Automatica_de_Casos_en_Procesamiento_de_Eventos_con_EPL_Automatic_Generation_of_Cases_in_Event_Processing_using_EPL/links/59675223458515e9af9e9e55/Generacion-Automatica-de-Casos-en-Procesamiento-de-Eventos-con-EPL-Automatic-Generation-of-Cases-in-Event-Processing-using-EPL.pd