Aplicaciones de la teoría de la información y la inteligencia artificial al testing de software

Tesis inédita de la Universidad Complutense de Madrid, Facultad de Informática, Departamento de Ingeniería de Sistemas lnformáticos y de Computación, leída el 4-05-2022Software Testing is a critical field for the software industry, as it has the main tools used to ensure the reliability of the produced software. Currently, mor then 50% of the time and resources for creating a software product are diverted to testing tasks, from unit testing to system testing. Moreover, there is a huge interest into automatising this field, as software gets bigger and the amount of required testing increases. however, software Testing is not only an industry oriented field; it is also a really interesting field with a noble goal (improving the reliability of software systems) that at the same tieme is full of problems to solve....Es Testing Software es un campo crítico para la industria del software, ya que éste contienen las principales herramientas que se usan para asegurar la fiabilidad del software producido. Hoy en día, más del 50% del tiempo y recursos necesarios para crear un producto software son dirigidos a tareas de testing, desde el testing unitario al testing a nivel de sistema. Más aún, hay un gran interés en automatizar este campo, ya que el software cada vez es más grande y la cantidad de testing requerido crece. Sin embargo, el Testing de Software no es solo un campo orientado a la industria; también es un campo muy interesante con un objetivo noble (mejorar la fiabilidad de los sistemas software) que al mismo tiempo está lleno de problemas por resolver...Fac. de InformáticaTRUEunpu

Extended Finite State Machine based test generation for an OpenFlow switch

Implementations of an OpenFlow (OF) switch, a crucial Software Defined Networking (SDN) component, are prone to errors caused by developer mistakes or/and ambiguous requirements stated in the OF documents. The paper is devoted to test derivation for related OF switch implementations. A model based test generation strategy is proposed. It relies on an Extended Finite State Machine (EFSM) specification that describes the functional behaviour of the switch-to-controller communication while potential faults/misconfigurations are expressed via corresponding mutation operators. We propose a method for deriving a test suite that contains distinguishing sequences for the specification EFSM and corresponding mutants. The proposed approach is implemented as a testbed to automatically derive and execute the test suites against different versions of an OF implementation. Preliminary experimental evaluation has shown the effectiveness of the proposed approach. Further on, the derived test suites have been able to detect a number of functional inconsistencies such as erroneous responses to the Flow Mod adding rules with specific 'action' values in an available Open vSwitch implementatio

Software engineering: Testing real-time embedded systems using timed automata based approaches

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.Real-time Embedded Systems (RTESs) have an increasing role in controlling society infrastructures that we use on a day-to-day basis. RTES behaviour is not based solely on the interactions it might have with its surrounding environment, but also on the timing requirements it induces. As a result, ensuring that an RTES behaves correctly is non-trivial, especially after adding time as a new dimension to the complexity of the testing process. This research addresses the problem of testing RTESs from Timed Automata (TA) specification by the following. First, a new Priority-based Approach (PA) for testing RTES modelled formally as UPPAAL timed automata (TA variant) is introduced. Test cases generated according to a proposed timed adequacy criterion (clock region coverage) are divided into three sets of priorities, namely boundary, out-boundary and in-boundary. The selection of which set is most appropriate for a System Under Test (SUT) can be decided by the tester according to the system type, time specified for the testing process and its budget. Second, PA is validated in comparison with four well-known timed testing approaches based on TA using Specification Mutation Analysis (SMA). To enable the validation, a set of timed and functional mutation operators based on TA is introduced. Three case studies are used to run SMA. The effectiveness of timed testing approaches are determined and contrasted according to the mutation score which shows that our PA achieves high mutation adequacy score compared with others. Third, to enhance the applicability of PA, a new testing tool (GeTeX) that deploys PA is introduced. In its current version, GeTeX supports Control Area Network (CAN) applications. GeTeX is validated by developing a prototype for that purpose. Using GeTeX, PA is also empirically validated in comparison with some TA testing approaches using a complete industrial-strength test bed. The assessment is based on fault coverage, structural coverage, the length of generated test cases and a proposed assessment factor. The assessment is based on fault coverage, structural coverage, the length of generated test cases and a proposed assessment factor. The assessment results confirmed the superiority of PA over the other test approaches. The overall assessment factor showed that structural and fault coverage scores of PA with respect to the length of its tests were better than the others proving the applicability of PA. Finally, an Analytical Hierarchy Process (AHP) decision-making framework for our PA is developed. The framework can provide testers with a systematic approach by which they can prioritise the available PA test sets that best fulfils their testing requirements. The AHP framework developed is based on the data collected heuristically from the test bed and data collected by interviewing testing experts. The framework is then validated using two testing scenarios. The decision outcomes of the AHP framework were significantly correlated to those of testing experts which demonstrated the soundness and validity of the framework.This study is funded by Damascus University, Syri

MuAlloy : an automated mutation system for alloy

textMutation is a powerful technique that researchers have studied for several decades in the context of imperative code. For example, mutation testing is commonly considered a '"gold standard"' for test suite quality. Mutation in the context of declarative languages is a less studied problem. This thesis introduces a foundation for mutation-driven analyses for Alloy, a first-order, declarative language based on relations. Specifically, we introduce a family of mutation operators for Alloy models and define algorithms for applying the operators on different parts of the models. We embody these operators and algorithms in our prototype tool MuAlloy that provides a GUI-based front-end for customizing the application of mutation operators. To demonstrate the potential of our approach, we illustrate the use of MuAlloy in two application scenarios: (1) mutation testing for Alloy (in the spirit of traditional mutation testing for imperative languages); and (2) program repair for Alloy using mutation.Electrical and Computer Engineerin

Higher Order Mutation Testing

Mutation testing is a fault-based software testing technique that has been studied widely for over three decades. To date, work in this field has focused largely on first order mutants because it is believed that higher order mutation testing is too computationally expensive to be practical. This thesis argues that some higher order mutants are potentially better able to simulate real world faults and to reveal insights into programming bugs than the restricted class of first order mutants. This thesis proposes a higher order mutation testing paradigm which combines valuable higher order mutants and non-trivial first order mutants together for mutation testing. To overcome the exponential increase in the number of higher order mutants a search process that seeks fit mutants (both first and higher order) from the space of all possible mutants is proposed. A fault-based higher order mutant classification scheme is introduced. Based on different types of fault interactions, this approach classifies higher order mutants into four categories: expected, worsening, fault masking and fault shifting. A search-based approach is then proposed for locating subsuming and strongly subsuming higher order mutants. These mutants are a subset of fault mask and fault shift classes of higher order mutants that are more difficult to kill than their constituent first order mutants. Finally, a hybrid test data generation approach is introduced, which combines the dynamic symbolic execution and search based software testing approaches to generate strongly adequate test data to kill first and higher order mutants

Evaluating and comparing fault-based testing strategies for general Boolean specifications: A series of experiments

A great amount of fault-based testing strategies have been proposed to generate test cases for detecting certain types of faults in Boolean specifications. However, most of the previous studies on these strategies were focused on the Boolean expressions in the disjunctive normal form (DNF), even the irredundant DNF (IDNF)-little work has been conducted to comprehensively investigate their performance on general Boolean specifications. In this study, we conducted a series of experiments to evaluate and compare 18 fault-based testing strategies using over 4000 randomly generated fault-seeded Boolean expressions. In the experiments, a testing strategy is regarded as effective and efficient if it can detect most of the seeded faults using a small number of test cases. Our experimental results show that if a testing strategy is highly effective and efficient when testing the Boolean expressions in the IDNF, it also shows high effectiveness and efficiency on general Boolean expressions. It is found that one family of fault-based testing strategies, namely MUMCUT, normally deliver the best performance among all the 18 strategies. Our study provides an in-depth understanding and insight of fault-based testing for general Boolean expressions

Towards an Automation of the Mutation Analysis Dedicated to Model Transformation

International audienceA major benefit of Model Driven Engineering (MDE) relies on the automatic generation of artefacts from high-level models through intermediary levels using model transformations. In such a process, the input must be well-designed and the model transformations should be trustworthy. Due to the specificities of models and transformations, classical software test techniques have to be adapted. Among these techniques, mutation analysis has been ported and a set of mutation operators has been defined. However, mutation analysis currently requires a considerable manual work and suffers from the test data set improvement activity. This activity is seen by testers as a difficult and time-consuming job, and reduces the benefits of the mutation analysis. This paper addresses the test data set improvement activity. Model transformation traceability in conjunction with a model of mutation operators, and a dedicated algorithm allow to automatically or semi-automatically produce test models that detect new faults. The proposed approach is validated and illustrated in a case study written in Kermeta

Towards Systematic Mutations for and with ATL Model Transformations

Model transformation is a key technique to automate software engineering tasks, such as generating implementations of software systems from higher-level models. To enable this automation, transformation engines are used to synthesize various types of software artifacts from models, where the rules according to which these artifacts are generated are implemented by means of dedicated model transformation languages. Hence, the quality of the generated software artifacts depends on the quality of the transformation rules applied to generate them. Thus, there is the need for approaches to certify their behavior for a selected set of test models. As mutation analysis has proven useful as a practical testing approach, we propose a set of mutation operators for the ATLAS Transformation Language (ATL) derived by a comprehensive language-centric synthesis approach. We describe the rationale behind each of the mutation operators and propose an automated process to generate mutants for ATL transformations based on a combination of generic mutation operators and higher-order transformations. Finally, we describe a cost-effective solution for executing the obtained mutants.European Commission ICT Policy Support Programme 317859Ministerio de Ciencia e Innovación TIN2011-2379