Automated inference of likely metamorphic relations for model transformations

Model transformations play a cornerstone role in Model-Driven Engineering (MDE) as they provide the essential mechanisms for manipulating and transforming models. Checking whether the output of a model transformation is correct is a manual and errorprone task, referred to as the oracle problem. Metamorphic testing alleviates the oracle problem by exploiting the relations among di erent inputs and outputs of the program under test, so-called metamorphic relations (MRs). One of the main challenges in metamorphic testing is the automated inference of likely MRs. This paper proposes an approach to automatically infer likely MRs for ATL model transformations, where the tester does not need to have any knowledge of the transformation. The inferred MRs aim at detecting faults in model transformations in three application scenarios, namely regression testing, incremental transformations and migrations among transformation languages. In the experiments performed, the inferred likely MRs have proved to be quite accurate, with a precision of 96.4% from a total of 4101 true positives out of 4254 MRs inferred. Furthermore, they have been useful for identifying mutants in regression testing scenarios, with a mutation score of 93.3%. Finally, our approach can be used in conjunction with current approaches for the automatic generation of test cases.Comisión Interministerial de Ciencia y Tecnología TIN2015-70560-RJunta de Andalucía P12-TIC-186

Introduction to the Special Issue on Test Oracles

Software testing is the de facto technique for correctness verification. Although there are different motivations and approaches to testing, fundamentally, it involves the running of test cases against the software under test. An important aspect is the ability to tell whether the execution results agree with the expected outcomes. The mechanism to determine the correctness of the results is called a test oracle. Unfortunately, such a mechanism does not always exist or may exist but cannot be feasibly used, which is known as the oracle problem. This is a major challenge to software testers, severely curtailing the application of testing approaches and methods to practical situations. In spite of the vast amount of interest in software testing, studies on test oracles drew much less attention than other aspects of software testing until quite recently. The increasing number of research activities surrounding the oracle problem reflects a growing interest in the area. This special issue documents some of the new contributions in software test oracles. We received a total of 19 submissions. Each submission has been reviewed by at least three reviewers. Six papers have been accepted for publication. In general, test oracle issues may involve identifying the expected program properties, test case generation, test sequencing and concurrency, test output verification, and fault localization. To deal with the identification of expected program properties, paper 1 titled “Automated Inference of Likely Metamorphic Relations for Model Transformations” by Troya et al. proposes a method of automatically inferring likely metamorphic relations, in the context of metamorphic testing, for the verification of model transformations. To handle test case generation and conduct run-time assertion checking, paper 2 titled “Contract-Based Testing for PHP with Praspel” by Dadeau et al. makes use of contracts, which enable the automation of these processes. To tackle the test sequencing question, paper 3 titled “Temporal Algebraic Query of Test Sequences” by Prasetya studies the question of test sequences using formalism based on algebraic relations and linear temporal logic. Paper 4 titled “Stateless Techniques for Generating Global and Local Test Oracles for Message-Passing Concurrent Programs” by Carver and Lei considers the state explosion problem of test sequences and proposes a stateless method to generate test oracles for concurrent programs. To address the test output verification problem, paper 5 titled “VISOR: A Fast Image Processing Pipeline with Scaling and Translation Invariance for Test Oracle Automation of Visual Output Systems” by Kıraç et al. focuses on the verification of visual output and introduces an automated test oracle that avoids false positives in image comparison caused by scaling and translation. To investigate software fault localization in the face of the oracle problem, paper 6 titled “Exploring the Usefulness of Unlabelled Test Cases in Software Fault Localization” by Zhang et al. develops an approach based on a core idea of test classification. We would like to thank the authors for their contributions, and the reviewers for their invaluable comments

Introduction to the special issue on test oracles

Software testing is the de facto technique for correctness verification. Although there are different motivations and approaches to testing, fundamentally, it involves the running of test cases against the software under test. An important aspect is the ability to tell whether the execution results agree with the expected outcomes. The mechanism to determine the correctness of the results is called a test oracle. Unfortunately, such a mechanism does not always exist or may exist but cannot be feasibly used, which is known as the oracle problem. This is a major challenge to software testers, severely curtailing the application of testing approaches and methods to practical situations. In spite of the vast amount of interest in software testing, studies on test oracles drew much less attention than other aspects of software testing until quite recently. The increasing number of research activities surrounding the oracle problem reflects a growing interest in the area. This special issue documents some of the new contributions in software test oracles. We received a total of 19 submissions. Each submission has been reviewed by at least three reviewers. Six papers have been accepted for publication. In general, test oracle issues may involve identifying the expected program properties, test case generation, test sequencing and concurrency, test output verification, and fault localization. To deal with the identification of expected program properties, paper 1 titled “Automated Inference of Likely Metamorphic Relations for Model Transformations” by Troya et al. proposes a method of automatically inferring likely metamorphic relations, in the context of metamorphic testing, for the verification of model transformations. To handle test case generation and conduct run-time assertion checking, paper 2 titled “Contract-Based Testing for PHP with Praspel” by Dadeau et al. makes use of contracts, which enable the automation of these processes. To tackle the test sequencing question, paper 3 titled “Temporal Algebraic Query of Test Sequences” by Prasetya studies the question of test sequences using formalism based on algebraic relations and linear temporal logic. Paper 4 titled “Stateless Techniques for Generating Global and Local Test Oracles for Message-Passing Concurrent Programs” by Carver and Lei considers the state explosion problem of test sequences and proposes a stateless method to generate test oracles for concurrent programs. To address the test output verification problem, paper 5 titled “VISOR: A Fast Image Processing Pipeline with Scaling and Translation Invariance for Test Oracle Automation of Visual Output Systems” by Kıraç et al. focuses on the verification of visual output and introduces an automated test oracle that avoids false positives in image comparison caused by scaling and translation. To investigate software fault localization in the face of the oracle problem, paper 6 titled “Exploring the Usefulness of Unlabelled Test Cases in Software Fault Localization” by Zhang et al. develops an approach based on a core idea of test classification. We would like to thank the authors for their contributions, and the reviewers for their invaluable comments

Introduction to the special issue on test oracles

Software testing is the de facto technique for correctness verification. Although there are different motivations and approaches to testing, fundamentally, it involves the running of test cases against the software under test. An important aspect is the ability to tell whether the execution results agree with the expected outcomes. The mechanism to determine the correctness of the results is called a test oracle. Unfortunately, such a mechanism does not always exist or may exist but cannot be feasibly used, which is known as the oracle problem. This is a major challenge to software testers, severely curtailing the application of testing approaches and methods to practical situations. In spite of the vast amount of interest in software testing, studies on test oracles drew much less attention than other aspects of software testing until quite recently. The increasing number of research activities surrounding the oracle problem reflects a growing interest in the area. This special issue documents some of the new contributions in software test oracles. We received a total of 19 submissions. Each submission has been reviewed by at least three reviewers. Six papers have been accepted for publication. In general, test oracle issues may involve identifying the expected program properties, test case generation, test sequencing and concurrency, test output verification, and fault localization. To deal with the identification of expected program properties, paper 1 titled “Automated Inference of Likely Metamorphic Relations for Model Transformations” by Troya et al. proposes a method of automatically inferring likely metamorphic relations, in the context of metamorphic testing, for the verification of model transformations. To handle test case generation and conduct run-time assertion checking, paper 2 titled “Contract-Based Testing for PHP with Praspel” by Dadeau et al. makes use of contracts, which enable the automation of these processes. To tackle the test sequencing question, paper 3 titled “Temporal Algebraic Query of Test Sequences” by Prasetya studies the question of test sequences using formalism based on algebraic relations and linear temporal logic. Paper 4 titled “Stateless Techniques for Generating Global and Local Test Oracles for Message-Passing Concurrent Programs” by Carver and Lei considers the state explosion problem of test sequences and proposes a stateless method to generate test oracles for concurrent programs. To address the test output verification problem, paper 5 titled “VISOR: A Fast Image Processing Pipeline with Scaling and Translation Invariance for Test Oracle Automation of Visual Output Systems” by Kıraç et al. focuses on the verification of visual output and introduces an automated test oracle that avoids false positives in image comparison caused by scaling and translation. To investigate software fault localization in the face of the oracle problem, paper 6 titled “Exploring the Usefulness of Unlabelled Test Cases in Software Fault Localization” by Zhang et al. develops an approach based on a core idea of test classification. We would like to thank the authors for their contributions, and the reviewers for their invaluable comments