3,147 research outputs found
Towards Automated Metamorphic Test Identification for Ocean System Models
Metamorphic testing seeks to verify software in the absence of test oracles.
Our application domain is ocean system modeling, where test oracles rarely
exist, but where symmetries of the simulated physical systems are known. The
input data set is large owing to the requirements of the application domain.
This paper presents work in progress for the automated generation of
metamorphic test scenarios using machine learning. We extended our previously
proposed method [1] to identify metamorphic relations with reduced
computational complexity. Initially, we represent metamorphic relations as
identity maps. We construct a cost function that minimizes for identifying a
metamorphic relation orthogonal to previously found metamorphic relations and
penalize for the identity map. A machine learning algorithm is used to identify
all possible metamorphic relations minimizing the defined cost function. We
propose applying dimensionality reduction techniques to identify attributes in
the input which have high variance among the identified metamorphic relations.
We apply mutation on these selected attributes to identify distinct metamorphic
relations with reduced computational complexity. For experimental evaluation,
we subject the two implementations of an ocean-modeling application to the
proposed method to present the use of metamorphic relations to test the two
implementations of this application.Comment: 5 Pages, 1 Figur
Mapping the Structure and Evolution of Software Testing Research Over the Past Three Decades
Background: The field of software testing is growing and rapidly-evolving.
Aims: Based on keywords assigned to publications, we seek to identify
predominant research topics and understand how they are connected and have
evolved.
Method: We apply co-word analysis to map the topology of testing research as
a network where author-assigned keywords are connected by edges indicating
co-occurrence in publications. Keywords are clustered based on edge density and
frequency of connection. We examine the most popular keywords, summarize
clusters into high-level research topics, examine how topics connect, and
examine how the field is changing.
Results: Testing research can be divided into 16 high-level topics and 18
subtopics. Creation guidance, automated test generation, evolution and
maintenance, and test oracles have particularly strong connections to other
topics, highlighting their multidisciplinary nature. Emerging keywords relate
to web and mobile apps, machine learning, energy consumption, automated program
repair and test generation, while emerging connections have formed between web
apps, test oracles, and machine learning with many topics. Random and
requirements-based testing show potential decline.
Conclusions: Our observations, advice, and map data offer a deeper
understanding of the field and inspiration regarding challenges and connections
to explore.Comment: To appear, Journal of Systems and Softwar
Metamorphic testing: testing the untestable
What if we could know that a program is buggy, even if we could not tell whether or not its observed output is correct? This is one of the key strengths of metamorphic testing, a technique where failures are not revealed by checking an individual concrete output, but by checking the relations among the inputs and outputs of multiple executions of the program under test. Two decades after its introduction, metamorphic testing has become a fully-fledged testing technique with successful applications in multiple domains, including online search engines, autonomous machinery, compilers, Web APIs, and deep learning programs, among others. This article serves as a hands-on entry point for newcomers to metamorphic testing, describing examples, possible applications, and current limitations, providing readers with the basics for the application of the technique in their own projects. IEE
Review of Software Testing Methods
With the increasing complexity of programs comes an increased focus on ensuring the quality of these programs. Essentially, it depends on improving the methods of testing the quality of these programs in the two phases of building these programs and after their operation. Therefore, we are developing software testing methods and methodologies. This paper aims to discuss software testing methods and their classifications according to their main properties and any software that suits each method. The majority of the literature on software testing methods and techniques is also included
Testing scientific software: techniques for automatic detection of metamorphic relations
2015 Spring.Includes bibliographical references.Scientific software plays an important role in critical decision making in fields such as the nuclear industry, medicine, and the military. Systematic testing of such software can help to ensure that it works as expected. Comprehensive, automated software testing requires an oracle to check whether the output produced by a test case matches the expected behavior of the program. But the challenges in creating suitable oracles limit the ability to perform automated testing of scientific software. For some programs, creating an oracle may be not possible since the correct output is not known a priori. Further, it may be impractical to implement an oracle for an arbitrary input due to the complexity of a program. The software testing community refers to such programs as non-testable. Many scientific programs fall into this category of non-testable programs, since they are either written to find answers that are previously unknown or they perform complex calculations. In this work, we developed techniques to automatically predict metamorphic relations by analyzing the program structure. These metamorphic relations can serve as automated partial test oracles in scientific software. Metamorphic testing is a method for automating the testing process for programs without test oracles. This technique operates by checking whether a program behaves according to a certain set of properties called metamorphic relations. A metamorphic relation is a relationship between multiple input and output pairs of the program. It specifies how the output should change following a specific change made to the input. A change in the output that differs from what is specified by the metamorphic relation indicates a fault in the program. Metamorphic testing can be effective in testing machine learning applications, bioinformatics programs, health-care simulations, partial differential equations and other programs. Unfortunately, finding appropriate metamorphic relations for use in metamorphic testing remains a labor intensive task that is generally performed by a domain expert or a programmer. In this work we applied novel machine learning based approaches to automatically derive metamorphic relations. We first evaluated the effectiveness of modeling the metamorphic relation prediction problem as a binary classification problem. We found that support vector machines are the most effective binary classifiers for predicting metamorphic relations. We also found that using walk-based graph kernels for feature extraction from graph-based program representations further improves the prediction accuracy. In addition, incorporating mathematical properties of operations in the graph kernel computation improves the prediction accuracy. Further, we found that control flow information of a function are more effective than data dependency information for predicting metamorphic relations. Finally we investigated the possibility of creating multi-label classifiers that can predict multiple metamorphic relations using a single classifier. Our empirical studies show that multi-label classifiers are not effective as binary classifiers for predicting metamorphic relations. Automated testing will make the testing process faster, reduce the testing cost and make it more reliable. Automated testing requires automated test oracles. Automatically discovering metamorphic relations is an important step towards automating oracle creation. Work presented here is the first attempt towards developing automated techniques for deriving metamorphic relations. Our work contributes toward automating the testing process of programs that face oracle problems
A Review of Software Reliability Testing Techniques
In the era of intelligent systems, the safety and reliability of software have received more attention. Software reliability testing is a significant method to ensure reliability, safety and quality of software. The intelligent software technology has not only offered new opportunities but also posed challenges to software reliability technology. The focus of this paper is to explore the software reliability testing technology under the impact of intelligent software technology. In this study, the basic theories of traditional software and intelligent software reliability testing were investigated via related previous works, and a general software reliability testing framework was established. Then, the technologies of software reliability testing were analyzed, including reliability modeling, test case generation, reliability evaluation, testing criteria and testing methods. Finally, the challenges and opportunities of software reliability testing technology were discussed at the end of this paper
The Integration of Machine Learning into Automated Test Generation: A Systematic Mapping Study
Context: Machine learning (ML) may enable effective automated test
generation.
Objective: We characterize emerging research, examining testing practices,
researcher goals, ML techniques applied, evaluation, and challenges.
Methods: We perform a systematic mapping on a sample of 102 publications.
Results: ML generates input for system, GUI, unit, performance, and
combinatorial testing or improves the performance of existing generation
methods. ML is also used to generate test verdicts, property-based, and
expected output oracles. Supervised learning - often based on neural networks -
and reinforcement learning - often based on Q-learning - are common, and some
publications also employ unsupervised or semi-supervised learning.
(Semi-/Un-)Supervised approaches are evaluated using both traditional testing
metrics and ML-related metrics (e.g., accuracy), while reinforcement learning
is often evaluated using testing metrics tied to the reward function.
Conclusion: Work-to-date shows great promise, but there are open challenges
regarding training data, retraining, scalability, evaluation complexity, ML
algorithms employed - and how they are applied - benchmarks, and replicability.
Our findings can serve as a roadmap and inspiration for researchers in this
field.Comment: Under submission to Software Testing, Verification, and Reliability
journal. (arXiv admin note: text overlap with arXiv:2107.00906 - This is an
earlier study that this study extends
Metamorphic Testing in Autonomous System Simulations
Metamorphic testing has proven to be effective for test case generation and
fault detection in many domains. It is a software testing strategy that uses
certain relations between input-output pairs of a program, referred to as
metamorphic relations. This approach is relevant in the autonomous systems
domain since it helps in cases where the outcome of a given test input may be
difficult to determine. In this paper therefore, we provide an overview of
metamorphic testing as well as an implementation in the autonomous systems
domain. We implement an obstacle detection and avoidance task in autonomous
drones utilising the GNC API alongside a simulation in Gazebo. Particularly, we
describe properties and best practices that are crucial for the development of
effective metamorphic relations. We also demonstrate two metamorphic relations
for metamorphic testing of single and more than one drones, respectively. Our
relations reveal several properties and some weak spots of both the
implementation and the avoidance algorithm in the light of metamorphic testing.
The results indicate that metamorphic testing has great potential in the
autonomous systems domain and should be considered for quality assurance in
this field.Comment: 8 pages, 5 figures, 48th Euromicro Conference Series on Software
Engineering and Advanced Applications (SEAA
- …