8,429 research outputs found
Towards Symbolic Model-Based Mutation Testing: Combining Reachability and Refinement Checking
Model-based mutation testing uses altered test models to derive test cases
that are able to reveal whether a modelled fault has been implemented. This
requires conformance checking between the original and the mutated model. This
paper presents an approach for symbolic conformance checking of action systems,
which are well-suited to specify reactive systems. We also consider
nondeterminism in our models. Hence, we do not check for equivalence, but for
refinement. We encode the transition relation as well as the conformance
relation as a constraint satisfaction problem and use a constraint solver in
our reachability and refinement checking algorithms. Explicit conformance
checking techniques often face state space explosion. First experimental
evaluations show that our approach has potential to outperform explicit
conformance checkers.Comment: In Proceedings MBT 2012, arXiv:1202.582
Your Proof Fails? Testing Helps to Find the Reason
Applying deductive verification to formally prove that a program respects its
formal specification is a very complex and time-consuming task due in
particular to the lack of feedback in case of proof failures. Along with a
non-compliance between the code and its specification (due to an error in at
least one of them), possible reasons of a proof failure include a missing or
too weak specification for a called function or a loop, and lack of time or
simply incapacity of the prover to finish a particular proof. This work
proposes a new methodology where test generation helps to identify the reason
of a proof failure and to exhibit a counter-example clearly illustrating the
issue. We describe how to transform an annotated C program into C code suitable
for testing and illustrate the benefits of the method on comprehensive
examples. The method has been implemented in STADY, a plugin of the software
analysis platform FRAMA-C. Initial experiments show that detecting
non-compliances and contract weaknesses allows to precisely diagnose most proof
failures.Comment: 11 pages, 10 figure
Automated metamorphic testing on the analyses of feature models
Copyright © 2010 Elsevier B.V. All rights reserved.Context: A feature model (FM) represents the valid combinations of features in a domain. The automated extraction of information from FMs is a complex task that involves numerous analysis operations, techniques and tools. Current testing methods in this context are manual and rely on the ability of the tester to decide whether the output of an analysis is correct. However, this is acknowledged to be time-consuming, error-prone and in most cases infeasible due to the combinatorial complexity of the analyses, this is known as the oracle problem.Objective: In this paper, we propose using metamorphic testing to automate the generation of test data for feature model analysis tools overcoming the oracle problem. An automated test data generator is presented and evaluated to show the feasibility of our approach.Method: We present a set of relations (so-called metamorphic relations) between input FMs and the set of products they represent. Based on these relations and given a FM and its known set of products, a set of neighbouring FMs together with their corresponding set of products are automatically generated and used for testing multiple analyses. Complex FMs representing millions of products can be efficiently created by applying this process iteratively.Results: Our evaluation results using mutation testing and real faults reveal that most faults can be automatically detected within a few seconds. Two defects were found in FaMa and another two in SPLOT, two real tools for the automated analysis of feature models. Also, we show how our generator outperforms a related manual suite for the automated analysis of feature models and how this suite can be used to guide the automated generation of test cases obtaining important gains in efficiency.Conclusion: Our results show that the application of metamorphic testing in the domain of automated analysis of feature models is efficient and effective in detecting most faults in a few seconds without the need for a human oracle.This work has been partially supported by the European Commission(FEDER)and Spanish Government under CICYT project SETI(TIN2009-07366)and the Andalusian Government project ISABEL(TIC-2533)
Recommended from our members
Using formal methods to support testing
Formal methods and testing are two important approaches that assist in the development of high quality software. While traditionally these approaches have been seen as rivals, in recent
years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing
Artificial table testing dynamically adaptive systems
Dynamically Adaptive Systems (DAS) are systems that modify their behavior and
structure in response to changes in their surrounding environment. Critical
mission systems increasingly incorporate adaptation and response to the
environment; examples include disaster relief and space exploration systems.
These systems can be decomposed in two parts: the adaptation policy that
specifies how the system must react according to the environmental changes and
the set of possible variants to reconfigure the system. A major challenge for
testing these systems is the combinatorial explosions of variants and
envi-ronment conditions to which the system must react. In this paper we focus
on testing the adaption policy and propose a strategy for the selection of
envi-ronmental variations that can reveal faults in the policy. Artificial
Shaking Table Testing (ASTT) is a strategy inspired by shaking table testing
(STT), a technique widely used in civil engineering to evaluate building's
structural re-sistance to seismic events. ASTT makes use of artificial
earthquakes that simu-late violent changes in the environmental conditions and
stresses the system adaptation capability. We model the generation of
artificial earthquakes as a search problem in which the goal is to optimize
different types of envi-ronmental variations
Improved test quality using robust unique input/output circuit sequences (UIOCs)
In finite state machine (FSM) based testing, the problem of fault masking in the unique input/ output (UIO) sequence may degrade the test performance of the UIO based methods. This paper investigates this problem and proposes the use of a new type of unique input/output circuit (UIOC) sequence for state verification, which may help to overcome the drawbacks that exist in the UIO based techniques. When constructing a UIOC, overlap and internal state observation schema are used to increase the robustness of a test sequence. Test quality is compared by using the forward UIO method (F-method), the backward UIO method (B-method) and the UIOC method (C-method)
separately. Robustness of the UIOCs constructed by the algorithm given in this paper is also compared with those constructed by the algorithm given previously. Experimental results suggest that the C-method outperforms the F- and the B-methods and the UIOCs constructed by the Algorithm given in this paper, are more robust than those constructed by other proposed algorithms
Mutation testing from probabilistic finite state machines
Mutation testing traditionally involves mutating a program in order to produce a set of mutants and using these mutants in order to either estimate the effectiveness of a test suite or to drive test generation. Recently, however, this approach has been applied to specifications such as those written as finite state machines. This paper extends mutation testing to finite state machine models in which transitions have associated probabilities. The paper describes several ways of mutating a probabilistic finite state machine (PFSM) and shows how test sequences that distinguish between a PFSM and its mutants can be generated. Testing then involves applying each test sequence multiple times, observing the resultant output sequences and using results from statistical sampling theory in order to compare the observed frequency of each output sequence with that expected
Recommended from our members
Comparing test sets and criteria in the presence of test hypotheses and fault domains
A number of authors have considered the problem of comparing test sets and criteria. Ideally
test sets are compared using a preorder with the property that test set T1 is at least as strong
as T2 if whenever T2 determines that an implementation p is faulty, T1 will also determine that
p is faulty. This notion can be extended to test criteria. However, it has been noted that very
few test sets and criteria are comparable under such an ordering; instead orderings are based
on weaker properties such as subsumes. This paper explores an alternative approach, in which
comparisons are made in the presence of a test hypothesis or fault domain. This approach allows
strong statements about fault detecting ability to be made and yet for a number of test sets and
criteria to be comparable. It may also drive incremental test generation
- …