Metamorphic testing, an innovative software testing technique, generates test cases based on domain specific properties. Model checking is a technique that verifies software designs against system properties. Motivated by the fact that both techniques are based on some properties of software under development, we investigate how to integrate metamorphic testing and model checking. This paper will introduce and discuss some potential topics on this new research direction

Chen, T

Huimin, L

Liu, H

Wang, D

Liu, Huai

Lin, H

Chen, TY

Victoria University Institutional Repository (VUIR)

On the integration of metamorphic testing and model checking

Name not available

RMIT Research Repository

?Thank
?
??????
???????
??????
?
?
Citatio
See th
Version
Copyri
Link to
??
?
you for do
??????????
??????????
??????????
n: 
is record i
:
ght Statem
 Published
?
wnloading
??????????
?????????????
??????????????????????????????????????????
n the RMI
ent: ©  
 Version:
 this docum
????????????
??????????
T Researc
ent from 
??????????
?
h Reposit
the RMIT R
??????????
ory at:  
esearch R
??????????
epository
??????????
????
??
PLEASE DO NOT REMOVE THIS PAGE
Liu, H, Wang, D, Huimin, L and Chen, T 2009, 'On the integration of metamorphic testing
and model checking', in Hans Weghorn and Pedro T. Isaias (ed.) Proceedings of the
International Association for Development of the Information Society (IADIS) International
Conference on Applied Computing (IADIS AC 2009), Rome, Italy, 19-21 November 2009,
pp. 299-302
https://researchbank.rmit.edu.au/view/rmit:20951
Published Version
2009 IADIS
http://www.iadisportal.org/digital-library/on-the-integration-of-metamorphic-...
ON THE INTEGRATION OF METAMORPHIC TESTING 
AND MODEL CHECKING 
Huai Liu1∗, Daoming Wang2, Huimin Lin2, Tsong Yueh Chen1 
1Centre for Software Analysis and Testing, Swinburne University of Technology, Australia 
2Institute of Software, Chinese Academy of Sciences, China 
ABSTRACT 
Metamorphic testing, an innovative software testing technique, generates test cases based on domain specific properties. 
Model checking is a technique that verifies software designs against system properties. Motivated by the fact that both 
techniques are based on some properties of software under development, we investigate how to integrate metamorphic 
testing and model checking. This paper will introduce and discuss some potential topics on this new research direction.  
1. INTRODUCTION 
In many software testing techniques, a set of program inputs are selected as test cases according to some 
criteria. The software under test is then run against these test cases. Testers verify the program outputs 
(namely, the testing results) based on a mechanism called oracle. However, in practice, there may exist a 
problem called oracle problem where (i) “there does not exist an oracle” or (ii) “it is theoretically possible, 
but practically too difficult to determine the correct output” (Chen et al., 2003). Recently a new software 
testing technique, namely metamorphic testing (Chen et al., 1998; Gotlieb and Botella, 2003), has been 
proposed to alleviate the oracle problem. In metamorphic testing, some properties are first identified from the 
software under test. Then, metamorphic testing generates test cases based on these properties. In addition, the 
testing results are verified against these properties, not the oracle. Metamorphic testing technique has been 
widely used in various areas of software engineering (Chen et al., 2003, 2009a, 2009b; Murphy et al., 2008).  
Model checking (Clarke et al., 1999) is an automatic technique to check if a finite state system satisfies its 
specifications which are expressed as temporal formulas. In model checking, a software design is checked 
against a set of properties specified by such formulas. When the design violates a property, a counterexample 
will be generated to point out the source of the error. This feature makes model checking practically useful.  
Although metamorphic testing and model checking deal with different problems in different areas, they 
are both based on some properties of the system under test. In this paper, we are motivated to look at whether 
these two innovative techniques can be integrated, aiming at improving the effectiveness of software 
verification and testing. This paper is organized as follows. In Sections 2 and 3, we briefly introduce 
metamorphic testing and model checking, respectively. In Section 4, we propose some potential research 
topics about the integration of metamorphic testing and model checking. In Section 5, we conclude this paper. 
2. METAMORPHIC TESTING 
In metamorphic testing, the program developers, testers, and/or end-users first identify some properties of the 
software under test. Some relations called metamorphic relations can then be constructed based on these 
properties. Source test cases are generated by some traditional test case selection techniques. Follow-up test 
cases are then constructed based on source test cases and metamorphic relations. The software under test is 
executed with all test cases as the program inputs. The outputs of the source and follow-up test cases are 
compared according to the metamorphic relations. If testing results do not satisfy a metamorphic relation, a 
failure is said to be detected. The basic process of metamorphic testing is shown in Figure 1.  
                                                 
∗
 Corresponding author. 
IADIS International Conference Applied Computing 2009
299
Since the proposal of metamorphic testing, it has been successfully applied to alleviate the oracle problem 
in testing various software products. For example, Murphy et al. (2008) have proposed some metamorphic 
relations that can be used for testing machine learning applications. Chen et al. (2009) detected a real-life bug 
in a bioinformatics program using metamorphic testing technique. In addition, metamorphic testing is also 
applicable to improve other software testing techniques (Chen et al., 2003; Chen et al., 2009b). 
 
                        
   
3. MODEL CHECKING 
Model checking is an automatic technique for verifying finite state concurrent systems. Figure 2 shows the 
basic process of model checking. Model checking mainly consists of three tasks. The first task is to convert a 
design into a formalism accepted by a model checker. Kripke structure (Clarke et al., 1999) is normally used 
to model the system. A Kripke structure is a type of state transition graph that consists of a set of states, a set 
of transitions between states, and a labeling function assigning atomic propositions to states.  
The second task is to state the properties that the system must satisfy. These properties are usually 
expressed using formulas of temporal logics (Emerson, 1990). Several temporal logics have been proposed to 
be used in model checking, such as linear temporal logic and computation tree logic. These logics provide 
different operators to express temporal properties, and vary in their expressive power.  
The last task is to verify the design against the properties. If the constructed Kripke structure satisfies the 
specifications, the model checker terminates with a “yes” answer, which implies that the design is guaranteed 
free of error with respect to the property. Otherwise, the model checker terminates with a “no” answer and 
provides a counterexample, a sequence of states that demonstrates where the design violates the expected 
property. This will help developers find the error in the system design. Many model checking algorithms 
(Gerth et al., 1995) have been proposed. When the system being verified has many concurrent parts or data 
variables, model checking suffers from the “state explosion problem”. Although techniques such as symbolic 
representation, abstraction, and partial order reduction have been developed to alleviate this problem (Clarke 
et al., 1999), it is still one of the most challenging problems in model checking research. Another technique 
called bounded model checking was introduced by Biere et al. (1999). Bounded model checking is 
particularly suitable for certain kinds of infinite state systems such as software systems.  
4. INTEGRATING METAMORPHIC TESTING AND MODEL 
CHECKING 
Some researchers have investigated how to combine model checking with some traditional software testing 
techniques (Peled et al., 1999; Groce et al., 2002). In this section, we attempt to highlight several potential 
research topics on how to integrate metamorphic testing and model checking, as listed in the following. Such 
integration is inspired by the fact that both metamorphic testing and model checking are conducted based on 
some properties of the system under test, as shown in Sections 2 and 3.  
• Selection of system properties.  
Figure 1. The process of metamorphic testing Figure 2. The process of model checking 
ISBN: 978-972-8924-97-3 © 2009 IADIS
300
In model checking, the system design is checked against some properties, which serve as the system 
specifications. In metamorphic testing, many properties are also derived from the system specifications, but 
these properties are identified specifically to examine whether the software under test is correct from certain 
perspectives, especially from the end-user’s perspective. Metamorphic testing involves a lot of user domain 
knowledge, so the properties identified in metamorphic testing are very useful to ensure that the software 
under test can be correctly executed by end-users. Research (Chen et al., 2004; Mayer and Guderlei, 2006) 
has been conducted to develop some approaches to selecting effective metamorphic relations. It is 
worthwhile to see whether such approaches could be successfully applied in model checking.  
• Enhancement of the effectiveness of detecting couterexamples. 
Counterexample generation is an important and useful feature of model checking, as counterexamples 
provide precise information for error fixing. Some researchers even used counterexamples to generate test 
cases (Beyer et al., 2004). Basically, model checker searches the state space of the Kripke structure to check 
if the structure satisfies its specifications. In case that it does not satisfy the specifications, a counterexample, 
usually in the form of a sequence of states, will be output to pinpoint the source of the error. Previous work 
(Chen et al., 1998) has shown that metamorphic testing can provide information about the constraints of the 
inputs that effectively reveal failures. Such information is very useful for debugging. It is thus natural to 
investigate the relationship between these failure-revealing constraints in metamorphic testing and the 
counterexamples in model checking. As shown in Figure 3, we propose to apply metamorphic relations into 
the selection of state sequences for checking. If the metamorphic relations are effective in revealing failure, it 
is intuitive that the resultant sequences of states will have a high chance to become a counterexample.  
 
                
 
 
 
• Integration of model-based testing and metamorphic testing.  
Model-based testing (Pretschner, 2005; Utting and Legeard, 2007) is a software testing technique that 
generates test cases based on the model of the software under test. Various algorithms (Wu and Lin, 2006; 
Satpathy and Ramesh, 2007) have been proposed to derive test cases from a model. Model checking can also 
be used to guide the model-based test case generation. Since different test cases may have different failure-
detection effectiveness, how to select “good” test cases is an important research topic for model-based testing. 
Since there are some successful investigations (Chen et al., 2004; Mayer and Guderlei, 2006) into selection 
of metamorphic relationship that could generate follow-up test cases which are more likely to reveal failures, 
it is intuitive to investigate how these results could be applied in the selection of good test cases for model-
based testing. Figure 4 gives a diagram to integrate metamorphic testing and model-based testing, where 
some metamorphic relations are used to help select model-based test cases.  
• Combination of partial order reduction and properties in metamorphic testing.  
Partial order reduction (Peled, 1996) is a technique to reduce the size of the state space that is searched by 
a model checker. Briefly speaking, this technique will provide the model checker a reduced state graph 
instead of the original full state graph, and such a reduced state graph will present a sufficient number of 
behaviours such that the model checker still provides right answers (Clarke et al., 1999). Apparently, the 
reduced state graph is a subset of the full state graph. The properties identified by metamorphic testing are 
also a subset of the system specifications. It is interesting to study whether the technique of deriving a subset 
Figure 3. Enhancing the effectiveness of 
detecting counterexamples. 
Figure 4. Integration of model-based testing and 
metamorphic testing 
IADIS International Conference Applied Computing 2009
301
(that is, the system properties) of specifications in metamorphic testing can be used to improve the technique 
of constructing a subset (that is, the reduced state graph) of the whole state space in partial order reduction. 
5. CONCLUSION 
Metamorphic testing and model checking are two innovative techniques for software testing and verification. 
Since both techniques are conducted based on some properties of the system under test, it is intuitive to 
consider the integration of these two innovative techniques to improve the effectiveness of software testing 
and verification. A few potential topics were proposed in this paper, and we believe that they are all 
promising and worthwhile to be investigated. 
ACKNOWLEDGEMENT 
This project is supported by the Australian Research Council and a National Natural Science Foundation of 
China grant (No. 60721061). 
REFERENCES 
Beyer, D., Chlipala, A. J., Henzinger, T. A., Jhala, R., Majumdar, R., 2004. Generating tests from counterexamples. 
Proceedings of ICSE2004, pp. 326-335.  
Biere, A., Cimatti, A., Clarke, E. M., Zhu, Y., 1999. Symbolic model checking without BDDs. Proceedings of 
TACAS1999, pp. 193-207.  
Chen, T. Y., Cheung, S. C., Yiu, S. M., 1998. Metamorphic testing: a new approach for generating next test cases. Tech. 
Rep. HKUST-CS98-01, Department of Computer Science, Hong Kong University of Science and Technology.  
Chen, T. Y., Ho, J. W. K., Liu, H., Xie, X., 2009a. An innovative approach for testing bioinformatics programs using 
metamorphic testing. BMC Bioinformatics, Vol. 10, Article 24.  
Chen, T. Y., Huang, D. H., Tse, T. H., Zhou, Z. Q., 2004. Case studies on the selection of useful relations in metamorphic 
testing. Proceedings of JIISIC2004, pp. 569-583.  
Chen, T. Y., Tse, T. H., Zhou, Z. Q., 2003. Fault-based testing without the need of oracles. Inform. Softw. Tech., Vol. 45, 
No. 1, pp. 1-9.  
Chen, T. Y., Tse, T. H., Zhou, Z. Q., 2009b. Semi-Proving: An Integrated Method for Program Proving, Testing, and 
Debugging. Accepted to appear in IEEE T. Softw. Eng.  
Clarke, E. M., Grumberg, O., Peled, D. A., 1999. Model Checking. The MIT Press.  
Emerson, E. A., 1990. Temporal and modal logic. In: Handbook of Theoretical Computer Science. The MIT Press.  
Gerth, R., Peled, D., Vardi, M. Y., Wolper, P., 1995. Simple on-the-fly automatic verification of linear temporal logic. 
Proceedings of PSTV1995, pp. 3-18.  
Gotlieb, A., Botella, B., 2003. Automated metamorphic testing. Proceedings of COMPSAC 2003, pp. 34-40.  
Groce, A., Peled, D., Yannakakis, M., 2002. Adaptive model checking. Proceedings of TACAS 2002, pp. 357–370.  
Mayer, J., Guderlei, R., 2006. An empirical study on the selection of good metamorphic relations. Proceedings of 
COMPSAC2006, pp. 475-484.  
Murphy, C., Kaiser, G. E., Hu, L., Wu, L., 2008. Properties of machine learning applications for use in metamorphic 
testing. Proceedings of SEKE2008, pp. 867-872.  
Peled, D., 1996. Combining partial order reductions with on-the-fly model checking. Formal Methods in System Design, 
Vol. 8, No. 1, pp. 39-64.  
Peled, D., Vardi, M. Y., Yannakakis, M., 1999. Black box checking. Proceedings of FORTE/PSTV1999, pp. 225-240.  
Pretschner, A., 2005. Model-based testing. Proceedings of ICSE 2005, pp. 722-723.  
Satpathy, M., Ramesh, S., 2007. Test case generation from formal models through abstraction refinement and model 
checking. Proceedings of A-MOST2007, pp. 85-94.  
Utting, M., Legeard, B., 2007. Practical Model-Based Testing: a Tools Approach. Morgan Kaufmann.  
Wu, P., Lin, H., 2006. Model-based testing of concurrent programs with predicate sequencing constraints. International 
Int. J. Softw. Eng. Know., Vol. 16, No. 5, pp. 727-746.  
ISBN: 978-972-8924-97-3 © 2009 IADIS
302


Wang, Daoming

Lin, Huimin

Chen, Tsong Yueh

Swinburne Research Bank

On the integration of metamorphic testing and model checking

Abstract

Similar works

Full text

Available Versions

Victoria University Institutional Repository (VUIR)

Name not available

RMIT Research Repository

Swinburne Research Bank