Automatic generation of test sequences form EFSM models using evolutionary algorithms

Automated test data generation through evolutionary testing (ET) is a topic of interest to the software engineering community. While there are many ET-based techniques for automatically generating test data from code, the problem of generating test data from an extended finite state machine (EFSMs) is more complex and has received little attention. In this paper, we introduce a novel approach that addresses the problem of generating input test sequences that trigger given feasible paths in an EFSM model by employing an ET-based technique. The proposed approach expresses the problem as a search for input parameters to be applied to a set of functions to be called sequentially. In order to apply ET-based technique, a new fitness function is introduced to cope with the case when a test target involves calls to a set of transitions sequentially. We evaluate our approach empirically using five sets of randomly generated paths through two EFSM case studies: INRES and class 2 transport protocols. In the experiments, we apply two search techniques: a random and an ET-based which utilizes our new fitness function. Experimental results show that the proposed approach produces input test sequences that trigger all the feasible paths used with a success rate of 100%, however, the random technique failed in most cases with a success rate of 20.8%

Constraint-Based Heuristic On-line Test Generation from Non-deterministic I/O EFSMs

We are investigating on-line model-based test generation from non-deterministic output-observable Input/Output Extended Finite State Machine (I/O EFSM) models of Systems Under Test (SUTs). We propose a novel constraint-based heuristic approach (Heuristic Reactive Planning Tester (xRPT)) for on-line conformance testing non-deterministic SUTs. An indicative feature of xRPT is the capability of making reasonable decisions for achieving the test goals in the on-line testing process by using the results of off-line bounded static reachability analysis based on the SUT model and test goal specification. We present xRPT in detail and make performance comparison with other existing search strategies and approaches on examples with varying complexity.Comment: In Proceedings MBT 2012, arXiv:1202.582

Search Based Software Engineering in Membrane Computing

This paper presents a testing approach for kernel P Systems (kP systems), based on test data generation for a given scenario. This method uses Genetic Algorithms to generate the input sets needed to trigger the given computation steps

A search-based technique for testing from extended finite state machine model

Extended finite state machines (EFSMs), and languages such as state-charts that are similar to EFSMs, are widely used to model state-based systems. When testing from an EFSM M it is common to aim to produce a set of test sequences (input sequences) that satisfies a test criterion that relates to the transition paths (TPs) of M that are executed by the test sequences. For example, we might require that the set of TPs triggered includes all of the transitions of M. One approach to generating such a set of test sequences is to split the problem into two stages: choosing a set of TPs that achieves the test criterion and then producing test sequences to trigger these TPs. However, the EFSM may contain infeasible TPs and the problem of generating a test sequence to trigger a given feasible TP (FTP) is generally uncomputable. In this paper we present a search-based approach that uses two techniques: (1) A TP fitness metric based on our previous work that estimates the feasibility of a given transition path; and (2) A fitness function to guide the search for a test sequence to trigger a given FTP. We evaluated our approach on five EFSMs: A simple in-flight safety system; a class II transport protocol; a lift system; an ATM; and the Inres initiator. In the experiments the proposed approach successfully tested approximately 96.75 % of the transitions and the proposed test sequence generation technique triggered all of the generated FTPs

Comparison of two approaches for test case generations from EFSMs.

Testing is one of the vital steps in software development process. To convey testing, test cases need to be generated to check whether an implementation conforms to the design specification. Design specifications are usually expressed as Extended Finite State Machines (EFSMs) and test cases are actually a path from the initial state to a specific state on that EFSM. One of the most difficult issues of test case generation for EFSMs comes from the fact that infeasible paths exist on EFSMs. Two approaches have been developed in earlier 90s\u27 to generate feasible paths from EFSMs: one is to develop algorithm to search EFSMs directly to generate feasible paths, and the other is to expand EFSMs into Finite State Machines (FSMs), followed by applying FSM techniques to generate feasible paths. Model checking method was proposed recently as a new approach for test case generation. It has some advantages over previous methods such as efficiency on number of states explored. However, by nature, it also has some disadvantages such as time inefficiency. Here we present a comparison between the model checking method and the previous expansion method from pragmatic aspect by running experiments. To carry on this comparison, we implemented a classical expansion algorithm, defined the translation from EFSMs to Promela models, and used SPIN model checker in the model checking approach. We have run sufficient number of test case generation experiments, compared the two approaches on their time consumptions, numbers of states explored, performance changes when EFSMs\u27 sizes increase etc. By this comparison, we can see the tradeoff between time consumptions and the number of states explored in the two approaches and observe their performance changes while EFSMs change. Finally, we show the existence of the trade-off between state efficiency and time efficiency of the two approaches, the impact of domain size of variable value, the native drawbacks of the expansion algorithm and the performance improvement by tuning Premela models.Dept. of Computer Science. Paper copy at Leddy Library: Theses & Major Papers - Basement, West Bldg. / Call Number: Thesis2005 .T36. Source: Masters Abstracts International, Volume: 44-03, page: 1415. Thesis (M.Sc.)--University of Windsor (Canada), 2005

Improving Software Quality by Synergizing Effective Code Inspection and Regression Testing

Software quality assurance is an essential practice in software development and maintenance. Evolving software systems consistently and safely is challenging. All changes to a system must be comprehensively tested and inspected to gain confidence that the modified system behaves as intended. To detect software defects, developers often conduct quality assurance activities, such as regression testing and code review, after implementing or changing required functionalities. They commonly evaluate a program based on two complementary techniques: dynamic program analysis and static program analysis. Using an automated testing framework, developers typically discover program faults by observing program execution with test cases that encode required program behavior as well as represent defects. Unlike dynamic analysis, developers make sure of the program correctness without executing a program by static analysis. They understand source code through manual inspection or identify potential program faults with an automated tool for statically analyzing a program. By removing the boundaries between static and dynamic analysis, complementary strengths and weaknesses of both techniques can create unified analyses. For example, dynamic analysis is efficient and precise but it requires selection of test cases without guarantee that the test cases cover all possible program executions, and static analysis is conservative and sound but it produces less precise results due to its approximation of all possible behaviors that may perform at run time. Many dynamic and static techniques have been proposed, but testing a program involves substantial cost and risks and inspecting code change is tedious and error-prone. Our research addresses two fundamental problems in dynamic and static techniques. (1) To evaluate a program, developers are typically required to implement test cases and reuse them. As they develop more test cases for verifying new implementations, the execution cost of test cases increases accordingly. After every modification, they periodically conduct regression test to see whether the program executes without introducing new faults in the presence of program evolution. To reduce the time required to perform regression testing, developers should select an appropriate subset of the test suite with a guarantee of revealing faults as running entire test cases. Such regression testing selection techniques are still challenging as these methods also have substantial costs and risks and discard test cases that could detect faults. (2) As a less formal and more lightweight method than running a test suite, developers often conduct code reviews based on tool support; however, understanding context and changes is the key challenge of code reviews. While reviewing code changes—addressing one single issue—might not be difficult, it is extremely difficult to understand complex changes—including multiple issues such as bug fixes, refactorings, and new feature additions. Developers need to understand intermingled changes addressing multiple development issues, finding which region of the code changes deals with a particular issue. Although such changes do not cause trouble in implementation, investigating these changes becomes time-consuming and error-prone since the intertwined changes are loosely related, leading to difficulty in code reviews. To address the limitations outlined above, our research makes the following contributions. First, we present a model-based approach to efficiently build a regression test suite that facilitates Extended Finite State Machines (EFSMs). Changes to the system are performed at transition level by adding, deleting or replacing transition. Tests are a sequence of input and expected output messages with concrete parameter values over the supported data types. Fully-observable tests are introduced whose descriptions contain all the information about the transitions executed by the tests. An invariant characterizing fully observable tests is formulated such that a test is fully-observable whenever the invariant is a satisfiable formula. Incremental procedures are developed to efficiently evaluate the invariant and to select tests from a test suite that are guaranteed to exercise a given change when the tests run on a modified EFSM. Tests rendered unusable due to a change are also identified. Overlaps among the test descriptions are exploited to extend the approach to simultaneously select and discard multiple tests to alleviate the test selection costs. Although test regression selection problem is NP-hard [78], the experimental results show the cost of our test selection procedure is still acceptable and economical. Second, to support code review and regression testing, we present a technique, called ChgCutter. It helps developers understand and validate composite changes as follows. It interactively decomposes these complex, composite changes into atomic changes, builds related change subsets using program dependence relationships without syntactic violation, and safely selects only related test cases from the test suite to reduce the time to conduct regression testing. When a code reviewer selects a change region from both original and changed versions of a program, ChgCutter automatically identifies similar change regions based on the dependence analysis and the tree-based code search technique. By automatically applying a change to the identified regions in an original program version, ChgCutter generates a program version which is a syntactically correct version of program. Given a generated program version, it leverages a testing selection technique to select and run a subset of the test suite affected by a change automatically separated from mixed changes. Based on the iterative change selection process, there can be each different program version that include its separated change. Therefore, ChgCutter helps code reviewers inspect large, complex changes by effectively focusing on decomposed change subsets. In addition to assisting understanding a substantial change, the regression testing selection technique effectively discovers defects by validating each program version that contains a separated change subset. In the evaluation, ChgCutter analyzes 28 composite changes in four open source projects. It identifies related change subsets with 95.7% accuracy, and it selects test cases affected by these changes with 89.0% accuracy. Our results show that ChgCutter should help developers effectively inspect changes and validate modified applications during development

Automated test sequence generation for finite state machines using genetic algorithms

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.Testing software implementations, formally specified using finite state automata (FSA) has been of interest. Such systems include communication protocols and control sections of safety critical systems. There is extensive literature regarding how to formally validate an FSM based specification, but testing that an implementation conforms to the specification is still an open problem. Two aspects of FSA based testing, both NP-hard problems, are discussed in this thesis and then combined. These are the generation of state verification sequences (UIOs) and the generation of sequences of conditional transitions that are easy to trigger. In order to facilitate test sequence generation a novel representation of the transition conditions and a number of fitness function algorithms are defined. An empirical study of the effectiveness on real FSA based systems and example FSAs provides some interesting positive results. The use of genetic algorithms (GAs) makes these problems scalable for large FSAs. The experiments used a software tool that was developed in Java

Search-based software engineering: A search-based approach for testing from extended finite state machine (EFSM) models

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.The extended finite state machine (EFSM) is a powerful modelling approach that has been applied to represent a wide range of systems. Despite its popularity, testing from an EFSM is a substantial problem for two main reasons: path feasibility and path test case generation. The path feasibility problem concerns generating transition paths through an EFSM that are feasible and satisfy a given test criterion. In an EFSM, guards and assignments in a path‟s transitions may cause some selected paths to be infeasible. The problem of path test case generation is to find a sequence of inputs that can exercise the transitions in a given feasible path. However, the transitions‟ guards and assignments in a given path can impose difficulties when producing such data making the range of acceptable inputs narrowed down to a possibly tiny range. While search-based approaches have proven efficient in automating aspects of testing, these have received little attention when testing from EFSMs. This thesis proposes an integrated search-based approach to automatically test from an EFSM. The proposed approach generates paths through an EFSM that are potentially feasible and satisfy a test criterion. Then, it generates test cases that can exercise the generated feasible paths. The approach is evaluated by being used to test from five EFSM cases studies. The achieved experimental results demonstrate the value of the proposed approach.Aleppo University, Syri