17 research outputs found

    Choice and chance:model-based testing of stochastic behaviour

    Get PDF
    Probability plays an important role in many computer applications. A vast number of algorithms, protocols and computation methods uses randomisation to achieve their goals. A crucial question then becomes whether such probabilistic systems work as intended. To investigate this, such systems are often subjected to a large number of well-designed test cases, that compare a observed behaviour to a requirements specification. Model-based testing is an innovative testing technique rooted in formal methods, that aims at automating this labour intense and often error-prone manual task. By providing faster and more thorough testing at lower cost, it has gained rapid popularity in industry and academia alike. However, classic model-based testing methods are insufficient when dealing with inherently stochastic systems. This thesis introduces a rigorous model-based testing framework, that is capable to automatically test such systems. The presented methods are capable of judging functional correctness, discrete probability choices, and hard and soft-real time constraints. The framework is constructed in a clear step-by-step approach. First, the model-based testing landscape is laid out, and related work is discussed. Next, we instantiate a model-based testing framework to highlight the purpose of individual theoretical components like, e.g., a conformance relation, test cases, and practical test generation algorithms. This framework is then conservatively extended by introducing discrete probability choices to the specification language. A last step further extends this probabilistic framework by adding hard and soft real time constraints. Classical functional correctness verdicts are thus extended with goodness of fit methods known from statistics. Proofs of the frameworkā€™s correctness are presented before its capabilities are exemplified by studying smaller scale case studies known from the literature. The framework reconciles non-deterministic and probabilistic choices in a fully-fledged way via the use of schedulers. Schedulers then become a subject worthy to study in their own rights. This is done in the second part of this thesis; we introduce a most natural equivalence relation based on schedulers for Markov automata, and compare its distinguishing power to notions of trace distributions and bisimulation relations. Lastly, the power of different scheduler classes of stochastic automata is investigated. We compare reachability probabilities of different schedulers by altering the information available to them. A hierarchy of scheduler classes is established, with the intent to reduce complexity of related problems by gaining near optimal results for smaller scheduler classes

    Investigation of similarity-based test case selection for specification-based regression testing.

    Get PDF
    uring software maintenance, several modiļ¬cations can be performed in a speciļ¬cation model in order to satisfy new requirements. Perform regression testing on modiļ¬ed software is known to be a costly and laborious task. Test case selection, test case prioritization, test suite minimisation,among other methods,aim to reduce these costs by selecting or prioritizing a subset of test cases so that less time, effort and thus money are involved in performing regression testing. In this doctorate research, we explore the general problem of automatically selecting test cases in a model-based testing (MBT) process where speciļ¬cation models were modiļ¬ed. Our technique, named Similarity Approach for Regression Testing (SART), selects subset of test cases traversing modiļ¬ed regions of a software systemā€™s speciļ¬cation model. That strategy relies on similarity-based test case selection where similarities between test cases from different software versions are analysed to identify modiļ¬ed elements in a model. In addition, we propose an evaluation approach named Search Based Model Generation for Technology Evaluation (SBMTE) that is based on stochastic model generation and search-based techniques to generate large samples of realistic models to allow experiments with model-based techniques. Based on SBMTE,researchers are able to develop model generator tools to create a space of models based on statistics from real industrial models, and eventually generate samples from that space in order to perform experiments. Here we developed a generator to create instances of Annotated Labelled Transitions Systems (ALTS), to be used as input for our MBT process and then perform an experiment with SART.In this experiment, we were able to conclude that SARTā€™s percentage of test suite size reduction is robust and able to select a sub set with an average of 92% less test cases, while ensuring coverage of all model modiļ¬cation and revealing defects linked to model modiļ¬cations. Both SART and our experiment are executable through the LTS-BT tool, enabling researchers to use our selections trategy andr eproduce our experiment.During software maintenance, several modiļ¬cations can be performed in a speciļ¬cation model in order to satisfy new requirements. Perform regression testing on modiļ¬ed software is known to be a costly and laborious task. Test case selection, test case prioritization, test suite minimisation,among other methods,aim to reduce these costs by selecting or prioritizing a subset of test cases so that less time, effort and thus money are involved in performing regression testing. In this doctorate research, we explore the general problem of automatically selecting test cases in a model-based testing (MBT) process where speciļ¬cation models were modiļ¬ed. Our technique, named Similarity Approach for Regression Testing (SART), selects subset of test cases traversing modiļ¬ed regions of a software systemā€™s speciļ¬cation model. That strategy relies on similarity-based test case selection where similarities between test cases from different software versions are analysed to identify modiļ¬ed elements in a model. In addition, we propose an evaluation approach named Search Based Model Generation for Technology Evaluation (SBMTE) that is based on stochastic model generation and search-based techniques to generate large samples of realistic models to allow experiments with model-based techniques. Based on SBMTE,researchers are able to develop model generator tools to create a space of models based on statistics from real industrial models, and eventually generate samples from that space in order to perform experiments. Here we developed a generator to create instances of Annotated Labelled Transitions Systems (ALTS), to be used as input for our MBT process and then perform an experiment with SART.In this experiment, we were able to conclude that SARTā€™s percentage of test suite size reduction is robust and able to select a sub set with an average of 92% less test cases, while ensuring coverage of all model modiļ¬cation and revealing defects linked to model modiļ¬cations. Both SART and our experiment are executable through the LTS-BT tool, enabling researchers to use our selections trategy andr eproduce our experiment

    Model Checking and Model-Based Testing : Improving Their Feasibility by Lazy Techniques, Parallelization, and Other Optimizations

    Get PDF
    This thesis focuses on the lightweight formal method of model-based testing for checking safety properties, and derives a new and more feasible approach. For liveness properties, dynamic testing is impossible, so feasibility is increased by specializing on an important class of properties, livelock freedom, and deriving a more feasible model checking algorithm for it. All mentioned improvements are substantiated by experiments

    Framework for Automated Functional Tests within Value-Added Service Environments

    Get PDF
    Full version unavailable due to 3rd party copyright restrictions.Recent years have witnessed that standard telecommunication services evolved more and more to next generation value-added services. This fact is accompanied by a change of service characteristics as new services are designed to fulfil the customerā€™s demands instead of just focussing on technologies and protocols. These demands can be very specific and, therefore, diverse potential service functionalities have to be considered by the service providers. To make matters worse for service providers, a fast transition from concept to market product and low price of a new service is required due to the increasing competition in the telecommunication industry. Therefore, effective test solutions need to be developed that can be integrated in current value-added service development life-cycles. Besides, these solutions should support the involvement of all participating stakeholders such as the service provider, the test developers as well as the service developers, and, in order to consider an agile approach, also the service customer. This thesis proposes a novel framework for functional testing that is based on a new sort of description language for value-added services (Service Test Description). Based on instances of the Service Test Description, sets of reusable test components described by means of an applied Statecharts notation are automatically selected and composed to so-called behaviour models. From the behaviour models, abstract test cases can be automatically generated which are then transformed to TTCN-3 test cases and then assembled to an Executable Test Suite. Within a TTCN-3 test system, the Executable Test Suite can be executed against the corresponding value-added service referred to as System Under Test. One benefit of the proposed framework is its application within standard development life-cycles. Therefore, the thesis presents a methodology that considers both service development and test development as parallel tasks and foresees procedures to synchronise the tasks and to allow an agile approach with customer involvement. The novel framework is validated through a proof-of-concept working prototype. Example value-added services have been chosen to illustrate the whole process from compiling instances of the Service Test Description until the execution of automated tests. Overall, this thesis presents a novel solution for service providers to improve the quality of their provided value-added services through automated functional testing procedures. It enables the early involvement of the customers into the service development life-cycle and also helps test developers and service developers to collaborate

    Model-Based Scenario Testing and Model Checking with Applications in the Railway Domain

    Get PDF
    This thesis introduces Timed Moore Automata, a specification formalism, which extends the classical Moore Automata by adding the concept of abstract timers without concrete delay time values, which can be started and reset, and which can change their state from running to elapsed. The formalism is used in real-world railway domain applications, and algorithms for the automated test data generation and explicit model checking of Timed Moore Automata models are presented. In addition, this thesis deals with test data generation for larger scale test models using standardized modeling formalisms. An existing framework for the automated test data generation is presented, and its underlying work-flow is extended and modified in order to allow user interaction and guidance within the generation process. As opposed to specifying generation constraints for entire test scenarios, the modified work flow then allows for an iterative approach to elaborating and formalizing test generation goals

    A Model-Driven Framework for Context-Dependent Component Testing

    Get PDF

    Fail-Safe Testing of Safety-Critical Systems

    Get PDF
    This dissertation proposes an approach for testing of safety-critical systems. It is based on a behavioral and a fault model. The two models are analyzed for compatibility and necessary changes are identified to make them compatible. Then transformation rules are used to transform the fault model into the same model type as the behavioral model. Integration rules define how to combine them. This approach results in an integrated model which then can be used to generate tests using a variety of testing criteria. The dissertation illustrates this general framework using a CEFSM for the behavioral model and a Fault Tree for the fault model. We apply the technique to a variety of applications such as a Gas burner, an Aerospace Launch System, and a Railroad Crossing Control System. We also investigate the scalability of the approach and compare its efficiency with integrating a state chart and a fault tree. Construction and Analysis of Distributed Processes (CADP) has been used as a supporting tool for this approach to generate test cases from the integrated model and to analyze the integrated model for some properties such as deadlock and livelock

    Higher Order Mutation Testing

    Get PDF
    Mutation testing is a fault-based software testing technique that has been studied widely for over three decades. To date, work in this field has focused largely on first order mutants because it is believed that higher order mutation testing is too computationally expensive to be practical. This thesis argues that some higher order mutants are potentially better able to simulate real world faults and to reveal insights into programming bugs than the restricted class of first order mutants. This thesis proposes a higher order mutation testing paradigm which combines valuable higher order mutants and non-trivial first order mutants together for mutation testing. To overcome the exponential increase in the number of higher order mutants a search process that seeks fit mutants (both first and higher order) from the space of all possible mutants is proposed. A fault-based higher order mutant classification scheme is introduced. Based on different types of fault interactions, this approach classifies higher order mutants into four categories: expected, worsening, fault masking and fault shifting. A search-based approach is then proposed for locating subsuming and strongly subsuming higher order mutants. These mutants are a subset of fault mask and fault shift classes of higher order mutants that are more difficult to kill than their constituent first order mutants. Finally, a hybrid test data generation approach is introduced, which combines the dynamic symbolic execution and search based software testing approaches to generate strongly adequate test data to kill first and higher order mutants

    Controllability problems in MSC-based testing

    Get PDF
    This is a pre-copyedited, author-produced PDF of an article accepted for publication in The Computer Journal following peer review. The definitive publisher-authenticated version [Dan, H and Hierons, RM (2012), "Controllability Problems in MSC-Based Testing", The Computer Journal, 55(11), 1270-1287] is available online at: http://comjnl.oxfordjournals.org/content/55/11/1270. Copyright @ The Authors 2011.In testing systems with distributed interfaces/ports, we may place a separate tester at each port. It is known that this approach can introduce controllability problems which have received much attention in testing from finite state machines. Message sequence charts (MSCs) form an alternative, commonly used, language for modelling distributed systems. However, controllability problems in testing from MSCs have not been thoroughly investigated. In this paper, controllability problems in MSC test cases are analysed with three notions of observability: local, tester and global. We identify two types of controllability problem in MSC-based testing. It transpires that each type of controllability problem is related to a type of MSC pathology. Controllability problems of timing are caused by races but not every race causes controllability problems; controllability problems of choice are caused by non-local choices and not every non-local choice causes controllability problems. We show that some controllability problems of timing are avoidable and some controllability problems of choice can be overcome when testers have better observational power. Algorithms are provided to tackle both types of controllability problems. Finally, we show how one can overcome controllability problems using a coordination service with status messages based on algorithms developed in this paper.EPSR
    corecore