Program simplification as a means of approximating undecidable propositions

We describe an approach which mixes testing, slicing, transformation and formal verification to investigate speculative hypotheses concerning a program, formulated during program comprehension activity. Our philosophy is that such hypotheses (which are typically undecidable) can, in some sense, be `answered' by a partly automated system which returns neither `true' nor `false' but a program (the `test program') which computes the answer. The motivation for this philosophy is the way in which, as we demonstrate, static analysis and manipulation technology can be applied to ensure that the resulting test program is significantly simpler than the original program, thereby simplifying the process of investigating the original hypothesi

Model-based Security Testing Using UMLsec A Case Study

AbstractDesigning and implementing security-critical systems correctly is very difficult. In practice, most vulnerabilities arise from bugs in implementations. We present work towards systematic specification-based testing of security-critical systems based on UMLsec models. We show how to systematically generate test sequences for security properties based on the model that can be used to test the implementation for vulnerabilities. We explain our method at the example of a part of the Common Electronic Purse Specifications (CEPS), a candidate for an international electronic purse standard

Testing data types implementations from algebraic specifications

Algebraic specifications of data types provide a natural basis for testing data types implementations. In this framework, the conformance relation is based on the satisfaction of axioms. This makes it possible to formally state the fundamental concepts of testing: exhaustive test set, testability hypotheses, oracle. Various criteria for selecting finite test sets have been proposed. They depend on the form of the axioms, and on the possibilities of observation of the implementation under test. This last point is related to the well-known oracle problem. As the main interest of algebraic specifications is data type abstraction, testing a concrete implementation raises the issue of the gap between the abstract description and the concrete representation. The observational semantics of algebraic specifications bring solutions on the basis of the so-called observable contexts. After a description of testing methods based on algebraic specifications, the chapter gives a brief presentation of some tools and case studies, and presents some applications to other formal methods involving datatypes

Delayed failure of software components using stochastic testing

The present research investigates the delayed failure of software components and addresses the problem that the conventional approach to software testing is unlikely to reveal this type of failure. Delayed failure is defined as a failure that occurs some time after the condition that causes the failure, and is a consequence of long-latency error propagation. This research seeks to close a perceived gap between academic research into software testing and industrial software testing practice by showing that stochastic testing can reveal delayed failure, and supporting this conclusion by a model of error propagation and failure that has been validated by experiment. The focus of the present research is on software components described by a request-response model. Within this conceptual framework, a Markov chain model of error propagation and failure is used to derive the expected delayed failure behaviour of software components. Results from an experimental study of delayed failure of DBMS software components MySQL and Oracle XE using stochastic testing with random generation of SQL are consistent with expected behaviour based on the Markov chain model. Metrics for failure delay and reliability are shown to depend on the characteristics of the chosen experimental profile. SQL mutation is used to generate negative as well as positive test profiles. There appear to be few systematic studies of delayed failure in the software engineering literature, and no studies of stochastic testing related to delayed failure of software components, or specifically to delayed failure of DBMS. Stochastic testing is shown to be an effective technique for revealing delayed failure of software components, as well as a suitable technique for reliability and robustness testing of software components. These results provide a deeper insight into the testing technique and should lead to further research. Stochastic testing could provide a dependability benchmark for component-based software engineering

Automated test generation from algebraic specifications

PhD ThesisThis thesis is a contribution to work on the specification-based testing of computing systems. The development of computing systems is a challenging task. A great deal of research has been directed at support for analysis, design and implementation aspects, yielding a wide range of development techniques. However, the crucial area of system testing remains relatively under-explored. Because a project may spend a good part of its budget on testing, even modest improvements to the cost-effectiveness of testing represent substantial improvements in project budgets. Relatively little literature has been devoted to the entire testing process, including specification, generation, execution and validation. Most of the academic literature seems to assume a revolutionary change of the testing framework. On the contrary industry follows a more traditional approach consisting of trusted methods and based on personal experience. There is a need for testing methods that improve the effectiveness of testing but do so at reasonable cost and which do not require a revolutionary change in the development technology. The novel goal of the work described in this thesis is to "lift" traditional testing so that it takes advantage of system specifications. We provide a framework - hepTEsT- which is motivated by this goal. To that end, hepTEsT is a framework consisting of a specification language, a technology for generating tests in accordance with test strategies, a means of applying the tests to the implementations and support for validation of outcomes against the specification-based tests. We will first categorise different testing methodologies and then examine some of the past and present approaches to test data: we develop only the necessary theoretical foundations for hepSPEc and always consider the requirements of testing. The formalism hepSPEc for system description is based upon a well-defined algebraic approach. It utilises a novel approach allowing the description of finite domains in a way suitable for engineering purposes. The engineers' tasks are to provide an adequate description of the system in hepSPEC. The approach proposed in this thesis is grounded in the traditional approach to testing where test data is provided to the system under test and the outcome is compared to the expected outcome. To enhance the capabilities of the framework a general order on test inputs is proposed to be used in test strategies. Traditional testing strategies requiring an order on test inputs are introduced and their realisation in hepTEsT discussed as well as a proposal of new strategies which lend themselves to this particular approach. The manipulation of the specification yields abstract test cases which are then transformed into test cases suitable for the chosen implementation of the system. This transformation, called test reification, is necessary to bridge the "abstraction gap" between the abstract specification-derived tests and the concrete implementation on which the test must run. The transformation is necessary in order for the approach to be practical and is achieved through homomorphisms which are expressed in specially adapted grammars. This transformation is also applied to the generated test outcome and is aimed there at easing test result validation. The utility of the hepTEsT approach is illustrated by means of a simple example, a larger case study and one carried out within the aviation industry