Testing object-oriented industrial software without precise oracles or results

Software testing such as object-oriented software Testing At the Class and Cluster LEvels, or TACCLE, can be achieved by defining the test objectives, selecting and executing test cases, and checking results. The software specifications are extracted from technical drawings of mechanical and electronic hardware designed by process engineers using an in-house technique. TACCLE enables software engineers to test each individual class independently, then test the interaction among classes. The TACCLE methodology is described in three sections that include class-level testing, cluster-level testing, and test case generation and test script translation. The notion of testing observational equivalence and bypassing the need for oracles is important in test automation for industrial projects where it is impractical to define a precise relationship between specification and the software under test.postprin

The Oracle Problem in Software Testing: A Survey

Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice

Testing data types implementations from algebraic specifications

Algebraic specifications of data types provide a natural basis for testing data types implementations. In this framework, the conformance relation is based on the satisfaction of axioms. This makes it possible to formally state the fundamental concepts of testing: exhaustive test set, testability hypotheses, oracle. Various criteria for selecting finite test sets have been proposed. They depend on the form of the axioms, and on the possibilities of observation of the implementation under test. This last point is related to the well-known oracle problem. As the main interest of algebraic specifications is data type abstraction, testing a concrete implementation raises the issue of the gap between the abstract description and the concrete representation. The observational semantics of algebraic specifications bring solutions on the basis of the so-called observable contexts. After a description of testing methods based on algebraic specifications, the chapter gives a brief presentation of some tools and case studies, and presents some applications to other formal methods involving datatypes

A Survey of Languages for Specifying Dynamics: A Knowledge Engineering Perspective

A number of formal specification languages for knowledge-based systems has been developed. Characteristics for knowledge-based systems are a complex knowledge base and an inference engine which uses this knowledge to solve a given problem. Specification languages for knowledge-based systems have to cover both aspects. They have to provide the means to specify a complex and large amount of knowledge and they have to provide the means to specify the dynamic reasoning behavior of a knowledge-based system. We focus on the second aspect. For this purpose, we survey existing approaches for specifying dynamic behavior in related areas of research. In fact, we have taken approaches for the specification of information systems (Language for Conceptual Modeling and TROLL), approaches for the specification of database updates and logic programming (Transaction Logic and Dynamic Database Logic) and the generic specification framework of abstract state machine

Using formal methods to support testing

Formal methods and testing are two important approaches that assist in the development of high quality software. While traditionally these approaches have been seen as rivals, in recent years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing

Equality to equals and unequals: a revisit of the equivalence and nonequivalence criteria in object-oriented software testing

published_or_final_versio

Testing from Structured Algebraic Specifications: The Oracle Problem

Work in the area of specification-based testing has pointed out that testing can be effectively used to verify programs against formal specifications. The aim is to derive test information from formal specifications so that testing can be rigorously applied whenever full formal verification is not cost-effective. However, there are still several obstacles to be overcome in order to establish testing as a standard in formal frameworks. Accurate interpretation of test results is an extremely critical one. This thesis is concerned with testing programs against structured algebraic specifications where axioms are expressed in first-order logic with equations, the usual connectives and quantifiers. The main issue investigated is the so-called oracle problem, that is, whether a decision procedure can be defined for interpreting the results of tests according to a formal specification. In this context, testing consists in checking whether specification axioms are satisfied by programs. Consequently, tests exercise operations referred to by the axioms and oracles evaluate the axioms according to the results produced by the tests. The oracle problem for flat (unstructured) specifications often reduces to the problem of comparing two values of a non-observable sort, namely the equality problem, and also how to deal with quantifiers which may demand infinite test sets. Equality on non-observable sorts is interpreted up to behavioural equivalence with observational equivalence as an important special case. However, a procedure for implementing such a behavioural equality may be hard to define or even impossible. In this thesis, a solution to the oracle problem for flat specifications is presented which tackles the equality problem by using a pair of approximate equalities, one finer than behavioural equality and one coarser, and taking the syntactic position of quantifiers in formulae into account. Additionally, when structured specifications are considered, the oracle problem can be harder. The reason is that specifications may be composed of parts over different signatures, and the structure must be taken into account in order to interpret test results according to specification axioms. Also, an implementation of hidden (non-exported) symbols may be required in order to check axioms which refer to them. Two solutions to the oracle problem for structured specifications are presented in this thesis based on a compositional and a non-compositional style of testing, namely structured testing and flat testing respectively. Structured testing handles the oracle problem more effectively than flat testing and under fewer assumptions. Furthermore, testing from structured specifications may require an approach which lies in between flat and structured testing. Therefore, based on normalisation of ordinary specifications, three normal forms are presented for defining a more practical and combined approach to testing and also coping more effectively with the oracle problem. The use of normal forms gives rise to a style of testing called semi-structured testing where some parts of the specification are replaced by normal forms and the result is checked using structured testing. Testing from normal forms can be very convenient whenever the original specification is too complex or oracles cannot be defined from it

A model of systems with modes and mode transitions

We propose a method of classifying the operation of a system into finitely many modes. Each mode has its own objectives for the system's behaviour and its own algorithms designed to accomplish its objectives. A central problem is deciding when to transition from one mode to some other mode, a decision that may be contested and involve partial or inconsistent information. We propose some general principles and model mathematically their conception of modes for a system. We derive a family of data types for analysing mode transitions; these are simplicial complexes, both abstract and concretely realised as geometric spaces in euclidean space . In the simplicial complex, a mode is represented by a simplex and each state of a system can be evaluated by mapping it into one or more simplices. This evaluation measures the extent to which different modes are appropriate for the state and can decide on a transition. To illustrate the general model in some detail, we work though a case study of an autonomous racing car

Automated Synthesis of Quantum Subcircuits

The quantum computer has become contemporary reality, with the first two-qubit machine of mere decades ago transforming into cloud-accessible devices with tens, hundreds, or--in a few cases--even thousands of qubits. While such hardware is noisy and still relatively small, the increasing number of operable qubits raises another challenge: how to develop the now-sizeable quantum circuits executable on these machines. Preparing circuits manually for specifications of any meaningful size is at best tedious and at worst impossible, creating a need for automation. This article describes an automated quantum-software toolkit for synthesis, compilation, and optimization, which transforms classically-specified, irreversible functions to both technology-independent and technology-dependent quantum circuits. We also describe and analyze the toolkit's application to three situations--quantum read-only memories, quantum random number generators, and quantum oracles--and illustrate the toolkit's start-to-finish features from the input of classical functions to the output of quantum circuits ready-to-run on commercial hardware. Furthermore, we illustrate how the toolkit enables research beyond circuit synthesis, including comparison of synthesis and optimization methods and deeper understanding of even well-studied quantum algorithms. As quantum hardware continues to develop, such quantum circuit toolkits will play a critical role in realizing its potential.Comment: 49 pages, 25 figures, 20 table