Testing by means of inductive program learning

Given a program P and a set of alternative programs //’, we generate a sequence of test cases that are adequate, in the sense that they distinguish the given program from all alternatives The, m(,thod is related to fault-based approaches to test case generation, but programs in P need not he s]mp]e mutations of P. The technique for generating an adequate test set is based on the inductive learning of programs from finite sets of input-output examples: given a partial test set. we generate inductively a program P ’ E P which is consistent with P on those input values; then we look for an input value that distinguishes P from P’, and we repeat the process until no program except P can be induced from the generated examples. We show that the obtained test set is adequate with respect to the alternatives belonging to P. The method IS made possible by a program induction procedure which has evolved from recent research in mnchine Iei]rnlng and inductive logic programming. An implemented version of the test case ~encration procedure is demonstrated on simple and more complex list-processing programs. and tb(, scalability of ’ the approach is discussed

Using formal methods to support testing

Formal methods and testing are two important approaches that assist in the development of high quality software. While traditionally these approaches have been seen as rivals, in recent years a new consensus has developed in which they are seen as complementary. This article reviews the state of the art regarding ways in which the presence of a formal specification can be used to assist testing

Assessment of a Space Shuttle trajectory evaluation system (DOLILU II)

DOLILU II is a ground control system that generates space shuttle\u27s launch trajectories, first stage guidance commands and verifies whether the generated trajectories are safe for the flight. It is a safety critical system and a high degree of confidence in its safety and reliability must be gained through assessment. We addressed three issues related to its safety and reliability assessment. We developed a reliability assessment framework for DOLILU II system. We proposed techniques to speed up test case execution and designed methodologies for the generation of input conditions needed to test the system.;We used a Bayesian statistical framework for reliability assessment. Bayesian statistics uses knowledge about the system to be incorporated into the reliability model before testing. DOLILU II has been operational for nearly five years. We use this information when developing the reliability model. This information is introduced in the form of prior beliefs.;DOLILU II system requires an average time of 30 minutes for each test run. This translates into a large time period required for testing to demonstrate that DOLILU II exhibits the required failure rate. Vertical slicing, a semantic transformation technique, is used to prove the possibility of parallel execution and enhance each test case execution.;DOLILU II is an on-demand system. Many test trajectories are needed for its assessment. Regression methods were used to develop models for the generation of input data

Guaranteed Verification of Dynamic Systems

Diese Arbeit beschreibt einen neuen Spezifikations- und Verifikationsansatz für dynamische Systeme. Der neue Ansatz ermöglicht dabei Ergebnisse, die per Definition frei von Fehlern 2. Art sind. Dies bedeutet, dass das Ergebnis der Verifikation keine versteckten Fehler enthalten kann. Somit können zuverlässige Ergebnisse für die Analyse von sicherheitskritischen Systemen generiert werden. Dazu wird ein neues Verständnis von mengenbasierter Konsistenz dynamischer Systeme mit einer gegebenen Spezifikation eingeführt. Dieses basiert auf der Verwendung von Kaucher Intervall Arithmetik zur Einschließung von Messdaten. Konsistenz wird anhand der vereinigten Lösungsmenge der Kaucher Arithmetik definiert. Dies führt zu mathematisch garantierten Ergebnissen. Die resultierende Methode kann das spezifizierte Verhalten eines dynamischen System auch im Falle von Rauschen und Sensorungenauigkeiten anhand von Messdaten verifizieren. Die mathematische Beweisbarkeit der Konsistenz wird für eine große Klasse von Systemen gezeigt. Diese beinhalten zeitinvariante, intervallartige und hybride Systeme, wobei letztere auch zur Beschreibung von Nichtlinearitäten verwendet werden können. Darüber hinaus werden zahlreiche Erweiterungen dargestellt. Diese führen bis hin zu einem neuartigen iterativen Identifikations- und Segmentierungsverfahren für hybride Systeme. Dieses ermöglicht die Verfikation hybrider Systeme auch ohne Wissen über Schaltzeitpunkte. Die entwickelten Verfahren können darüber hinaus zur Diagnose von dynamischen Systemen verwendet werden, falls eine ausreichend schnelle Berechnung der Ergebnisse möglich ist. Die Verfahren werden erfolgreich auf eine beispielhafte Variation verschiedener Tanksysteme angewendet. Die neuen Theorien, Methoden und Algortihmen dieser Arbeit bilden die Grundlage für eine zuverlässige Analyse von hochautomatisierten sicherheitskritischen Systemen

Quality measures and assurance for AI (Artificial Intelligence) software

This report is concerned with the application of software quality and evaluation measures to AI software and, more broadly, with the question of quality assurance for AI software. Considered are not only the metrics that attempt to measure some aspect of software quality, but also the methodologies and techniques (such as systematic testing) that attempt to improve some dimension of quality, without necessarily quantifying the extent of the improvement. The report is divided into three parts Part 1 reviews existing software quality measures, i.e., those that have been developed for, and applied to, conventional software. Part 2 considers the characteristics of AI software, the applicability and potential utility of measures and techniques identified in the first part, and reviews those few methods developed specifically for AI software. Part 3 presents an assessment and recommendations for the further exploration of this important area

Formal functional testing of graphical user interfaces.

SIGLEAvailable from British Library Document Supply Centre- DSC:DX177960 / BLDSC - British Library Document Supply CentreGBUnited Kingdo

A Machine With Class: A Framework for Object Generation, Integration and Language Authentication (FROGILA)

The object technology model is constantly evolving to address the software crisis problem. This novel idea which informed and currently guides the design style of most modern scalable software systems has caused a strong belief that the object-oriented technology is the ultimate answer to the software crisis, i.e. applying an object-oriented development method will eventually lead to quality code. It is important to emphasise that object-orientedness does not make testing obsolete. As a matter of fact, some aspects of its very nature introduce new problems into the production of correct programs and their testing due to paradigmatic features like encapsulation, inheritance, polymorphism and dynamic binding as this research work shows. Most work in testing research has centred on procedure-oriented software with worthwhile methods of testing having been developed as a result. However, those cannot be applied directly to object-oriented software owing to the fact that the architectures of such systems differ on many key issues. In this thesis, we investigate and review the problems introduced by the features of the object technology model and then proceed to show why traditional structured software testing techniques are insufficient for testing object-oriented software by comparing the fundamental differences in their architecture. Also, by reviewing Weyuker’s test adequacy axioms we show that program-based testing and specification-based testing are orthogonal and complementary. Thus, a software testing methodology that is solely based on one of these approaches (i.e. program-based or specification-based testing) cannot adequately cover all the essential paths of the system under test or satisfactorily guarantee correctness in practice. We argue that a new method is required which integrates the benefits of the two approaches and further builds upon their individual strengths to create a more meaningful, practical and reliable solution. To this end, this thesis introduces and discusses a new automaton-based framework formalism for object-oriented classes called the Class-Machine and a test method that is based on this formalism. Here, the notion of a class or the idea behind classification in object-oriented languages is embodied within a machine framework. The Class-Machine model represents a polymorphic abstraction for heterogeneous families of Object-Machines that model a real life problem in a given domain; these Object-Machines are instances of different concrete machine types. The Class-Machine has an extensible machine implementation as well as an extensible machine interface. Thus, the Class-Machine is introduced as a formal framework for generating autonomous Object-Machines (i.e. Object-Machine Generator) that share common Generic Class-Machine States and Specific Object-Machine States. The states of these Object-Machines are manipulated by a set of processing functions (i.e. Class-Machine Methods and Object-Machine Methods) that must satisfy a set of preconditions before they are allowed to modify the state(s) of the Object-Machines. The Class-Machine model can also be viewed as a platform for integrating a society of communicating Object-Machines. To verify and completely test systems that adhere to the Class-Machine framework, a novel testing method is proposed i.e. the fault-finders (f²) - a distributed family of software checkers specifically designed to crawl through a Class-Machine implementation to look for a particular type of fault and tell us the location of the fault in the program (i.e. the class under test). Given this information, we can statistically show the distribution of faults in an object-oriented system and then provide a probabilistic assertion of the number and type of faults that remain undetected after testing is completed. To address the problems caused through the encapsulation mechanism, this thesis introduces and discusses another novel framework formalism that has complete visibility on all the encapsulated methods, memory states of the instance and class variables of a given Object-Machine or Class-Machine system under test. We call this the Class Machine Friend Function (CMƒƒ). In order to further illustrate all the fundamental theoretical ideas and paradigmatic features inherent within our proposed Class-Machine model, this thesis considers four different Class-Machine case studies. Finally, to further show that the Class-Machine theoretical purity does not mitigate against practical concerns, our novel object-oriented specification, verification, debugging and testing approaches proposed in this thesis are exemplified in an automated testing tool called: The Class-Machine Testing Tool (CMTT)

Specification Directed Module Testing

Abstract — If a program is developed from a specification in a mathematically rigorous manner, work done in the development can be utilized in the testing of the program. We can apply the better understanding afforded by these methods to provide a more thorough check on the correct operation of the program under test. This should lead to earlier detection of faults (making it easier to determine their causes), more useful debugging information, and a greater confidence in the correctness of the final product. Overall, a more systematic approach should expedite the task of the program tester, and improve software reliability. The testing techniques described in this paper apply to testing of abstract data types (modules, packages). The techniques utilize information generated during refinement of a data type, such as the data type invariant and the relationship between the specification and implementation states; this information is used to specify parts of the code to be written for testing. The techniques are illustrated by application to the implementation of a symbol table as an ordered list and as a height-balanced tree. Index Terms — Abstract data types, data type invariant, modules, module testing, packages, pre- and postconditions, retrieval function, software reliability, specification language—Z