Formal Testing of Object-Oriented Software: from the Method to the Tool

This thesis presents a method and a tool for test set selection, dedicated to object-oriented applications and based on formal specifications. Testing is one method to increase the quality of today's extraordinary complex software. The aim is to find program errors with respect to given criteria of correctness. In the case of formal testing, the criterion of correctness is the formal specification of the tested application: program behaviors are compared to those required by the specification. In this context, the difficulty of testing object-oriented software arises from the fact that the behavior of an object does not only depend on the input values of the parameters of its operations, but also on its current state, and generally on the current states of other related objects. This combinatorial explosion requires carefully selecting pertinent test sets of reasonable size. This thesis proposes a formal testing method which takes this issue into account. Our approach is based on two different formalisms: a specification language well adapted to the expression of system properties from the specifier's point of view, and a test language well adapted to the description of test sets from the tester's point of view. Specifications are written in an object-oriented language, CO-OPN (Concurrent Object-Oriented Petri Nets), based on synchronized algebraic Petri nets and devoted to the specification of concurrent systems. Test sets are expressed using a very simple temporal logic, HML (Hennessy-Milner Logic), whose logic formulas can be executed by a program. There exists a full agreement, shown in this thesis, between the CO-OPN and HML satisfaction relationships: the program satisfies its specification if and only if it satisfies the exhaustive test set derived from this specification. The exhaustive test set expresses all the specification properties. The exhaustive test set is generally infinite. Its size is reduced by applying hypotheses to the program behavior. These hypotheses define test selection strategies and reflect common test practices. The quality of the test sets thus selected only depends on the pertinence of the hypotheses. Concretely, the reduction is achieved by associating to each hypothesis applied to the program, a constraint on the test set. Our method proposes a set of elementary constraints: syntactic constraints on the structure of the tests and semantic constraints which allow to instantiate the test variables so as to cover the different classes of behaviors induced by the specification (subdomain decomposition). Elementary constraints can be combined to form complex constraints. Finally, the constraint system defined on the exhaustive test set is solved, and the solution leads to a pertinent test set of reasonable size. Thanks to the CO-OPN semantics, which allows to compute all the correct and incorrect behaviors induced by a specification, our method is able to test, on the one hand that a program does possess correct behaviors, and on the other hand that a program does not possess incorrect behaviors. An advantage of this approach is to provide through the tests, an observational description of valid and invalid implementations. Our testing method exhibits the advantage of being formal, and thus allows a semi-automation of the test selection process. A new tool, called CO-OPNTEST, is presented in this thesis. This tool assists the tester during the construction of constraints to apply to the exhaustive test set; afterward it automatically generates a test set satisfying these constraints. The CO-OPNTEST architecture is composed of a PROLOG kernel and a Java graphical interface. The kernel is an equational resolution procedure based on logic programming. It includes control mechanisms for subdomain decomposition. The graphical interface allows a user-friendly definition of the test constraints. The CO-OPNTEST tool has generated test sets for several case studies in a simple, rapid and efficient way. In particular, it has generated test sets for an industrial case study of realistic size: the control program of a production cell [Lewerentz 95]. CO-OPNTEST and its application to significant examples demonstrate the pertinence of our approach

Une Méthode de Tests Fonctionnelle Générale

Nous nous proposons de développer une méthode d'assistance à la sélection de jeux de tests à partir de spécifications formelles. Nos objectifs étant de déboucher sur une méthode de tests qui soit générale, c'est à dire qui ne soit pas dépendante d'un type de spécification formelle particulier. La méthode de tests qui nous intéresse est de type fonctionnel, ou "boîte noire", par opposition aux méthodes classiques de type structurel, ou "boîte blanche". Il s'agit d'évaluer les fonctionnalités d'un programme, sans analyser les détails de son code, mais en confrontant les résultats obtenus par l'exécution de ce code à la spécification formelle de l'application. Contrairement au code d'un programme, sa spécification formelle n'a pas pour vocation d'être exécutée, mais de décrire les comportements de l'application à l'aide de formules exprimées dans un langage logique spécialisé. La sélection d'un jeu de tests judicieux pour chaque formule permet de couvrir toutes les fonctionnalités attendues du système, et en particulier de tester les comportements critiques

Testing Object-Oriented Software

This tutorial describes the approach we develop for the formal testing of OO software. We also describe the operational techniques, that can be used, to select test cases

State-based Monitoring and Goal-driven Project Steering: Field Study of the SEMAT Essence Framework

At Carnegie Mellon University in Silicon Valley, the graduate master program ends with a practicum project during which students serve as software engineering consultants for an industry client. In this context, students are challenged to demonstrate their ability to work on self-managing and self-organizing teams. This paper presents a field study of the Software Engineering Method and Theory (SEMAT) Essence framework. The objective is to evaluate the effectiveness of the Essence’s novel state-based monitoring and goal-driven steering approach. The researchers conducted the study on seven graduate master student teams applying the approach throughout their practicum projects. The research methodology involves weekly observation and recording of each team’s state progression and collecting students’ reflection on the application of the approach. The main result validates that the approach provides student teams with a holistic, lightweight, non-prescriptive and method-agnostic way to monitor progress and steer projects, as well as an effective structure for retrospective and risk management. The monitoring and steering mechanisms are most effective during project initiation. The benefit decreases later in the project lifecycle. The conclusion is that the approach adds value to the whole learning and teaching experience and hence could be leveraged in software engineering education.</p

Testing the Production Cell Case Study from CO-OPN Specification

The purpose of this work is to evaluate, by means of a case study, our formal testing method for object-oriented software. For this purpose we propose the development of an object-oriented application of realistic size, addressing all the phases of the software life-cycle: requirements, analysis, design, formal description, implementation and testing. Analysis and design are performed with the Fusion method, formal description with the CO-OPN language, implementation with Ada 95 and test selection with our testing method. The description of the production cell case study and Fusion models are presented in technical report EPFL-DI 98/298. This report presents the formal description and testing phase

Modelling the Production Cell Case Study using the Fusion Method

The purpose of this work is to evaluate, by means of a case study, our formal testing method for object-oriented software. For this purpose we propose the development of an object-oriented application of realistic size, addressing all the phases of the software life-cycle: requirements, analysis, design, formal description, implementation and testing. Analysis and design are performed with the Fusion method, formal description with the CO-OPN language, implementation with Ada 95 and test selection with our testing method. This report presents the description of the production cell case study and Fusion models. The formal description and testing phase are presented in technical report EPFL-DI 98/299

A Theory of Specification-Based Testing for Object-Oriented Software

The current strategies for testing object-oriented software all lack the formal basis which is necessary to perform this task efficiently. We propose the adaptation to object-oriented software of an existing theory of testing for stateless ADTs, to find errors in a class by checking that its implementation meets its specification. We present shortly in an informal way an object-oriented language, CO-OPN/2, in which language we will write the specification. We introduce a notion of test that takes into account the possible and impossible sequences of call of class methods. We examine the black-box test procedure, and give techniques to select a finite and pertinent test set from an exhaustive test set, including all the possible behaviors of the class under test, by applying test reduction hypothesis. We also study the construction of an oracle, the procedure that analyses the results of the tests, adapted to object-oriented software

Test Selection for Object-Oriented Software Based on Formal Specifications

We propose a method of test selection based on formal specifications, for functional testing of object-oriented software. This method is based on rigorous theoretical foundations. To limit the size of test sets, we present several hypotheses specific to object-oriented systems, used to reduce an exhaustive test set into a pertinent test set while preserving its qualities: regularity hypotheses are used to constrain the shape of tests while uniformities, with or without subdomain decomposition, are used to limit the values of the variables appearing in the tests. Care is taken to provide a method for which operational methods can be implemented

Integrating the Unified Modeling Language with an Architecture Description Language

These paper presents an approach for integrating UML with an ADL. The integration would encompass the advantages of both languages. It would give formal semantics to UML constructs and thus would provide UML with a theoretical foundation for architecture modeling. Furthermore, the integration would provide benefits for both ADL and UML users: it will enable ADL users to utilize general-purpose UML tools, and will enable UML users to utilize ADL validation capabilities. The result would be a rigorous software development process that is currently lacking