Belinfante, Axel

Brinksma, Hendrik

de Vries, R.G.

Feenstra, J.

Heemstra de Groot, S.M.

Tretmans, G.J.

English

NARCIS 

Côte de Resyste -- Automatic Model-Based Testing of Communication Protocols

Brinksma, Ed

Feenstra, Jan

Tretmans, Jan

de Vries, René

University of Twente Research Information

Côte de ResysteAutomatic model-based testing of communication protocolsAxel Belinfante, Ed Brinksma, Jan Feenstra,Jan Tretmans and Ren´ de VriesUniversity of TwenteFormal Methods and Tools group, Department of Computer ScienceP.O. Box 217, 7500 AE Enschede, The Netherlands{belinfan, brinksma, feenstra, tretmans, rdevries}@cs.utwente.nl1. Software TestingSoftware quality is an issue that currently attracts a lot ofattention. Software invades everywhere in our society andlife and we are increasingly dependent on it. Moreover, thecomplexity of software is still growing. This also appliesto software in mobile systems. Consequently, the quality,functional correctness and reliability of software is an issueof increasing importance and growing concern. Systematictesting of software plays an important rôle in the quest forimproved quality. Despite its importance, testing is oftenan under-exposed phase in the software development pro-cess. Moreover, testing turns out to be expensive, difficultand problematic. One source of problems can be an im-precise or ambiguous specification, so that a good basis fortesting is missing. Another reason is the usually manualand laborious testing process without effective automation,so that testing is error-prone and consumes many resources.Furthermore, quite often the testing phase gets jammed be-tween moving code delivery dates and fixed custom deliv-ery dates. On the other hand, research and development intesting have been rather immature. Testing methodology ismostly governed by heuristics. Fortunately, this situation isgradually improving. Triggered by the quest for improvedquality and imposed by increased product liability, testingis considered more important and treated more seriously.Being a software tester is becoming a true profession.2. Côte de ResysteThe projectCôte de Resyste(COnformance TEsting OFREactive SYSTEms) aims at improving the testing processby using formal methods. InCôte de Resystewe developtheories, methods and tools to enable fully automatic test-ing of software systems based on formal specifications. Indoing so,Côte de Resysteconcentrates on functional testingof reactive systems. Functional testing involves checkingthe correct behaviour of a system: does the system do whatit should do (as opposed to, e.g., testing the performance orrobustness). Reactive systems are mostly technical, event-driven systems in which stimulus/response behaviour is im-portant, such as embedded systems, communication proto-cols and process control software. Administrative systemsare typically not reactive systems.Côte de Resysteis a cooperation between Philips Re-search Laboratories Eindhoven, Lucent Technologies R&DCentre Enschede, Eindhoven University of Technology andthe University of Twente. It is a 4 year, 23 man-year projectsupported by the Dutch Technology Foundation STW [2, 4].Our main challenge is to develop test techniques and toolswith a high practical applicability, while starting from awell-defined and sound theoretical basis. The applicabilityand usability is evaluated by performing case studies sup-plied by Philips, Lucent and associated partners.3. Formal MethodsCurrently, most system specifications are written in nat-ural languages, such as English or Dutch. Although suchinformal specifications are easily accessible, they are oftenincomplete and liable to different and possibly inconsistentinterpretations. Such ambiguities are not a good basis fortesting: if it is not clear what a system shall do, it is difficultto test whether it does what it should do. With formal meth-ods systems are specified and modelled by applying tech-niques from mathematics and logic. Such formal specifica-tions and models have a precise, unambiguous semantics,which enables the analysis of systems and the reasoningabout them with mathematical precision and rigour. More-over, formal languages are more easily amenable to auto-matic processing by means of tools. For example, tools ex-ist that are able to verify fully automatically the absence ofdeadlock based on a formal description of the design. Untilrecently formal methods were a merely academic topic, butnow their use in industrial software development is increas-ing, in particular for safety critical systems and for telecom-munication software. One particular Dutch project whereformal methods have been used successfully is the controlsystem for the storm surge barrier in the Nieuwe Waterweg.Figure 1. The storm surge barrier in the Nieuwe Wa-terweg near Hoek van Holland is completely softwarecontrolled. Flooding of Rotterdam is avoided andmust be reliable. During the system design the formalmethodsZ and PROMELA have been used. Testingwas performed based on these models.4. Testing with Formal MethodsA formal specification is a precise, complete, consistentand unambiguous basis for design and code developmentas well as for testing. This is a first big advantage in con-trast with traditional testing processes where such a test ba-sis is often lacking. A second advantage of the use of for-mal specifications for testing is their suitability to automaticprocessing by means of tools. Algorithms have been de-veloped which derive tests from a formal specification [3].These algorithms have their theoretical underpinning in thetheories of labelled transition systems, process algebra andtesting preorders. Moreover, these algorithms have beenimplemented in tools leading to automatic, faster and lesserror-prone test generation. This opens the way towardscompletely automatic testing where the system under testand its formal specification are the only required prerequi-sites. Formal methods provide a rigorous and sound basisfor algorithmic and automatic generation of tests. Tests canbe formally proved to be valid, i.e., they test what should betested, and only that.5. TORX: A Tool for Formal TestingWithin Côte de Resystewe are developing the formaltesting tool TORX. TORX integrates automatic test gener-ation and automatic test execution in ano -the-flymanner.On-the-fly testing implies that derived test actions are im-mediately executed and, moreover, that only the part of thetest that will actually be executed is derived ‘lazy test gener-ation’. TORX is a prototype tool with which some academicand industrial case studies have been successfully tested.One of the examples was a chat-box protocol, which wastested based on specifications in the formal languages LO-TOS and Promela [1]. Due to the on-the-fly testing methodlong tests consisting of more than 500,000 test events couldbe generated and executed completely automatically. Froma set of 28 mutants of the chat-box protocol, with carefullyinserted faults, all erroneous implementations could be suc-cessfully detected which could not be achieved with tradi-tional testing tools.the stimulusverifythe responsethe stimulusobservethe responseselectTorX UnderTestSystemspecificationsendFigure 2. The TORX tool selects stimuli from thespecification and stimulates the system under test.The responses are observed and verified against thespecification.6. Applications of Testing with FormalMethodsCurrently, TORX and its accompanying test methods areevaluated by applying them to different case studies. One ofthe evaluation studies concerns a communication protocolbetween video recorders and television sets for download-ing channel presets. The results of this study are promis-ing: some faults were detected which had slipped throughthe conventional testing procedures. The second case studyis Lucent Technologies V5.1 access network protocol. Athird case study is a mobile application in the context of the‘Rekeningrijden’ project. Interpay is developing a systemfor automated fee charging for turnpike roads. The systemshould debit an electronic purse and register a balance incre-ment on an account at the toll gate when a vehicle passes.To ensure that the payment transactions are correctly pro-cessed by the system, the system should be tested. Thistesting activity should increase the confidence in the correct2functioning of the system. Due to the fact that many carscan pass a toll gate simultaneously, the number of paralleltransactions in progress can be large. This fact contributesto the complexity of the generation, execution and analysisof test experiments. The testing activity done byCôte deResysteand Interpay is scheduled in the beginning of 2001.7. PerspectivesCurrent work concentrates on improving TORX, on de-veloping methods for effective selection of test sets andon generic test execution environments. The case stud-ies strengthen our view that within a few years it will bepossible to perform automatic testing of reactive, modestlycomplex, industrial software systems based on their formalspecifications. The expectation is that the extra effort re-quired for developing the necessary formal system speci-fications will be more than compensated by faster, cheaperand more effective testing. This does not mean that all prob-lems have been completely solved, yet. One of the most im-portant research questions, which is currently investigated,is how the completeness and coverage of an automaticallygenerated test suite can be expressed, measured and, ulti-mately, controlled. Even more intriguing is the questionhow test suite coverage can be related to a measure of prod-uct quality. After all, product quality is the only actual rea-son to perform testing.References[1] A. Belinfante, J. Feenstra, R. d. Vries, J. Tretmans, N. Goga,L. Feijs, S. Mauw, and L. Heerink. Formal test automation:A simple experiment. In G. Csopaki, S. Dibuz, and K. Tar-nay, editors,12th Int. Workshop on Testing of CommunicatingSystems, pages 179–196. Kluwer Academic Publishers, 1999.[2] Dutch Technology Foundation STW.Côte de Resyste–COnformance TEsting of REactive SYSTEms. Projectproposal STW TIF.4111, University of Twente, Eind-hoven University of Technology, Philips Research Labo-ratories, KPN Research, Utrecht, The Netherlands, 1996.http://fmt.cs.utwente.nl/CdR .[3] J. Tretmans. Test generation with inputs, outputs and repeti-tive quiescence.Software—Concepts and Tools, 17(3):103–120, 1996. Also: Technical Report No. 96-26, Centrefor Telematics and Information Technology, University ofTwente, The Netherlands.[4] R. d. Vries, J. Tretmans, A. Belinfante, J. Feenstra, L. Feijs,S. Mauw, N. Goga, L. Heerink, and A. d. Heer. Côtede Resyste in PROGRESS. In S. T. Foundation, editor,PROGRESS2000 – Workshop on Embedded Systems, pages157–164, Utrecht, The Netherlands, October 13 2000.3

Côte de Resyste:Automatic Model-Based Testing of Communication Protocols

Côte de Resyste -- Automatic Model-Based Testing of Communication Protocols

Abstract

Similar works

Full text

Available Versions

NARCIS

University of Twente Research Information