We introduce a generic component-based design of a run-time checker,

identify its components and their requirements, and evaluate

existing state of the art tools instantiating each component

Boer, F.S. (Frank) de

Gouw, C.P.T. (Stijn) de

English

We introduce a generic component-based design of a run-time checker,
\nidentify its components and their requirements, and evaluate
\nexisting state of the art tools instantiating each component

CWI's Institutional Repository

Run-Time Verification of Black-BoxComponents using Behavioral Specifications:An Experience Report on Tool Development ?Frank S. de Boer1,2 and Stijn de Gouw1,21 CWI, Amsterdam, The Netherlands2 Leiden University, The NetherlandsAbstract. We introduce a generic component-based design of a run-time checker, identify its components and their requirements, and eval-uate existing state of the art tools instantiating each component.1 IntroductionRun-time assertion checking is one of the most useful techniques for detectingfaults, and can be applied during any program execution context, includingdebugging, testing, and production [3]. Compared to program logics, assertionchecking emphasizes executable specifications. Whereas program logics staticallycover all possible execution paths, run-time assertion checking is fully automated,and applies on demand to the actual runs of the program.By their very nature, assertions are state-based in that they describe proper-ties of the program variables (fields of classes and local variables of methods). Ingeneral, assertions expressed in languages supporting design by contract (like theJava Modeling Language (JML) [1]) cannot be used to specify the interactionprotocol between objects or components, in contrast to other formalisms suchas message sequence charts and UML sequence diagrams. Nor can state-basedassertions be used to specify component interfaces since such interfaces do nothave a state3.This paper reports on an integrated tool environment which provides asmooth integration of the specification and run-time checking of both data- andprotocol-oriented properties of component interfaces. The basic idea underlyingour framework is the representation of message sequences as words of a languagegenerated by a grammar. The formalism of attribute grammars allows the high-level specification of user-defined abstractions of message sequences in termsof attributes of grammars describing these sequences. We introduce a generic? This research is partly funded by the EU project FP7-231620 HATS:Highly Adaptable and Trustworthy Software using Formal Models(http://www.hats-project.eu/)3 JML uses model variables for interface specifications. However, a separate representsclause is needed for a full specification, and such clauses can only be defined oncean implementation has been given (and is not implementation independent).2component-based design which supports run-time checking of assertions aboutthese attributes, which involves parsing the generated sequences of messages. Weidentify the components and their requirements, and evaluate existing state ofthe art tools which instantiate the components of the generic tool architecture.Related Work. A preliminary version describing a prototype of an instantiationof our tool architecture was presented at the workshop “Formal Techniques forJava-Like Programs 2010” and appeared in its informal proceedings4. This pro-totype was based on state of the art tools. However, for industrial usage we needa component-based design (as described above), and an experience report onvarious instantiations of the generic tool design.There exist many other interesting approaches to run-time verification andmonitoring of message sequences which however do not address its integrationwith the general context of run-time assertion checking, e.g. JML: PQL, Trace-matches, JmSeq, LARVA, Jass and JavaMOP. Due to space limitations we donot further discuss these approaches individually.2 The Modeling FrameworkAbstracting from implementation details (such as field values of objects), anexecution of a Java program can be represented by its global communicationhistory : the sequence of messages corresponding to the invocation and comple-tion of (possibly static) methods. Similarly, the execution of a single object canbe represented by its local communication history, which consists of all messagessent and received by that object. The behavior of a program (or object) can thenbe defined as the set of its allowed histories. Whether a history is allowed de-pends in general both on data (the contents of the messages, e.g. parameter andreturn values of method calls) and protocol (the order between messages). Thequestion arises how such allowed sets of histories can be defined conveniently. Inthis section we show how attribute grammars provide a powerful and declarativeway to define such sets. We will use the interface of the Java BufferedReader(Figure 1) as a running example to explain the basic modeling concepts.interface BufferedReader {void close();int read();}Fig. 1. Relevant methods of theBufferedReader InterfaceS ::= open C1 assert open.caller != null==> open.caller == C1.caller;| C ::= read C1 C.caller = C1.caller;| close S1 C.caller = close.caller;|  C.caller = null;Fig. 2. Extended Attribute Grammar mod-eling the behavior of a BufferedReaderTo each method m in the interface we associate two communication events:‘call-m’ and ‘return-m’. The observable communication history of an object of a4 Available in the ACM Digital Library with the title “Prototyping a tool environmentfor run-time assertion checking in JML with communication histories”, authored byFrank S. de Boer, Stijn de Gouw and Jurgen Vinju3class implementing the above interface consists of sequences of communicationevents.Context-free grammars provide a declarative way to define the allowed histo-ries of an object. The context-free grammar underlying the attribute grammarin Figure 2 generates the valid histories for BufferedReader, describing theprefix closure of sequences of the terminals call-BufferedReader’, ‘call-read’ and‘call-close’ as given by the regular expression (call-BufferedReader call-read*call-close). Note that since grammars specify invariant properties of the ongo-ing behavior of an object, they must be prefix-closed. In general, communicationevents form the terminal symbols of the grammar, and non-terminal symbolsspecify the valid sequences of communication events.While context-free grammars provide a convenient way to specify the pro-tocol structure of the valid histories, they do not take data such as parametersand return values of method calls and returns into account. Thus the questionarises how to specify the data-flow of the valid histories. To that end, we extendthe grammar with attributes. A terminal symbol ‘call-m’ has built-in attributes‘caller’, ‘callee’ and the parameter names for respectively the actual parametersand object identities of the caller and callee. A terminal ‘return-m’ addition-ally has an attribute result referring to the return value. Non-terminals haveuser-defined attributes to define data properties of sequences of events. Howeverthe attributes themselves do not alter the language generated by the attributegrammar, they only define properties of data-flow of the history. We extend theattribute grammar with assertions to specify properties of attributes. For exam-ple, in the attribute grammar in Figure 2 a user-defined attribute ‘caller’ for thenon-terminal ‘C’ is defined storing the identity of the object which closed theBufferedReader (and is null if the reader was not closed yet). The assertionallows only those histories in which the object which opened (created) the readeralso closed it.Assertions can be placed at any position in a production rule and are eval-uated there. Note that assertions appearing directly before a terminal can beseen as a precondition of a terminal, whereas post-conditions are placed directlyafter the terminal. This is in fact a generalization of traditional pre- and post-conditions for methods as used in design-by-contract: a single terminal ‘call-m’can appear in multiple productions, each of which followed by a different asser-tion. Hence different preconditions (or postconditions) can be used for the samemethod, depending on the context (grammar production) in which the call wasmade.3 Generic Tool ArchitectureGiven a Java interface specified with an attribute grammar, we would like to testwhether an object implementing the interface satisfies the properties defined inthe grammar at every point in its lifetime. In this section we describe a generictool architecture which achieves this. Four different components are combined:a state-based assertion checker, a parser generator, a debugger and a generaltool for meta-programming. Traditionally these tools are used for very diverse4purposes and don’t need to interact with each other. We therefore investigaterequirements needed to achieve a seamless integration of these components, mo-tivated by describing the workflow of the run-time checker.Suppose that during execution of a Java program, a method of a class (sub-sequently referred to as CUT, the ‘class under test’) which implements an inter-face specified by an attribute grammar is called. The new history of the objecton which the method was called should be updated to reflect the addition ofthe method call. To represent the history of an object of CUT, the Meta-Programming tool generates for each method m in CUT two classes call-mand return-m. These classes contain the following fields: the object identitityof the callee, the identity of the caller and the actual parameters. Additionallyreturn-m contains a field result containing the return value. A Java List con-taining instances of call-m and return-m then stores the history of an objectof CUT.Fig. 3. Generic Tool ArchitectureThe meta-programming tool furthergenerates code for a wrapper class whichreplaces the original main class. Thiswrapper class contains a field H, a Javamap containing pairs (id, h) of an ob-ject identity id and its local history h.The new main class executes the origi-nal program inside the Debugger. TheDebugger is responsible for monitoringexecution of the program. It must be ca-pable of temporarily ‘pausing’ the pro-gram whenever a call or return occurs,and execute user-defined code to updateH appropriately . Moreover the Debug-ger must be able to read the identity ofthe callee, caller and parameters/return-value.After the history is updated the run-time checker must decide whether it stillsatisfies the specification (the attribute grammar). Observe that a communica-tion history can be seen as a sequence of tokens (in our setting: communicationevents). Since the attribute grammar together with the assertions generate thelanguage of all valid histories, checking whether a history satisfies the specifica-tion reduces to deciding whether the history can be parsed by a parser for theattribute grammar, where moreover during parsing the assertions must evaluateto true. Therefore the Parser Generator creates a parser for the given attributegrammar. Since the history is a heterogenous list of call-m and return-m ob-jects, the parser must support parsing streams of tokens with user-defined types.Assertions in general describe properties of Java objects, and the grammar con-tains assertions over attributes, the attributes must be normal Java variables.Consequently the parser generator must allow arbitrary user-defined java code(to set the attribute value) in rule actions. The use of Java code ensures theattribute values are computable. Since assertions are allowed in-between any5two (non)-terminals, the parser generator should support user-defined actionsbetween arbitrary grammar symbols. At run-time, the parser is triggered when-ever the history of an object is updated. The result is either a parse error,which indicates that the current communication history has violated the pro-tocol structure specified by the attribute grammar, or a parse tree with newattribute values. During parsing, the Assertion Checker evaluates the asser-tions in the grammar on the newly computed attribute values. To avoid parsingthe whole history of a given object each time a new call or return is appended,ideally the parser should support incremental parsing [4]. An incremental parsercomputes a parse tree for the new history based on the parse trees for prefixes ofthe history. In our setting, the attribute grammar specifies invariant propertiesof the ongoing behavior. Hence the parser constructs a new parse tree after eachcall/return, consequently parse trees for all prefixes of the current history canbe exploited for incremental parsing.4 Instantiating the Generic Tool ArchitectureThe previous section introduced the generic tool architecture, which was basedon four different components: meta-programming, debugger, parser generatorand state-based run-time assertion checker. Here we instantiate these four com-ponents with particular (state of the art) tools, and report our experiences.Rascal [5] is a powerful tool-supported meta-programming language tailoredfor program analysis, program transformation and code generation. We wrote aRascal program of approximately 600 lines in total which generates the classescall-m, return-m, the new main class, and glue code to trigger the debuggerand parser. Rascal is still in an alpha stage, it is not fully backwards compatibleand we discovered numerous bugs in Rascal during development of the Rascalprogram. However overall our experience was quite positive. All bugs were fixedquickly by the Rascal team, and its powerful parsing, pattern matching andtransforming concrete syntax features proved indispensable.We evaluated Sun’s implementation of the Java Debugging Interface for thedebugger component. It is part of the standard Java Development Kit, hencemaintenance of the debugger is practically guaranteed. The Sun debugger startsthe original user program in a virtual machine which is monitored for occurencesof MethodEntryEvent (method calls) and MethodExitEvent (method returns).It allows defining event handlers which are executed whenever such events occur.It also allows retrieving the caller, callee, parameters values and return value ofevents using StackFrames. The Sun debugger meets all requirements for thedebugger stated above. As the main disadvantage, we found that the currentimplementation of the debugger is very slow. In fact it was responsible for themajority of the overhead of the run-time checker. This is not necessarily prob-lematic: as testing is done during development, the debugger will typically not bepresent in performance critical production code. Moreover, one usually wants totest only up to a certain bound (for instance, in time, or in the number of events),and report on results once the bound is exceeded. Nonetheless, for testing up tohuge bounds, a different implementation for the debugger is needed.6We instantiated the parser generator component with ANTLR, a state ofthe art parser generator. It generates fast recursive descent parsers for Javaand allows grammar actions and custom token streams. It even supports con-ditional productions: such productions are only chosen during parsing wheneveran associated Boolean expression (the condition) is true. Attribute grammarswith conditional productions express protocols that depend on data which aretypically not context-free. ANTLR can only handle LL(*) grammars5, and itlacks support for incremental parsing, though this is planned by the ANTLRdevelopers. We could not find any Java parser generator which supports generalcontext-free grammars and incremental parsing of attribute grammars.We tested two state-based assertion languages: standard Java assertions andthe Java Modeling Language (JML). Both languages suffice for our purposes.JML is far more expressive than the standard Java assertions, though its toolsupport is not ready for industrial usage. In particular, the last stable version ofthe JML run-time assertion checker dates back over 8 years, when for instancegenerics were not supported yet. The main reason is that JML’s run-time asser-tion checker only works with a proprietary implementation of the Java compiler,and unsurprisingly it is costly to update the proprietary compiler each time thestandard compiler is updated. This problem is recognized by the JML develop-ers [2]. OpenJML, a new pre-alpha version of the JML run-time assertion checkerintegrates into the standard Java compiler, and initial tests with it providedmany valuable input for real industrial size applications. See the Sourceforgetracker for the kind of issues we have encountered when using OpenJML.A (variant of) the above tool suite can be obtained from http://www.cwi.nl/~cdegouw. It was applied successfully to an industrial size case study of theeCommerce software company Fredhopper.References1. L. Burdy, Y. Cheon, D. R. Cok, M. D. Ernst, J. R. Kiniry, G. T. Leavens, K. R. M.Leino, and E. Poll. An overview of JML tools and applications. InternationalJournal on Software Tools for Technology Transfer, 7(3):212–232, 2005.2. P. Chalin, P. R. James, and G. Karabotsos. Jml4: Towards an industrial grade ivefor java and next generation research platform for jml. In VSTTE, pages 70–83,2008.3. L. A. Clarke and D. S. Rosenblum. A historical perspective on runtime assertionchecking in software development. ACM SIGSOFT Software Engineering Notes,31(3):25–37, 2006.4. G. Hedin. Incremental attribute evaluation with side-effects. In D. Hammer, editor,Compiler Compilers and High Speed Compilation, 2nd CCHSC Workshop, BerlinGDR, October 10-14, 1988, Proceedings, volume 371 of Lecture Notes in ComputerScience, pages 175–189. Springer, 1988.5. P. Klint, T. van der Storm, and J. Vinju. Rascal: a domain specific language forsource code analysis and manipulation. In A. Walenstein and S. Schupp, editors,SCAM 2009, pages 168–177, 2009.5 A strict subset of the context-free grammars. Left-recursive grammars are not LL(*).

Run-Time Verification of Black-Box Components Using Behavioral Specifications: An Experience Report on Tool Development

Abstract

Similar works

Full text

Available Versions

CWI's Institutional Repository