Inferring Concise Specifications of APIs

Modern software relies on libraries and uses them via application programming interfaces (APIs). Correct API usage as well as many software engineering tasks are enabled when APIs have formal specifications. In this work, we analyze the implementation of each method in an API to infer a formal postcondition. Conventional wisdom is that, if one has preconditions, then one can use the strongest postcondition predicate transformer (SP) to infer postconditions. However, SP yields postconditions that are exponentially large, which makes them difficult to use, either by humans or by tools. Our key idea is an algorithm that converts such exponentially large specifications into a form that is more concise and thus more usable. This is done by leveraging the structure of the specifications that result from the use of SP. We applied our technique to infer postconditions for over 2,300 methods in seven popular Java libraries. Our technique was able to infer specifications for 75.7% of these methods, each of which was verified using an Extended Static Checker. We also found that 84.6% of resulting specifications were less than 1/4 page (20 lines) in length. Our technique was able to reduce the length of SMT proofs needed for verifying implementations by 76.7% and reduced prover execution time by 26.7%

Automatic Functional Testing of GUIs

Functional testing of GUIs can be automated using a test oracle derived from the GUI’s specification and from a restricted set of randomised test data. As test data, a set of randomly distorted test objects seems to work well, especially starting as we do with a ‘perfect’ object and then distorting this more and more as the test progresses. The number of test cases needed seems to be much smaller than that reported in other random testing papers. More work is needed to see if the approach is generally applicable: if so, the test engineer can spend his time writing GUI specifications at a high level of abstraction, rather than hand-generating test cases

Grey-box Testing and Verification of Java/JML

International audienceWe present in this paper the application of constraint solving techniques to the validation and automated test cases generation for Java programs, annotated with JML specifications. The Java/JML code is translated into a constraint representation based on a subset of the set-theory, which is well-suited for modelling object-oriented programs. Symbolic code execution techniques can then be applied to produce test cases, using classical structural test selection criteria, or to detect possible runtime errors, and non-conformances between the Java code and its embedded JML model

Combining over- and under-approximating program analyses for automatic software testing

This dissertation attacks the well-known problem of path-imprecision in static program analysis. Our starting point is an existing static program analysis that over-approximates the execution paths of the analyzed program. We then make this over-approximating program analysis more precise for automatic testing in an object-oriented programming language. We achieve this by combining the over-approximating program analysis with usage-observing and under-approximating analyses. More specifically, we make the following contributions. We present a technique to eliminate language-level unsound bug warnings produced by an execution-path-over-approximating analysis for object-oriented programs that is based on the weakest precondition calculus. Our technique post-processes the results of the over-approximating analysis by solving the produced constraint systems and generating and executing concrete test-cases that satisfy the given constraint systems. Only test-cases that confirm the results of the over-approximating static analysis are presented to the user. This technique has the important side-benefit of making the results of a weakest-precondition based static analysis easier to understand for human consumers. We show examples from our experiments that visually demonstrate the difference between hundreds of complicated constraints and a simple corresponding JUnit test-case. Besides eliminating language-level unsound bug warnings, we present an additional technique that also addresses user-level unsound bug warnings. This technique pre-processes the testee with a dynamic analysis that takes advantage of actual user data. It annotates the testee with the knowledge obtained from this pre-processing step and thereby provides guidance for the over-approximating analysis. We also present an improvement to dynamic invariant detection for object-oriented programming languages. Previous approaches do not take behavioral subtyping into account and therefore may produce inconsistent results, which can throw off automated analyses such as the ones we are performing for bug-finding. Finally, we address the problem of unwanted dependencies between test-cases caused by global state. We present two techniques for efficiently re-initializing global state between test-case executions and discuss their trade-offs. We have implemented the above techniques in the JCrasher, Check 'n' Crash, and DSD-Crasher tools and present initial experience in using them for automated bug finding in real-world Java programs.Ph.D.Committee Chair: Smaragdakis, Yannis; Committee Member: Dwyer, Matthew; Committee Member: Orso, Alessandro; Committee Member: Pande, Santosh; Committee Member: Rugaber, Spence

Theories in Practice: Easy-to-Write Specifications that Catch Bugs

Automated testing during development helps ensure that software works according to the test suite. Traditional test suites verify a few well-picked scenarios or example inputs. However, such example-based testing does not uncover errors in legal inputs that the test writer overlooked. We propose theory-based testing as an adjunct to example-based testing. A theory generalizes a (possibly infinite) set of example-based tests. A theory is an assertion that should be true for any data, and it can be exercised by human-chosen data or by automatic data generation. A theory is expressed in an ordinary programming language, it is easy for developers to use (often even easier than example-based testing), and it serves as a lightweight form of specification. Six case studies demonstrate the utility of theories that generalize existing tests to prevent bugs, clarify intentions, and reveal design problems

The AutoProof Verifier: Usability by Non-Experts and on Standard Code

Formal verification tools are often developed by experts for experts; as a result, their usability by programmers with little formal methods experience may be severely limited. In this paper, we discuss this general phenomenon with reference to AutoProof: a tool that can verify the full functional correctness of object-oriented software. In particular, we present our experiences of using AutoProof in two contrasting contexts representative of non-expert usage. First, we discuss its usability by students in a graduate course on software verification, who were tasked with verifying implementations of various sorting algorithms. Second, we evaluate its usability in verifying code developed for programming assignments of an undergraduate course. The first scenario represents usability by serious non-experts; the second represents usability on "standard code", developed without full functional verification in mind. We report our experiences and lessons learnt, from which we derive some general suggestions for furthering the development of verification tools with respect to improving their usability.Comment: In Proceedings F-IDE 2015, arXiv:1508.0338

Praspel: Contract-Driven Testing for PHP using Realistic Domains

We present an integrated contract-based testing framework for PHP. It relies on a behavioral interface specification language called Praspel, for "PHP Realistic Annotation and Specification Language". Using Praspel developers can easily annotate their PHP scripts with formal contracts, namely class invariants, and method pre- and postconditions. These contracts describe assertions either by predicates or by assigning realistic domains to data. Realistic domains introduce types in PHP and describe complex structures frequently encountered in applications, such as email addresses or SQL queries. Realistic domains display two properties: predicability, which allows to check if a data belongs to a given realistic domain, and samplability, which allows to generate valid data. This paper introduces coverage criteria dedicated to contracts, designed to exhibit relevant behaviors of the annotated methods. Test data are then computed to satisfy these coverage criteria, by using dedicated data generators for complex realistic domains, such as arrays or strings. This framework has been implemented and disseminated within the PHP community, which gave us feedback on their usage of the tool and the relevance of this integrated process with respect to their practice of manual testing

Code generation for event-B

Stepwise refinement and Design-by-Contract are two formal approaches for modelling systems. These approaches are widely used in the development of systems. Both approaches have (dis-)advantages: in stepwise refinement a model starts with an abstraction of the system and more details are added through refinements. Each refinement must be provably consistent with the previous one. Hence, reasoning about abstract models is possible. A high level of expertise is necessary in mathematics to have a good command of the underlying languages, techniques and tools, making this approach less popular. Design-by-Contract, on the other hand, works on the program rather than the program model, so developers in the software industry are more likely to have expertise in it. However, the benefit of reasoning over more abstract models is lost. A question arises: is it possible to combine both approaches in the development of systems, providing the user with the benefits of both? This thesis answers this question by translating the stepwise refinement method with EventB to Design-by-Contract with Java and JML, so users can take full advantage of both formal approaches without losing their benefits. This thesis presents a set of syntactic rules that translates Event-B to JML-annotated Java code. It also presents the implementation of the syntactic rules as the EventB2Java tool. We used EventB2Java to translate several Event-B models. The tool generated JML-annotated Java code for all the considered Event-B models that serve as final implementation. We also used EventB2Java for the development of two software applications. Additionally, we compared EventB2Java against two other tools that also generate Java code from Event-B models. EventB2Java enables users to start the software development process in Event-B, where users can model the system and prove its consistency, to then transition to JML-annotated Java code, where users can continue the development process