197 research outputs found
Property-Based Testing - The ProTest Project
The ProTest project is an FP7 STREP on property based testing. The purpose of the project is to develop software engineering approaches to improve reliability of service-oriented networks; support fault-finding and diagnosis based on specified properties of the system. And to do so we will build automated tools that will generate and run tests, monitor execution at run-time, and log events for analysis.
The Erlang / Open Telecom Platform has been chosen as our initial implementation vehicle due to its robustness and reliability within the telecoms sector. It is noted for its success in the ATM telecoms switches by Ericsson, one of the project partners, as well as for multiple other uses such as in facebook, yahoo etc. In this paper we provide an overview of the project goals, as well as detailing initial progress in developing property based testing techniques and tools for the concurrent functional programming language Erlang
Simplifying Contract-Violating Traces
Contract conformance is hard to determine statically, prior to the deployment
of large pieces of software. A scalable alternative is to monitor for contract
violations post-deployment: once a violation is detected, the trace
characterising the offending execution is analysed to pinpoint the source of
the offence. A major drawback with this technique is that, often, contract
violations take time to surface, resulting in long traces that are hard to
analyse. This paper proposes a methodology together with an accompanying tool
for simplifying traces and assisting contract-violation debugging.Comment: In Proceedings FLACOS 2012, arXiv:1209.169
Finding The Lazy Programmer's Bugs
Traditionally developers and testers created huge numbers of explicit tests, enumerating interesting cases, perhaps
biased by what they believe to be the current boundary conditions of the function being tested. Or at
least, they were supposed to.
A major step forward was the development of property testing. Property testing requires the user to write a few
functional properties that are used to generate tests, and requires an external library or tool to create test data
for the tests. As such many thousands of tests can be created for a single property. For the purely functional
programming language Haskell there are several such libraries; for example QuickCheck [CH00], SmallCheck
and Lazy SmallCheck [RNL08].
Unfortunately, property testing still requires the user to write explicit tests. Fortunately, we note there are
already many implicit tests present in programs. Developers may throw assertion errors, or the compiler may
silently insert runtime exceptions for incomplete pattern matches.
We attempt to automate the testing process using these implicit tests. Our contributions are in four main
areas: (1) We have developed algorithms to automatically infer appropriate constructors and functions needed
to generate test data without requiring additional programmer work or annotations. (2) To combine the
constructors and functions into test expressions we take advantage of Haskell's lazy evaluation semantics by
applying the techniques of needed narrowing and lazy instantiation to guide generation. (3) We keep the type
of test data at its most general, in order to prevent committing too early to monomorphic types that cause
needless wasted tests. (4) We have developed novel ways of creating Haskell case expressions to inspect elements
inside returned data structures, in order to discover exceptions that may be hidden by laziness, and to make
our test data generation algorithm more expressive.
In order to validate our claims, we have implemented these techniques in Irulan, a fully automatic tool for
generating systematic black-box unit tests for Haskell library code. We have designed Irulan to generate high
coverage test suites and detect common programming errors in the process
Towards a theory of reach
When testing a program, there are usually some parts that are rarely executed and hence more difficult to test. Finding inputs that guarantee that such parts are executed is an example of a reach problem, which in general seeks to ensure that targeted parts of a program are always executed. In previous work, Naylor and Runciman have developed a reachability solver for Haskell, based on the use of lazy narrowing from functional logic programming. Their work was focused on practical issues concerning implementation and performance. In this paper, we lay the groundwork for an underlying theory of such a system, by formally establishing the correctness of a simple reach solver
Random Testing For Language Design
Property-based random testing can facilitate formal verification, exposing errors early on in the proving process and guiding users towards correct specifications and implementations. However, effective random testing often requires users to write custom generators for well-distributed random data satisfying complex logical predicates, a task which can be tedious and error prone.
In this work, I aim to reduce the cost of property-based testing by making such generators easier to write, read and maintain. I present a domain-specific language, called Luck, in which generators are conveniently expressed by decorating predicates with lightweight annotations to control both the distribution of generated values and the amount of constraint solving that happens before each variable is instantiated.
I also aim to increase the applicability of testing to formal verification by bringing advanced random testing techniques to the Coq proof assistant. I describe QuickChick, a QuickCheck clone for Coq, and improve it by incorporating ideas explored in the context of Luck
to automatically derive provably correct generators for data constrained by inductive relations.
Finally, I evaluate both QuickChick and Luck in a variety of complex case studies from programming languages literature, such as information-flow abstract machines and type systems for lambda calculi
αCheck: a mechanized metatheory model-checker
The problem of mechanically formalizing and proving metatheoretic properties
of programming language calculi, type systems, operational semantics, and
related formal systems has received considerable attention recently. However,
the dual problem of searching for errors in such formalizations has attracted
comparatively little attention. In this article, we present Check, a
bounded model-checker for metatheoretic properties of formal systems specified
using nominal logic. In contrast to the current state of the art for metatheory
verification, our approach is fully automatic, does not require expertise in
theorem proving on the part of the user, and produces counterexamples in the
case that a flaw is detected. We present two implementations of this technique,
one based on negation-as-failure and one based on negation elimination, along
with experimental results showing that these techniques are fast enough to be
used interactively to debug systems as they are developed.Comment: Under consideration for publication in Theory and Practice of Logic
Programming (TPLP
Improving the Network Scalability of Erlang
As the number of cores grows in commodity architectures so does the likelihood of failures. A distributed actor model potentially facilitates the development of reliable and scalable software on these architectures. Key components include lightweight processes which ‘share nothing’ and hence can fail independently. Erlang is not only increasingly widely used, but the underlying actor model has been a beacon for programming language design, influencing for example Scala, Clojure and Cloud Haskell.
While the Erlang distributed actor model is inherently scalable, we demonstrate that it is limited by some pragmatic factors. We address two network scalability issues here: globally registered process names must be updated on every node (virtual machine) in the system, and any Erlang nodes that communicate maintain an active connection. That is, there is a fully connected O(n2) network of n nodes.
We present the design, implementation, and initial evaluation of a conservative extension of Erlang — Scalable Distributed (SD) Erlang. SD Erlang partitions the global namespace and connection network using s_groups. An s_group is a set of nodes with its own process namespace and with a fully connected network within the s_group, but only individual connections outside it. As a node may belong to more than one s_group it is possible to construct arbitrary connection topologies like trees or rings.
We present an operational semantics for the s_group functions, and outline the validation of conformance between the implementation and the semantics using the QuickCheck automatic testing tool. Our preliminary evaluation in comparison with distributed Erlang shows that SD Erlang dramatically improves network scalability even if the number of global operations is tiny (0.01%). Moreover, even in the absence of global operations the reduced connection maintenance overheads mean that SD Erlang scales better beyond 80 nodes (1920 cores)
- …