Conjectures, tests and proofs: An overview of theory exploration

A key component of mathematical reasoning is the ability to formulate interesting conjectures about a problem domain at hand. In this paper, we give a brief overview of a theory exploration system called QuickSpec, which is able to automatically discover interesting conjectures about a given set of functions. QuickSpec works by interleaving term generation with random testing to form candidate conjectures. This is made tractable by starting from small sizes and ensuring that only terms that are irreducible with respect to already discovered conjectures are considered. QuickSpec has been successfully applied to generate lemmas for automated inductive theorem proving as well as to generate specifications of functional programs. We give an overview of typical use-cases of QuickSpec, as well as demonstrating how to easily connect it to a theorem prover of the user’s choice

A generic synthesisable test bench

Writing test benches is one of the most frequently-performed tasks in the hardware development process. The ability to reuse common test bench features is therefore key to productivity. In this paper, we present a generic test bench, parameterised by a specification of correctness, which can be used to test any design. Our test bench provides several important features, including automatic test-sequence generation and shrinking of counter-examples, and is fully synthesisable, allowing rigorous testing on FPGA as well as in simulation. The approach is easy to use, cheap to implement, and encourages the formal specification of hardware components through the reward of automatic testing and simple failure cases.This work was supported by DARPA/AFRL contracts FA8750- 10-C-0237 (CTSRD) and FA8750-11-C-0249 (MRC2), and EPSRC grant EP/K008528/1 (REMS).This is the author accepted manuscript. The final version is available from IEEE via http://dx.doi.org/10.1109/MEMCOD.2015.734047

Random testing of interrupt-driven software

ManuscriptInterrupt-driven embedded software is hard to thoroughly test since it usually contains a very large number of executable paths. Developers can test more of these paths using random interrupt testing-firing random interrupt handlers at random times. Unfortunately, na¨ıve application of random testing to interrupt-driven software does not work: some randomly generated interrupt schedules violate system semantics, causing spurious failures. The contribution of this paper is the design, implementation, and experimental evaluation of RID, a restricted interrupt discipline that hardens embedded software with respect to unexpected interrupts, making it possible to perform random interrupt testing and also protecting it from spurious interrupts after deployment. We evaluate RID by implementing it in TinyOS and then using random interrupt testing to find bugs and also to drive applications toward their worst-case stack depths

Strong Automated Testing of OCaml Libraries

National audienceWe present Monolith, a programmable tool that helps apply random testing or fuzz testing to an OCaml library. Monolith provides a rich specification language, which allows the user to describe her library's API, and an engine, which generates clients of this API and executes them. This reduces the problem of testing a library to the problem of testing a complete program, one that is effectively addressed by off-the-shelf fuzzers such as AFL

Testitapausten generointi muodollisten spesifikaatioiden perusteella

Quality is a persistent problem in software development. The main method for quality assurance, testing, is a significant part of any software project. As software development processes move towards continuous integration and deployment, there is demand to increase the level of automation in testing. Conventionally, test automation refers to automating the execution of tests. However, the test cases are written one-by-one, which can get very repetitive. This is also more costly, especially in the maintenance phase. An alternative to these example-based tests are property-based tests, where properties define sets of tests at once. In other words test cases are generated from these properties. Testing targets a certain interface, such as a class, an HTTP API or a GUI. A set of properties – a formal specification – describes the desired behavior of the interface. There is always a human element to specification, because correctness largely depends on the situation and is often subjective. However, verifying that a system adheres to that specification can be automated, at least to some extent. Test case generation is mostly used in functional programming, but this thesis explores it in the object-oriented paradigm as well. Other means of verification are also discussed, and central theoretical concepts are covered. A case study is described, where test cases were generated for the M-Files API – an object-oriented .NET API.Ohjelmiston laatu on ongelma, joka tuskin tulee katoamaan. Laadunvarmistus tapahtuu pääosin testaamalla, ja se on merkittävä osa mitä tahansa ohjelmistoprojektia. Modernit ohjelmistokehitysprosessit perustuvat jatkuvalle integraatiolle ja käyttöönotolle, mikä vaatii testaukselta korkeampaa automaatiotasoa. Perinteisesti testiautomaatio viittaa testien automaattiseen suoritukseen. Tällöin automaatiotasossa on kuitenkin parannettavaa, sillä testien kirjoittaminen voi olla hyvinkin mekaanista toistoa. Tämä taas tulee kalliiksi etenkin ohjelmiston ylläpitovaiheessa. Perinteisten esimerkkipohjaisten testien vaihtoehto on ominaisuuspohjainen testaus, jossa yksittäisten testien sijaan määritellään testijoukkoja. Toisin sanoen ominaisuudet toimivat testitapauksien generoinnin lähtökohtana. Testaamisen kohteena on aina jokin rajapinta, kuten luokka, HTTP API tai graafinen käyttöliittymä. Ominaisuuksien joukko – muodollinen spesifikaatio – kuvailee rajapinnan haluttua toimintaa. Spesifikaation määrittelyssä on aina inhimillinen osuus, sillä järjestelmän virheettömyys on aina paljolti subjektiivista. Sen varmistaminen, että järjestelmä vastaa jotain tiettyä muodollista spesifikaatiota, on kuitenkin yleensä mahdollista automatisoida, ainakin jossain määrin. Testitapausten generoinnin hyödyntäminen on yleisimmillään funktionaalisessa ohjelmoinnissa, mutta tässä työssä sitä tarkastellaan myös olio-ohjelmoinnin yhteydessä. Muitakin verifiointimenetelmiä käsitellään lyhyesti, sekä keskeiset teoreettiset käsitteet käydään läpi. Lisäksi esimerkkitapauksena tutkitaan testien generointia M-Files API:lle, joka on oliopainotteinen .NET-rajapinta

Automated Test Generation Based on an Applicational Model

Context: As testing is an extremely costly and time-consuming process, tools to automatically generate test cases have been proposed throughout the literature. OutSystems provides a software development environment where with the aid of the visual OutSystems language, developers can create their applications in an agile form, thus improving their productivity. Problem: As OutSystems aims at accelerating software development, automating the test case generation activity would bring great value to their clients. Objectives: The main objectives of this work are to: develop an algorithm that generates, automatically, test cases for OutSystems applications and evaluates the coverage they provide to the code, according to a set of criteria. Methods: The OutSystems language is represented as a graph to which developers can then add pieces of code by dragging nodes to the screen and connecting them to the graph. The methodology applied in this work consists in traversing these graphs with depth and breadth-first search algorithms, employing a boundary-value analysis to identify the test inputs and a cause-effect graphing to reduce the number of redundant inputs generated. To evaluate these test inputs, coverage criteria regarding the control flow of data are analysed according to node, branch, condition, modified condition-decision and multiple condition coverage. Results: This tool is able to generate test inputs that cover 100% of reachable code and the methodologies employed help greatly in reducing the inputs generated, as well as displaying a minimum set of test inputs with which the developer is already able to cover all traversable code. Usability tests also yield very optimistic feedback from users. Conclusions: This work’s objectives were fully met, seen as we have a running tool able to act upon a subset of the OutSystems applicational model. This work provides crucial information for assessing the quality of OutSystems applications, with value for OutSystems developers, in the form of efficiency and visibility

Property-based testing for functional programs

This thesis advances the view that property-based testing is a powerful way of testing functional programs, that has advantages not shared by traditional unit testing. It does this by showing two new applications of property-based testing to functional programming as well as a study of the effectiveness of property-based testing.First, we present a tool, QuickSpec, which attempts to infer an equational specification from a functional program with the help of testing. The resulting specifications can be used to improve your understanding of the code or as properties in a test suite. The tool is applicable to quite a wide variety of situations.Second, we describe a system that helps to find race conditions in Erlang programs. It consists of two parts: a randomised scheduler to provoke unusual behaviour in the program under test and allow replayability of test cases, and a module that tests that all of the functions of an API behave atomically with respect to each other.Finally, we present an experiment we carried out to compare property-based testing against test-driven development. The results were inconclusive, but in the process we developed a black-box algorithm for automatically grading student programs by testing, by inferring for each program a set of bugs that the program contains

Property-based testing for functional programs

This thesis advances the view that property-based testing is a powerful way of testing functional programs, that has advantages not shared by traditional unit testing. It does this by showing two new applications of property-based testing to functional programming as well as a study of the effectiveness of property-based testing.First, we present a tool, QuickSpec, which attempts to infer an equational specification from a functional program with the help of testing. The resulting specifications can be used to improve your understanding of the code or as properties in a test suite. The tool is applicable to quite a wide variety of situations.Second, we describe a system that helps to find race conditions in Erlang programs. It consists of two parts: a randomised scheduler to provoke unusual behaviour in the program under test and allow replayability of test cases, and a module that tests that all of the functions of an API behave atomically with respect to each other.Finally, we present an experiment we carried out to compare property-based testing against test-driven development. The results were inconclusive, but in the process we developed a black-box algorithm for automatically grading student programs by testing, by inferring for each program a set of bugs that the program contains