Automatic Generation of Load Tests

Load tests aim to validate whether system performance is acceptable under peak conditions. Existing test generation techniques induce load by increasing the size or rate of the input. Ignoring the particular input values, however, may lead to test suites that grossly mischaracterize a system’s performance. To address this limitation we introduce a mixed symbolic execution based approach that is unique in how it 1) favors program paths associated with a performance measure of interest, 2) operates in an iterative-deepening beam-search fashion to discard paths that are unlikely to lead to high-load tests, and 3) generates a test suite of a given size and level of diversity. An assessment of the approach shows it generates test suites that induce program response times and memory consumption several times worse than the compared alternatives, it scales to large and complex inputs, and it exposes a diversity of resource consuming program behavior

Über logisch-funktionale Programmierung und deren Anwendung zum Testen

Die vorliegende Arbeit untersucht die Implementierung logisch-funktionaler Programmierung und deren Anwendung zur automatischen Generierung von Testdaten. Logisch-funktionale Programmierung vereint zwei deklarative Programmierparadigmen, funktionale Programmierung und Logikprogrammierung, in einem einheitlichen Programmiermodell. Es stellt sich heraus, dass diese Kombination zur Spezifikation und automatischen Generierung von Testdaten gut geeignet ist. Motiviert durch die erkannte Verwandtschaft von Testdatengenerierung und logisch-funktionaler Programmierung, stellt die Arbeit einen neuen Ansatz vor, logisch-funktionale Programme in rein funktionalen Programmiersprachen auszudrücken

A Survey of Symbolic Execution Techniques

Many security and software testing applications require checking whether certain properties of a program hold for any possible usage scenario. For instance, a tool for identifying software vulnerabilities may need to rule out the existence of any backdoor to bypass a program's authentication. One approach would be to test the program using different, possibly random inputs. As the backdoor may only be hit for very specific program workloads, automated exploration of the space of possible inputs is of the essence. Symbolic execution provides an elegant solution to the problem, by systematically exploring many possible execution paths at the same time without necessarily requiring concrete inputs. Rather than taking on fully specified input values, the technique abstractly represents them as symbols, resorting to constraint solvers to construct actual instances that would cause property violations. Symbolic execution has been incubated in dozens of tools developed over the last four decades, leading to major practical breakthroughs in a number of prominent software reliability applications. The goal of this survey is to provide an overview of the main ideas, challenges, and solutions developed in the area, distilling them for a broad audience. The present survey has been accepted for publication at ACM Computing Surveys. If you are considering citing this survey, we would appreciate if you could use the following BibTeX entry: http://goo.gl/Hf5FvcComment: This is the authors pre-print copy. If you are considering citing this survey, we would appreciate if you could use the following BibTeX entry: http://goo.gl/Hf5Fv

Tickling Java with a Feather

Fragments of mainstream programming languages are formalised in order to show desirable properties of their static semantics. We ask if said formalisms could also be used to define a test suite for the mainstream programming language in question, and what the utility of such a suite would be. In this work, we present our findings from testing Java with Featherweight Java (FJ). We take the syntax and binding structure of FJ to define an instance space of non-isomorphic test programs and implementations of FJ type checkers to provide oracles for our tests, to ensure the mainstream implementation conforms with the expectations of FJ. Using these, we evaluate (using code coverage techniques) how much of the Sun OpenJDK javac can be tested by FJ. © 2009 Elsevier B.V. All rights reserved

Automating Black-Box Property Based Testing

<p>Black-box property based testing tools like QuickCheck allow developers to write elegant logical specifications of their programs, while still permitting unrestricted use of the same language features and libraries that simplify writing the programs themselves. This is an improvement over unit testing because a single property can replace a large collection of test cases, and over more heavy-weight white-box testing frameworks that impose restrictions on how properties and tested code are written. In most cases the developer only needs to write a function returning a boolean, something any developer is capable of without additional training. </p> <p> This thesis aims to further lower the threshold for using property based testing by automating some problematic areas, most notably generating test data for user defined data types. Writing procedures for random test data generation by hand is time consuming and error prone, and most fully automatic algorithms give very poor random distributions for practical cases. </p><p> Several fully automatic algorithms for generating test data are presented in this thesis, along with implementations as Haskell libraries. These algorithms all fit nicely within a framework called sized functors, allowing re-usable generator definitions to be constructed automatically or by hand using a few simple combinators. </p><p> Test quality is another difficulty with property based testing. When a property fails to find a counterexample there is always some uncertainty in the strength of the property as a specification. To address this problem we introduce a black-box variant of mutation testing. Usually mutation testing involves automatically introducing errors (mutations) in the source code of a tested program to see if a test suite can detect it. Using higher order functions, we mutate functions without accessing their source code. The result is a very light-weight mutation testing procedure that automatically estimates property strength for QuickCheck properties. </p