LittleDarwin: a Feature-Rich and Extensible Mutation Testing Framework for Large and Complex Java Systems

Mutation testing is a well-studied method for increasing the quality of a test suite. We designed LittleDarwin as a mutation testing framework able to cope with large and complex Java software systems, while still being easily extensible with new experimental components. LittleDarwin addresses two existing problems in the domain of mutation testing: having a tool able to work within an industrial setting, and yet, be open to extension for cutting edge techniques provided by academia. LittleDarwin already offers higher-order mutation, null type mutants, mutant sampling, manual mutation, and mutant subsumption analysis. There is no tool today available with all these features that is able to work with typical industrial software systems.Comment: Pre-proceedings of the 7th IPM International Conference on Fundamentals of Software Engineerin

Modelling Realistic User Behaviour in Information Systems Simulations as Fuzzing Aspects

In this paper we contend that the engineering of information systems is hampered by a paucity of tools to tractably model, simulate and predict the impact of realistic user behaviours on the emergent properties of the wider socio-technical system, evidenced by the plethora of case studies of system failure in the literature. We address this gap by presenting a novel approach that models ideal user behaviour as workflows, and introduces irregularities in that behaviour as aspects which fuzz the model. We demonstrate the success of this approach through a case study of software development workflows, showing that the introduction of realistic user behaviour to idealised workflows better simulates outcomes reported in the empirical software engineering literature

Spotting Trees with Few Leaves

We show two results related to the Hamiltonicity and $k$-Path algorithms in undirected graphs by Bj\"orklund [FOCS'10], and Bj\"orklund et al., [arXiv'10]. First, we demonstrate that the technique used can be generalized to finding some $k$-vertex tree with $l$ leaves in an $n$-vertex undirected graph in $O^*(1.657^k2^{l/2})$ time. It can be applied as a subroutine to solve the $k$-Internal Spanning Tree ($k$-IST) problem in $O^*(\min(3.455^k, 1.946^n))$ time using polynomial space, improving upon previous algorithms for this problem. In particular, for the first time we break the natural barrier of $O^*(2^n)$. Second, we show that the iterated random bipartition employed by the algorithm can be improved whenever the host graph admits a vertex coloring with few colors; it can be an ordinary proper vertex coloring, a fractional vertex coloring, or a vector coloring. In effect, we show improved bounds for $k$-Path and Hamiltonicity in any graph of maximum degree $\Delta=4,\ldots,12$ or with vector chromatic number at most 8

Do Null-Type Mutation Operators Help Prevent Null-Type Faults?

The null-type is a major source of faults in Java programs, and its overuse has a severe impact on software maintenance. Unfortunately traditional mutation testing operators do not cover null-type faults by default, hence cannot be used as a preventive measure. We address this problem by designing four new mutation operators which model null-type faults explicitly. We show how these mutation operators are capable of revealing the missing tests, and we demonstrate that these mutation operators are useful in practice. For the latter, we analyze the test suites of 15 open-source projects to describe the trade-offs related to the adoption of these operators to strengthen the test suite

Accelerated Model Checking of Parametric Markov Chains

Parametric Markov chains occur quite naturally in various applications: they can be used for a conservative analysis of probabilistic systems (no matter how the parameter is chosen, the system works to specification); they can be used to find optimal settings for a parameter; they can be used to visualise the influence of system parameters; and they can be used to make it easy to adjust the analysis for the case that parameters change. Unfortunately, these advancements come at a cost: parametric model checking is---or rather was---often slow. To make the analysis of parametric Markov models scale, we need three ingredients: clever algorithms, the right data structure, and good engineering. Clever algorithms are often the main (or sole) selling point; and we face the trouble that this paper focuses on -- the latter ingredients to efficient model checking. Consequently, our easiest claim to fame is in the speed-up we have often realised when comparing to the state of the art

Reducing the Cost of Grammar-Based Testing Using Pattern Coverage

Part 2: Test Derivation MethodsInternational audienceIn grammar-based testing, context-free grammars may be used to generate relevant test inputs for language processors, or meta programs, such as programming language compilers, refactoring tools, and implementations of software quality metrics. This technique can be used to test these meta programs, but the amount of sentences, and syntax trees thereof, which needs to be generated to obtain reasonable coverage of the input language is exponential.Pattern matching is a programming language feature used often when writing meta programs. Pattern matching helps because it automates the frequently occurring task of detecting shapes in, and extracting information from syntax trees. However, meta programs which contain many patterns are difficult to test using only randomly generated sentences from grammar rules. The reason is that statistically it is uncommon to directly generate sentences which accidentally match the patterns in the code.To solve this problem, in this paper we extract information from the patterns in the code of meta programs to guide the sentence generation process. We introduce a new coverage criterion, called Pattern Coverage, which focuses on providing a test strategy to reduce the amount of test necessary cases, while covering the relevant parts of the meta program. An initial experimental evaluation is presented and the result is compared with traditional grammar-based testing

Practical Verification of Decision-Making in Agent-Based Autonomous Systems

We present a verification methodology for analysing the decision-making component in agent-based hybrid systems. Traditionally hybrid automata have been used to both implement and verify such systems, but hybrid automata based modelling, programming and verification techniques scale poorly as the complexity of discrete decision-making increases making them unattractive in situations where complex log- ical reasoning is required. In the programming of complex systems it has, therefore, become common to separate out logical decision-making into a separate, discrete, component. However, verification techniques have failed to keep pace with this devel- opment. We are exploring agent-based logical components and have developed a model checking technique for such components which can then be composed with a sepa- rate analysis of the continuous part of the hybrid system. Among other things this allows program model checkers to be used to verify the actual implementation of the decision-making in hybrid autonomous systems

Software Testing Techniques Revisited for OWL Ontologies

Ontologies are an essential component of semantic knowledge bases and applications, and nowadays they are used in a plethora of domains. Despite the maturity of ontology languages, support tools and engineering techniques, the testing and validation of ontologies is a field which still lacks consolidated approaches and tools. This paper attempts at partly bridging that gap, taking a first step towards the extension of some traditional software testing techniques to ontologies expressed in a widely-used format. Mutation testing and coverage testing, revisited in the light of the peculiar features of the ontology language and structure, can can assist in designing better test suites to validate them, and overall help in the engineering and refinement of ontologies and software based on them

Gaining certainty about uncertainty: Testing cyber-physical systems in the presence of uncertainties at the application level

A cyber-physical system (CPS) comprises several connected, embedded systems and is additionally equipped with sensors and actuators. Thus, CPSs can communicate with their cyber environment and measure and interact with their physical environment. Due to the complexity of their operational environment, assumptions the manufacturer have made may not hold in operation. During an unforeseen environmental situation, a CPS may expose behavior that negatively impactsits reliability. This may arise due to insufficiently considered environmental conditions during the design of a CPS, or – even worse – it is impossible to anticipate such conditions. In the U-Test project, we are developing a configurable search-based testing framework that exploits information from functional testing and from declarative descriptions of uncertainties. Itaims at revealing unintended behavior in the presence of uncertainties. This framework enables testing for different scenarios of uncertainty and thus, allows to achieve a certain coverage of those, and to find unknown uncertainty scenarios