MKorat : a novel approach for memorizing the Korat search and some potential applications

Writing logical constraints that describe properties of desired inputs enables an effective approach for systematic software testing, which can find many bugs. The key problem in systematic constraint-based testing is efficiently exploring very large spaces of all possible inputs to enumerate desired valid inputs. The Korat technique provides an effective solution to this problem. Korat uses desired input properties written as imperative predicates and implements a backtracking search that prunes large parts of the input space and enumerates all non-isomorphic inputs within a given bound on input size. Despite the effectiveness of Korat’s pruning, systematically creating and running large numbers of tests can be costly in practice. Previous work introduced parallel test generation and execution using Korat to make it more practical. We build on a specific algorithm, SEQ-ON, introduced in previous work for equi-distancing candidate inputs, which allows re-execution of Korat for input generation using parallel workers with evenly distributed workload. Our key insight is that the Korat search typically encounters many consecutive candidates that are all invalid inputs and such invalid ranges of candidates can be memoized succinctly to optimize re-execution of Korat. We introduce a novel approach for memoizing Korat’s checking of consecutive invalid candidates, embody the approach into three new techniques based on SEQ-ON, evaluate the techniques using a standard suite of data structure subjects to show the efficacy of our approach, and show some potential applications of it in two new application domains for Korat. We believe our work opens a promising new direction to optimize solving of imperative constraints and using them in novel application domains.Electrical and Computer Engineerin

Automatic Test Generation for Space

The European Space Agency (ESA) uses an engine to perform tests in the Ground Segment infrastructure, specially the Operational Simulator. This engine uses many different tools to ensure the development of regression testing infrastructure and these tests perform black-box testing to the C++ simulator implementation. VST (VisionSpace Technologies) is one of the companies that provides these services to ESA and they need a tool to infer automatically tests from the existing C++ code, instead of writing manually scripts to perform tests. With this motivation in mind, this paper explores automatic testing approaches and tools in order to propose a system that satisfies VST needs

Korat-API : an API for building constraint solving problems for Korat

This thesis introduces the foundation of an API for building constraint solving problems for the Korat solver for imperative predicates. Our goal is two-fold: (1) to facilitate the use of Korat as a backend solver for applications that desire using it as a constraint solving engine; and (2) to facilitate optimized analyses using Korat, which follow the spirit of modern constraint solving and software testing techniques. We describe the API and how it uses the core Korat engine, and demonstrate the benefits in two application contexts: (1) using Korat as a backend engine for model counting; and (2) using Korat as test generator. We believe our work introduces a promising approach for making the ability of Korat to efficiently solve imperative predicates more widely applicable, possibly even in new application contexts where Korat has net been used before.Electrical and Computer Engineerin

Pairwise-Korat : automated testing using Korat in an industrial setting

textIn this report, we present an algorithm for testing applications which takes structurally complex test inputs. The algorithm, Pairwise-Korat, adopts Korat -- an algorithm for constraint-based generation of structurally complex test inputs. Korat takes (1) an imperative predicate which specifies the desired structural integrity constraints and (2) a finitization which bounds the desired test inputs size. Korat performs a systematic search to generate all test inputs (within the bounds) for which the predicate returns true. We present how to generate test inputs in Korat and how to execute test inputs in parallel. The inputs that Korat generates enable bounded-exhaustive testing that checks the code under test exhaustively for all inputs within the given bounds. We also describe a novel methodology for reducing the number of equivalent inputs that Korat generates. Our development of test input generation and the methodology for reducing equivalent inputs are motivated by testing applications developed at eBay. The experimental results show that the Pairwise-Korat achieves great performance in finding defects and increasing test coverage and the algorithm outperforms current manual solutions adopted at the company.Electrical and Computer Engineerin

JWalk: a tool for lazy, systematic testing of java classes by design introspection and user interaction

Popular software testing tools, such as JUnit, allow frequent retesting of modified code; yet the manually created test scripts are often seriously incomplete. A unit-testing tool called JWalk has therefore been developed to address the need for systematic unit testing within the context of agile methods. The tool operates directly on the compiled code for Java classes and uses a new lazy method for inducing the changing design of a class on the fly. This is achieved partly through introspection, using Java’s reflection capability, and partly through interaction with the user, constructing and saving test oracles on the fly. Predictive rules reduce the number of oracle values that must be confirmed by the tester. Without human intervention, JWalk performs bounded exhaustive exploration of the class’s method protocols and may be directed to explore the space of algebraic constructions, or the intended design state-space of the tested class. With some human interaction, JWalk performs up to the equivalent of fully automated state-based testing, from a specification that was acquired incrementally

Contract-driven data structure repair : a novel approach for error recovery

textSoftware systems are now pervasive throughout our world. The reliability of these systems is an urgent necessity. A large degree of research effort on increasing software reliability is dedicated to requirements, architecture, design, implementation and testing---activities that are performed before system deployment. While such approaches have become substantially more advanced, software remains buggy and failures remain expensive. We take a radically different approach to reliability from previous approaches, namely contract-driven data structure repair for runtime error recovery, where erroneous executions of deployed software are corrected on-the-fly using rich behavioral contracts. Our key insight is to transform the software contract---which gives a high level description of the expected behavior---to an efficient implementation which repairs the erroneous data structures in the program state upon an error. To improve efficiency, scalability, and effectiveness of repair, in addition to rich behavioral contracts, we leverage the current erroneous state, dynamic behavior of the program, as well as repair history and abstraction. A core technical problem our approach to repair addresses is construction of structurally complex data that satisfy desired properties. We present a novel structure generation technique based on dynamic programming---a classic optimization approach---to utilize the recursive nature of the structures. We use our technique for constraint-based testing. It provides better scalability than previous work. We applied it to test widely-used web browsers and found some known and unknown bugs. Our use of dynamic programming in structure generation opens a new future direction to tackle the scalability problem of data structure repair. This research advances our ability to develop correct programs. For programs that already have contracts, error recovery using our approach can come at a low cost. The same contracts can be used for systematically testing code before deployment using existing as well as our new techniques. Thus, we enable a novel unification of software verification and error recovery.Electrical and Computer Engineerin

Enhancing usability and applicability of Korat

textSoftware testing is an integral part of the software development cycle, and involves various techniques to test software components and applications. Specification-based testing focuses on expected functionality as described in given specifications. Korat is a tool for generating structurally complex test inputs for specification-based testing of Java programs that operate on such inputs. Korat uses specifications written as Java predicates, that describe properties of expected input structures and efficiently generates all non-isomorphic valid structures within given bounds on input size. This report describes the software requirements, application design and implementation details of our effort to improve usability and applicability of Korat. Our work involves functional enhancements to the classic Korat tool to provide support for the following elements: Graphical User Interface (GUI), Java Universal Network/Graph Framework (JUNG) output, Finite State Machine Domain (FSM), and JavaScript Object Notation (JSON) graph archival.Electrical and Computer Engineerin

Rule-based Test Generation with Mind Maps

This paper introduces basic concepts of rule based test generation with mind maps, and reports experiences learned from industrial application of this technique in the domain of smart card testing by Giesecke & Devrient GmbH over the last years. It describes the formalization of test selection criteria used by our test generator, our test generation architecture and test generation framework.Comment: In Proceedings MBT 2012, arXiv:1202.582