API Documentation with Executable Examples

The rise of component-based software development has created an urgent need for effective API documentation. Experience has shown that it is hard to create precise and readable documentation. Prose documentation can provide a good overview but lacks precision. Formal methods offer precision but the resulting documentation is expensive to develop. Worse, few developers have the skill or inclination to read formal documentation. We present a pragmatic solution to the problem of API documentation. We augment the prose documentation with executable test cases, including expected outputs, and use the prose plus the test cases as the documentation. With appropriate tool support, the test cases are easy to develop and read. Such test cases constitute a completely formal, albeit partial, specification of input/output behavior. Equally important, consistency between code and documentation is demonstrated by running the test cases. This approach provides an attractive bridge between formal and informal documentation. We also present a tool that supports compact and readable test cases, and generation of test drivers and documentation, and illustrate the approach with detailed case studies

State-based testing - a new method for testing object-oriented programs

State-based testing is a new method for testing object-oriented programs. The information stored in the state of an object is of two kinds: control-information and data-storage. The control-information transitions are modelled as a finite state automaton. Every operation of the class under test is considered as a mapping from starting states to a finishing states dependent upon the parameters passed. The possible parameter values are analysed for significant values which combined with the invocation of an operation can be used to represent stimuli applied to an object under test. State-based testing validates the expected transformations that can occur within a class. Classes are modelled using physical values assigned to the attributes of the class. The range of physical values is reduced by the use of a technique based on equivalence partitioning. This approach has a number of advantages over the conceptual modelling of a class, in particular the ease of manipulation of physical values and the independence of each operation from the other operations provided by an object. The technique when used in conjunction with other techniques provides an adequate level of validation for object-oriented programs. A suite of prototype tools that automate the generation of state-based test cases are outlined. These tools are used in four case studies that are presented as an evaluation of the technique. The code coverage achieved with each case study is analysed for the factors that affect the effectiveness of the state-based test suite. Additionally, errors have been seeded into 2 of the classes to determine the effectiveness of the technique for detecting errors on paths that are executed by the test suite. 92.5% of the errors seeded were detected by the state-based test-suite

Test case prioritization

Regression testing is an expensive software engineering activity intended to provide confidence that modifications to a software system have not introduced faults. Test case prioritization techniques help to reduce regression testing cost by ordering test cases in a way that better achieves testing objectives. In this thesis, we are interested in prioritizing to maximize a test suite's rate of fault detection, measured by a metric, APED, trying to detect regression faults as early as possible during testing. In previous work, several prioritization techniques using low-level code coverage information had been developed. These techniques try to maximize APED over a sequence of software releases, not targeting a particular release. These techniques' effectiveness was empirically evaluated. We present a larger set of prioritization techniques that use information at arbitrary granularity levels and incorporate modification information, targeting prioritization at a particular software release. Our empirical studies show significant improvements in the rate of fault detection over randomly ordered test suites. Previous work on prioritization assumed uniform test costs and fault seventies, which might not be realistic in many practical cases. We present a new cost-cognizant metric, APFD[subscript c], and prioritization techniques, together with approaches for measuring and estimating these costs. Our empirical studies evaluate prioritization in a cost-cognizant environment. Prioritization techniques have been developed independently with little consideration of their similarities. We present a general prioritization framework that allows us to express existing prioritization techniques by a framework algorithm using parameters and specific functions. Previous research assumed that prioritization was always beneficial if it improves the APFD metric. We introduce a prioritization cost-benefit model that more accurately captures relevant cost and benefit factors, and allows practitioners to assess whether it is economical to employ prioritization. Prioritization effectiveness varies across programs, versions, and test suites. We empirically investigate several of these factors on substantial software systems and present a classification-tree-based predictor that can help select the most appropriate prioritization technique in advance. Together, these results improve our understanding of test case prioritization and of the processes by which it is performed