Black-Box Test Case Generation from TFM Module Interface Specications and Usage Statistics

In this thesis, we propose a black-box testing method that derives important test cases by including usage statistics, and enables a product manager to make a release decision with the rationale, the important use cases specified in the usage statistics are tested and have no error. First, we propose a method to specify components with Trace Function Method (TFM) module interface specifications. Then, we propose a way to associate module usage statistics with the TFM module interface specification. Finally, we propose a method to generate a prioritized list of black-box test cases for component testing and integration testing from the TFM module interface specification with usage statistics

Regression Testing and Test selection in Research

ABSTRACT Regression testing is a costly but crucial problem in software development. Both the research community and the industry have paid much attention to this problem. However, are the issues they concerned the same? The paper try to do the survey of current research on regression testing and current practice in industry and also try to find out whether there are gaps between them. The observations show that although some issues are concerned both by the research community and the industry gay, there do exist gaps. Keywords Regression Testing, Software Engineering, Software Maintenance The goal of this project is to address the gaps between current research and current application of regression in industry by studying the current research on regressing testing and the literatures on existing testing tools which are widely used in the industry. Once the gaps are addressed, we will be able to ask the question---why this happens and new research direction and new guideline for the practice will be proposed. The rest of the paper is organized as following: Section 2 is a briefly review of the current regression testing research literatures. In Section 3, the current popular commercial tools will first be introduced, and then some case studies are presented as they are some practices to apply the regression testing technology. Some interesting observations have been found according to Section 2 and Section 3, and they are presented in Section 4. Conclusions are given In Section 5

State-based testing - a new method for testing object-oriented programs

State-based testing is a new method for testing object-oriented programs. The information stored in the state of an object is of two kinds: control-information and data-storage. The control-information transitions are modelled as a finite state automaton. Every operation of the class under test is considered as a mapping from starting states to a finishing states dependent upon the parameters passed. The possible parameter values are analysed for significant values which combined with the invocation of an operation can be used to represent stimuli applied to an object under test. State-based testing validates the expected transformations that can occur within a class. Classes are modelled using physical values assigned to the attributes of the class. The range of physical values is reduced by the use of a technique based on equivalence partitioning. This approach has a number of advantages over the conceptual modelling of a class, in particular the ease of manipulation of physical values and the independence of each operation from the other operations provided by an object. The technique when used in conjunction with other techniques provides an adequate level of validation for object-oriented programs. A suite of prototype tools that automate the generation of state-based test cases are outlined. These tools are used in four case studies that are presented as an evaluation of the technique. The code coverage achieved with each case study is analysed for the factors that affect the effectiveness of the state-based test suite. Additionally, errors have been seeded into 2 of the classes to determine the effectiveness of the technique for detecting errors on paths that are executed by the test suite. 92.5% of the errors seeded were detected by the state-based test-suite

The Effectiveness of <i>t</i>-Way Test Data Generation

Modern society is increasingly dependent on the correct functioning of software and increasingly so in areas that are considered safety related or safety critical. Therefore, there is an increasing need to be able to verify and validate that the software is in fact correct and will perform its intended function. Many approaches to this problem have been proposed; however, none seems likely to supplant the role of testing in the near future. If we accept that there is, and will be, a continuing need to be able to test software then the question becomes one of how can this be done effectively, both in terms of ability to detect errors and in terms of cost. One avenue of research that offers prospects of improving both of these aspects is the automatic generation of test data. There has recently been a large amount of work conducted in this area. One particularly promising direction has been the application of ideas from the field of experimental design and in particular, the field of t-way adequate factorial designs. The area however, is not without issues; there is evidence that the technique is capable of detecting errors but that evidence is not unequivocal. Moreover, as with almost all work in the area of automatic test generation, there has been very little comparative work comparing the technique with other test data generation techniques. Worse, there has been effectively no work done that compares any automatic test data generation technique with the effectiveness of tests generated by humans. Another major issue with the technique is the number of tests that applying the technique can result in. This implies that there is a need for an automated oracle if the technique is to be successfully applied. The flaw with this is of course that in most situations the oracle is the human that is conducting the tests, a point often ignored in testing research. The work presented here addresses both of these points. To do this I have used a code base taken from an industrial engine control system that has an existing set of high quality unit tests developed by hand. To complement this, several other techniques for automatically generating test data have been applied, namely random testing, random experimental designs and a technique for generating single factor experiments. To address the issue of being able to compare the error detection ability of all of the sets of test vectors, rather than the usual effectiveness surrogates of code coverage I have used mutation analysis on the code base to directly measure the ability of each set of test vectors to discover common coding errors. The results presented here show that test data generation techniques based on t-way factorial designs are at least as effective as handgenerated tests and superior to random testing and the factor experimental technique. The oracle problem associated with the factorial design techniques was addressed using a test set minimisation approach. The mutation tool monitored which vectors could “kill” which code mutants. After a subset of the test vectors had been run, the most effective vectors were retained and the rest discarded. Likewise, mutants that were killed were removed from further consideration and the process repeated. Experimental results show that this minimisation procedure is effective at reducing computational overhead and is capable of producing final sets of test vectors that are comparable in size with the sets of hand-generated tests and so amenable to final hand checking

Software test and evaluation study phase I and II : survey and analysis

Issued as Final report, Project no. G-36-661 (continues G-36-636; includes A-2568

Test case prioritization

Regression testing is an expensive software engineering activity intended to provide confidence that modifications to a software system have not introduced faults. Test case prioritization techniques help to reduce regression testing cost by ordering test cases in a way that better achieves testing objectives. In this thesis, we are interested in prioritizing to maximize a test suite's rate of fault detection, measured by a metric, APED, trying to detect regression faults as early as possible during testing. In previous work, several prioritization techniques using low-level code coverage information had been developed. These techniques try to maximize APED over a sequence of software releases, not targeting a particular release. These techniques' effectiveness was empirically evaluated. We present a larger set of prioritization techniques that use information at arbitrary granularity levels and incorporate modification information, targeting prioritization at a particular software release. Our empirical studies show significant improvements in the rate of fault detection over randomly ordered test suites. Previous work on prioritization assumed uniform test costs and fault seventies, which might not be realistic in many practical cases. We present a new cost-cognizant metric, APFD[subscript c], and prioritization techniques, together with approaches for measuring and estimating these costs. Our empirical studies evaluate prioritization in a cost-cognizant environment. Prioritization techniques have been developed independently with little consideration of their similarities. We present a general prioritization framework that allows us to express existing prioritization techniques by a framework algorithm using parameters and specific functions. Previous research assumed that prioritization was always beneficial if it improves the APFD metric. We introduce a prioritization cost-benefit model that more accurately captures relevant cost and benefit factors, and allows practitioners to assess whether it is economical to employ prioritization. Prioritization effectiveness varies across programs, versions, and test suites. We empirically investigate several of these factors on substantial software systems and present a classification-tree-based predictor that can help select the most appropriate prioritization technique in advance. Together, these results improve our understanding of test case prioritization and of the processes by which it is performed