A Passive Test Oracle Using a Component's API

A test oracle is a mechanism that is used during testing to determine whether a software component behaves correctly or not. The test oracle problem is widely acknowledged in the software testing literature and many methods for test oracle development have been proposed. Most of these methods use specifications or other resources to develop test oracles. A passive test oracle checks the behaviour of the component, but does not reproduce this behaviour. In this paper, we present a technique that develops passive test oracles for components using their APIs. This simple technique can be applied to any software component that is accessed through an API. In an initial experiment, we found that test oracles developed this way were more effective at finding faults with a relatively small number of test cases than test oracles developed from a formal specification and developed as a parallel implementation

Melhoria da testabilidade de classes usando o conceito de autoteste

Orientador: Eliane MartinsDissertação (mestrado) - Universidade Estadual de Campinas. Instituto de ComputaçãoResumo: O esforço e custo adicionais exigidos pelo processo de teste de um software ou componente depende em geral de quão fácil é testá-Io. No caso de componentes reutilizáveis essa habilidade se toma mais importante ainda, pois eles devem ser testados durante seu desenvolvimento, toda vez que forem reutilizados em um novo contexto, e a cada vez que sofram alguma alteração. Uma proposta para construir classes com maior testabilidade foi feita fazendo-se uma analogia entre circuitos integrados e classes. Foi proposta a construção de classes autotestáveis e tal como em circuitos integrados, estas classes teriam mecanismos embutidos específicos para teste. Outro fator importante para testabilidade do software é a possibilidade de reutilização de testes. Por esta razão foi empregada nesse estudo a técnica incremental hierárquica, que leva em conta a hierarquia de herança permitindo que, em alguns casos, seja possível reutilizar casos de testes gerados para a classe base nos testes de suas classes derivadas. Este trabalho apresenta a metodologia definida para construir uma classe autotestável, bem como o protótipo construído para automatizar parte desta metodologia, mostrando assim a sua viabilidadeAbstract: In general the additional effort and cost required by the testing process depends on the testability of the software or component being tested. Testability is more important for reusable components because they need testing during their development and afier each change (either in the class or in the environrnent). An approach comparing classes and integrated circuits of hardware had been proposed for constructing classes with higher testability. The resulting class is called builtin self-testing class, because it contains built-in testing mechanisms. Class testability will be increased if reuse could be extended to the test cases too. The incremental hierarchical technique considers the inheritance relationship to test classes by reusing test cases of a parent class when testing a subclass. This work presentes: a methodology to construct built-in self-testing classes and a prototyping tool, ConCAT, that automates part of this methodology, by showing its feasibilityMestradoMestre em Ciência da Computaçã

Automated test of evolving software

A thesis submitted to the University of Luton, in partial fulfilment of the requirements for the degree of Doctor of PhilosophyComputers and the software they run are pervasive, yet released software is often unreliable, which has many consequences. Loss of time and earnings can be caused by application software (such as word processors) behaving incorrectly or crashing. Serious disruption can occur as in the l4th August 2003 blackouts in North East USA and Canadal, or serious injury or death can be caused as in the Therac-25 overdose incidents. One way to improve the quality of software is to test it thoroughly. However, software testing is time consuming, the resources, capabilities and skills needed to carry it out are often not available and the time required is often curtailed because of pressures to meet delivery deadlines3. Automation should allow more thorough testing in the time available and improve the quality of delivered software, but there are some problems with automation that this research addresses. Firstly, it is difficult to determine ifthe system under test (SUT) has passed or failed a test. This is known as the oracle problem4 and is often ignored in software testing research. Secondly, many software development organisations use an iterative and incremental process, known as evolutionary development, to write software. Following release, software continues evolving as customers demand new features and improvements to existing ones5. This evolution means that automated test suites must be maintained throughout the life ofthe software. A contribution of this research is a methodology that addresses automatic generation of the test cases, execution of the test cases and evaluation of the outcomes from running each test. "Predecessor" software is used to solve the oracle problem. This is software that already exists, such as a previous version of evolving software, or software from a different vendor that solves the same, or similar, problems. However, the resulting oracle is assumed not be perfect, so rules are defined in an interface, which are used by the evaluator in the test evaluation stage to handle the expected differences. The interface also specifies functional inputs and outputs to the SUT. An algorithm has been developed that creates a Markov Chain Transition Matrix (MCTM) model of the SUT from the interface. Tests are then generated automatically by making a random walk of the MCTM. This means that instead of maintaining a large suite of tests, or a large model of the SUT, only the interface needs to be maintained. 1) NERC Steering Group (2004). Technical Analysis ofthe August 14,2003, Blackout: What Happened, Why, and What Did We Learn? July 13th 2004. Available from: ftp:/ /www.nerc.com/pub/sys/all_ updl/docslblackoutINERC ]inatBlackout_Report _ 07_13_ 04.pdf 2) Leveson N. G., Turner C. S. (1993) An investigation of the Therac-25 accidents. IEEE Computer, Vo126, No 7, Pages 18-41. 3) LogicaCMG (2005) Testing Times for Board Rooms. Available from http://www.logicacmg.com/pdf/trackeditestingTimesBoardRooms.pdf 4) Bertolino, A. (2003) Software Testing Research and Practice, ASM 2003, Lecture Notes in Computer Science, Vol 2589, Pages 1-21. 5) Sommerville, 1. (2004) Software Engineering, 7th Edition. Addison Wesley. ISBN 0-321-21026-3

The Effectiveness of <i>t</i>-Way Test Data Generation

Modern society is increasingly dependent on the correct functioning of software and increasingly so in areas that are considered safety related or safety critical. Therefore, there is an increasing need to be able to verify and validate that the software is in fact correct and will perform its intended function. Many approaches to this problem have been proposed; however, none seems likely to supplant the role of testing in the near future. If we accept that there is, and will be, a continuing need to be able to test software then the question becomes one of how can this be done effectively, both in terms of ability to detect errors and in terms of cost. One avenue of research that offers prospects of improving both of these aspects is the automatic generation of test data. There has recently been a large amount of work conducted in this area. One particularly promising direction has been the application of ideas from the field of experimental design and in particular, the field of t-way adequate factorial designs. The area however, is not without issues; there is evidence that the technique is capable of detecting errors but that evidence is not unequivocal. Moreover, as with almost all work in the area of automatic test generation, there has been very little comparative work comparing the technique with other test data generation techniques. Worse, there has been effectively no work done that compares any automatic test data generation technique with the effectiveness of tests generated by humans. Another major issue with the technique is the number of tests that applying the technique can result in. This implies that there is a need for an automated oracle if the technique is to be successfully applied. The flaw with this is of course that in most situations the oracle is the human that is conducting the tests, a point often ignored in testing research. The work presented here addresses both of these points. To do this I have used a code base taken from an industrial engine control system that has an existing set of high quality unit tests developed by hand. To complement this, several other techniques for automatically generating test data have been applied, namely random testing, random experimental designs and a technique for generating single factor experiments. To address the issue of being able to compare the error detection ability of all of the sets of test vectors, rather than the usual effectiveness surrogates of code coverage I have used mutation analysis on the code base to directly measure the ability of each set of test vectors to discover common coding errors. The results presented here show that test data generation techniques based on t-way factorial designs are at least as effective as handgenerated tests and superior to random testing and the factor experimental technique. The oracle problem associated with the factorial design techniques was addressed using a test set minimisation approach. The mutation tool monitored which vectors could “kill” which code mutants. After a subset of the test vectors had been run, the most effective vectors were retained and the rest discarded. Likewise, mutants that were killed were removed from further consideration and the process repeated. Experimental results show that this minimisation procedure is effective at reducing computational overhead and is capable of producing final sets of test vectors that are comparable in size with the sets of hand-generated tests and so amenable to final hand checking

AUTOMATED MODULE TESTING IN PROLOG

We present tools and techniques for writing scripts in Prolog that automatically test modules implemented in C. Both the input generation and the test oracle problems are addressed, focusing on a balance between the adequacy of the test inputs and the cost of developing the output oracle. We investigate automated input generation according to functional testing, random testing, and a new approach based on trace invariants. For each input generation scheme we have developed a mechanism for generating the expected outputs. The methods are described and illustrated in detail. Script development and maintenance costs appear to be reasonable and run-time performance acceptable