12 research outputs found
Test oracle assessment and improvement
We introduce a technique for assessing and improving test oracles by reducing the incidence of both false positives and false negatives. We prove that our approach can always result in an increase in the mutual information between the actual and perfect oracles. Our technique combines test case generation to reveal false positives and mutation testing to reveal false negatives. We applied the decision support tool that implements our oracle improvement technique to five real-world subjects. The experimental results show that the fault detection rate of the oracles after improvement increases, on average, by 48.6% (86% over the implicit oracle). Three actual, exposed faults in the studied systems were subsequently confirmed and fixed by the developers
Optimizing for the Number of Tests Generated in Search Based Test Data Generation with an Application to the Oracle Cost Problem
Previous approaches to search based test data generation tend to focus on coverage, rather than oracle cost. While there may be an aspiration that systems should have models, checkable specifications and/or contract driven development, this sadly remains an aspiration; in many real cases, system behaviour must be checked by a human. This painstaking checking process forms a significant cost, the oracle cost, which previous work on automated test data generation tends to overlook. One simple way to reduce oracle cost consists of reducing the number of tests generated. In this paper we introduce three algorithms which do this without compromising coverage achieved. We present the results of an empirical study of the effectiveness of the three algorithms on five benchmark programs containing non trivial search spaces for branch coverage. The results indicate that it is, indeed, possible to make reductions in the number of test cases produced by search based testing, without loss of coverage
Contracts in Practice
Contracts are a form of lightweight formal specification embedded in the
program text. Being executable parts of the code, they encourage programmers to
devote proper attention to specifications, and help maintain consistency
between specification and implementation as the program evolves. The present
study investigates how contracts are used in the practice of software
development. Based on an extensive empirical analysis of 21 contract-equipped
Eiffel, C#, and Java projects totaling more than 260 million lines of code over
7700 revisions, it explores, among other questions: 1) which kinds of contract
elements (preconditions, postconditions, class invariants) are used more often;
2) how contracts evolve over time; 3) the relationship between implementation
changes and contract changes; and 4) the role of inheritance in the process. It
has found, among other results, that: the percentage of program elements that
include contracts is above 33% for most projects and tends to be stable over
time; there is no strong preference for a certain type of contract element;
contracts are quite stable compared to implementations; and inheritance does
not significantly affect qualitative trends of contract usage
Mapeo sistemático y estudio de caso sobre técnicas de generación automática de pruebas
Incluye bibliografía.El trabajo tiene como objetivo explorar la generación automática de casos de prueba a través de la identificación de las técnicas y sus problemas e investigar su utilidad práctica. Se realizó un mapeo sistemático de la literatura, extendiendo dos trabajos relacionados a la tesis para conocer las técnicas y los problemas investigados. A partir de este mapeo se realizó un estudio de caso con herramientas de generación para evaluar su eficacia con respecto a la detección de defectos
Evolutionary computing driven search based software testing and correction
For a given program, testing, locating the errors identified, and correcting those errors is a critical, yet expensive process. The field of Search Based Software Engineering (SBSE) addresses these phases by formulating them as search problems. This dissertation addresses these challenging problems through the use of two complimentary evolutionary computing based systems. The first one is the Fitness Guided Fault Localization (FGFL) system, which novelly uses a specification based fitness function to perform fault localization. The second is the Coevolutionary Automated Software Correction (CASC) system, which employs a variety of evolutionary computing techniques to perform testing, correction, and verification of software. In support of the real world application of these systems, a practitioner\u27s guide to fitness function design is provided. For the FGFL system, experimental results are presented that demonstrate the applicability of fitness guided fault localization to automate this important phase of software correction in general, and the potential of the FGFL system in particular. For the fitness function design guide, the performance of a guide generated fitness function is compared to that of an expert designed fitness function demonstrating the competitiveness of the guide generated fitness function. For the CASC system, results are presented that demonstrate the system\u27s abilities on a series of problems of both increasing size as well as number of bugs present. The system presented solutions more than 90% of the time for versions of the programs containing one or two bugs. Additionally, scalability results are presented for the CASC system that indicate that success rate linearly decreases with problem size and that the estimated convergence rate scales at worst linearly with problem size --Abstract, page ii