Semi-automatic Search-Based Test Generation

Abstract—Search-based testing techniques can efficiently generate test data to achieve high code coverage. However, when the fitness function does not provide sufficient guidance, the search will only generate optimal results by chance. Yet, where the search algorithm struggles, a human tester with domain knowledge can often produce solutions easily. We therefore include the tester in the test generation process: When the search stagnates, the tester is given an opportunity to improve the current solution, and these improvements are fed back to the search. In particular, relevant problems occur often when generating tests for object-oriented languages, where test cases are sequences of method calls. Constructing complex objects through sequences of method calls is difficult, and often the traditional branch distance offers little guidance – yet for a human tester the same task is often trivial. In this paper, we present a semi-automatic test generation approach based on our search-based EVOSUITE tool, and evaluate the usefulness and potential on a set of example classes. Keywords-test case generation; search-based testing; manual testing I

Search-based inference of polynomial metamorphic relations

Metamorphic testing (MT) is an effective methodology for testing those so-called 'non-testable' programs (e.g., scientific programs), where it is sometimes very difficult for testers to know whether the outputs are correct. In metamorphic testing, metamorphic relations (MRs) (which specify how particular changes to the input of the program under test would change the output) play an essential role. However, testers may typically have to obtain MRs manually. In this paper, we propose a search-based approach to automatic inference of polynomial MRs for a program under test. In particular, we use a set of parameters to represent a particular class of MRs, which we refer to as polynomial MRs, and turn the problem of inferring MRs into a problem of searching for suitable values of the parameters. We then dynamically analyze multiple executions of the program, and use particle swarm optimization to solve the search problem. To improve the quality of inferred MRs, we further use MR filtering to remove some inferred MRs. We also conducted three empirical studies to evaluate our approach using four scientific libraries (including 189 scientific functions). From our empirical results, our approach is able to infer many high-quality MRs in acceptable time (i.e., from 9.87 seconds to 1231.16 seconds), which are effective in detecting faults with no false detection. ? 2014 ACM.EI

Guiding Random Graphical and Natural User Interface Testing Through Domain Knowledge

Users have access to a diverse set of interfaces that can be used to interact with software. Tools exist for automatically generating test data for an application, but the data required by each user interface is complex. Generating realistic data similar to that of a user is difficult. The environment which an application is running inside may also limit the data available, or updates to an operating system can break support for tools that generate test data. Consequently, applications exist for which there are no automated methods of generating test data similar to that which a user would provide through real usage of a user interface. With no automated method of generating data, the cost of testing increases and there is an increased chance of bugs being released into production code. In this thesis, we investigate techniques which aim to mimic users, observing how stored user interactions can be split to generate data targeted at specific states of an application, or to generate different subareas of the data structure provided by a user interface. To reduce the cost of gathering and labelling graphical user interface data, we look at generating randomised screen shots of applications, which can be automatically labelled and used in the training stage of a machine learning model. These trained models could guide a randomised approach at generating tests, achieving a significantly higher branch coverage than an unguided random approach. However, for natural user interfaces, which allow interaction through body tracking, we could not learn such a model through generated data. We find that models derived from real user data can generate tests with a significantly higher branch coverage than a purely random tester for both natural and graphical user interfaces. Our approaches use no feedback from an application during test generation. Consequently, the models are “generating data in the dark”. Despite this, these models can still generate tests with a higher coverage than random testing, but there may be a benefit to inferring the current state of an application and using this to guide data generation

Mapeo sistemático y estudio de caso sobre técnicas de generación automática de pruebas

Incluye bibliografía.El trabajo tiene como objetivo explorar la generación automática de casos de prueba a través de la identificación de las técnicas y sus problemas e investigar su utilidad práctica. Se realizó un mapeo sistemático de la literatura, extendiendo dos trabajos relacionados a la tesis para conocer las técnicas y los problemas investigados. A partir de este mapeo se realizó un estudio de caso con herramientas de generación para evaluar su eficacia con respecto a la detección de defectos

Enhancing Search-Based Techniques with Information Control Dependencies

Software testing is a very important task during software development and it can be used to improve the quality and reliability of the software system. One potential way to reduce the cost and increase the efficiency of software testing is to generate test data automatically. Search-based approaches successfully generate unit tests for object-oriented programs, like Java. However, challenges, such as the large size of the search space, and the presence of complex predicates’ target branches, negatively affect the approaches, and, thus, cannot achieve high structural coverage for certain programs. The aim of this thesis is to propose enhancement techniques to improve the effectiveness of search based testing approaches and address the challenges posed by the object-oriented programs. Rather than randomly generating a sequence of method calls, our ongoing work is to focus on using static analysis to define the hidden data dependencies on predicates’ target branches (i.e., uncovered branches) and exploit method dependence relations (MDR) approach to precisely identify the method/constructors and its parameters. This method dependence information is employed to reduce the search space and used to guide the search toward regions that lead to full (or at least high) structural coverage