Finding a boundary between valid and invalid regions of the input space

In the context of robustness testing, the boundary between the valid and invalid regions of the input space can be an interesting source of erroneous inputs. Knowing where a specific software under test (SUT) has a boundary is essential for validation in relation to requirements. However, finding where a SUT actually implements the boundary is a non-trivial problem that has not gotten much attention. This paper proposes a method of finding the boundary between the valid and invalid regions of the input space. The proposed method consists of two steps. First, test data generators, directed by a search algorithm to maximise distance to known, valid test cases, generate valid test cases that are closer to the boundary. Second, these valid test cases undergo mutations to try to push them over the boundary and into the invalid part of the input space. This results in a pair of test sets, one consisting of test cases on the valid side of the boundary and a matched set on the outer side, with only a small distance between the two sets. The method is evaluated on a number of examples from the standard library of a modern programming language. We propose a method of determining the boundary between valid and invalid regions of the input space and apply it on a SUT that has a non-contiguous valid region of the input space. From the small distance between the developed pairs of test sets, and the fact that one test set contains valid test cases and the other invalid test cases, we conclude that the pair of test sets described the boundary between the valid and invalid regions of that input space. Differences of behaviour can be observed between different distances and sets of mutation operators, but all show that the method is able to identify the boundary between the valid and invalid regions of the input space. This is an important step towards more automated robustness testing.Comment: 10 pages, conferenc

A Testability Analysis Framework for Non-Functional Properties

This paper presents background, the basic steps and an example for a testability analysis framework for non-functional properties

Transferring Interactive Search-Based Software Testing to Industry

Search-Based Software Testing (SBST) is the application of optimization algorithms to problems in software testing. In previous work, we have implemented and evaluated Interactive Search-Based Software Testing (ISBST) tool prototypes, with a goal to successfully transfer the technique to industry. While SBSE solutions are often validated on benchmark problems, there is a need to validate them in an operational setting. The present paper discusses the development and deployment of SBST tools for use in industry and reflects on the transfer of these techniques to industry. In addition to previous work discussing the development and validation of an ISBST prototype, a new version of the prototype ISBST system was evaluated in the laboratory and in industry. This evaluation is based on an industrial System under Test (SUT) and was carried out with industrial practitioners. The Technology Transfer Model is used as a framework to describe the progression of the development and evaluation of the ISBST system. The paper presents a synthesis of previous work developing and evaluating the ISBST prototype, as well as presenting an evaluation, in both academia and industry, of that prototype's latest version. This paper presents an overview of the development and deployment of the ISBST system in an industrial setting, using the framework of the Technology Transfer Model. We conclude that the ISBST system is capable of evolving useful test cases for that setting, though improvements in the means the system uses to communicate that information to the user are still required. In addition, a set of lessons learned from the project are listed and discussed. Our objective is to help other researchers that wish to validate search-based systems in industry and provide more information about the benefits and drawbacks of these systems.Comment: 40 pages, 5 figure

EvoMaster: A Search-Based System Test Generation Tool

Testing web/enterprise applications is complex and expensive when done manually. Often, software testing takes up to half of the development time and cost for a system. So much testing is needed because the cost of software failure is simply too large: for example, in 2017, 304 software failures (reported in the media) impacted 3.6 billion people and $1.7 trillion in assets worldwide (Tricentis, 2017). Unfortunately, due to its high cost, software testing is often left incomplete, and only applied partially.To address this problem, in Software Engineering (SE) research a lot of effort has been spent in trying to design and implement novel techniques aimed at automating several different tasks, where software testing is among the most studied tasks. Search-Based Software Test- ing (SBST) (Harman et al., 2012) casts the problem of software testing as an optimization problem, aimed at, for example, maximizing code coverage and fault detection.Our SBST tool called EvoMaster addresses these challenges by using evolutionary techniques to automatically generate test cases. It currently focuses on RESTful web services, which are the pillars of modern web and enterprise applications (Allamaraju, 2010; Fielding, 2000).Fil: Arcuri, Andrea. Kristiania University College; NoruegaFil: Galeotti, Juan Pablo. Consejo Nacional de Investigaciones Científicas y Técnicas. Oficina de Coordinación Administrativa Ciudad Universitaria. Instituto de Investigación en Ciencias de la Computación. Universidad de Buenos Aires. Facultad de Ciencias Exactas y Naturales. Instituto de Investigación en Ciencias de la Computación; Argentina. Universidad de Buenos Aires. Facultad de Ciencias Exactas y Naturales. Departamento de Computación; ArgentinaFil: Marculescu, Bogdan. Kristiania University College; NoruegaFil: Zhang, Man. Kristiania University College; Norueg

Building an open-source system test generation tool: lessons learned and empirical analyses with EvoMaster

Research in software testing often involves the development of software prototypes. Like any piece of software, there are challenges in the development, use and verification of such tools. However, some challenges are rather specific to this problem domain. For example, often these tools are developed by PhD students straight out of bachelor/master degrees, possibly lacking any industrial experience in software development. Prototype tools are used to carry out empirical studies, possibly studying different parameters of novel designed algorithms. Software scaffolding is needed to run large sets of experiments efficiently. Furthermore, when using AI-based techniques like evolutionary algorithms, care needs to be taken to deal with their randomness, which further complicates their verification. The aforementioned represent some of the challenges we have identified for this domain. In this paper, we report on our experience in building the open-source EvoMaster tool, which aims at system-level test case generation for enterprise applications. Many of the challenges we faced would be common to any researcher needing to build software testing tool prototypes. Therefore, one goal is that our shared experience here will boost the research community, by providing concrete solutions to many development challenges in the building of such kind of research prototypes. Ultimately, this will lead to increase the impact of scientific research on industrial practice.publishedVersio

Interactive Search-Based Testing of Embedded Software: Understanding the Benefits and Difficulties of Industrial Application

The ubiquity of software has wide-ranging consequences for its development and testing. Increasingly often, software is developed and tested by engineers specialized in other areas. Embedded software, for example, is developed ad-hoc, for each product, by systems and domain engineers. Supporting testing activities in this context requires a highly flexible approach, powerful enough to create useful test cases, yet simple enough to not require specialized training in software testing. Search-based software testing promises the ability to generate and evaluate large numbers of test cases at minimal cost. It is, however, a set of complex techniques that cannot be used off-the-shelf as part of the software development process of a company. The objective of the work presented in this thesis is to investigate the applicability of Search-Based Software Testing in an industrial environment. A second objective was identifying additional knowledge gaps relating to integrating such techniques in existing software development processes. Investigating how meaningful interaction is to take place, what information is needed in order for both stakeholders to be able to achieve their objectives is a third goal. The findings are obtained by means of a series of case studies in a company developing both embedded software and the tools to enable embedded software development. A prototype Interactive Search-Based Software Testing (ISBST) system was developed that uses user interaction to drive the search-based component towards interesting test cases. The ISBST system was evaluated constantly, and improved based on the findings of each case study. The latest case study was an empirical evaluation of the system with the engineers, both software engineers and domain specialists, in the company. The empirical work includes both qualitative and quantitative data, with a focus on the exploratory study of the practical factors affecting the use of the ISBST system. A key early finding is that interactivity is essential when implementing search-based techniques in the industrial context described above. Repeated validations conducted with the company yielded additional information on the practicalities of interaction. The strength of SBST proved useful in investigating areas of the test space that were normally overlooked due to limitations in terms of resources. At the same time, developers were able to use their experience and intuition to guide the SBST system towards test cases that were more likely to be problematic. Overall, the results obtained indicate that the search-based techniques provide a useful complement to existing testing techniques. SBST, in its interactive form, can be a useful complement to existing testing techniques. An Interactive SBST (ISBST) system has been developed as a result of this research. Results show that this system is usable by the developers of embedded software, that often specialize on acquiring domain knowledge rather than software engineering expertise