Automated Blackbox and Whitebox Testing of RESTful APIs with EvoMaster

© 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.We apply EvoMaster to eight representational state transfer application programming interfaces; show how the tool can be used to automatically generate test cases that can find several bugs, even when using a naive black-box approach; and discuss challenges that must be taken into account.acceptedVersio

Enhancing Resource-Based Test Case Generation for RESTful APIs with SQL Handling

Nowadays, many companies use RESTful web services to develop their enterprise applications. These web services typically interact with databases. In REST, resource handling is a fundamental concept, where resources are manipulated by exposing HTTP endpoints. Rd-MIO* is an evolutionary algorithm which is specialized in test generation for such kind of services, i.e., RESTful APIs, via manipulating resources in various ways using HTTP actions (e.g., GET and POST). In this paper, we further extended Rd-MIO* by employing SQL commands to manipulate the resources for test generation, directly into the databases. We implemented our novel technique as an extension of the EvoMaster tool. To evaluate our approach, we selected Rd-MIO* as a baseline technique and conducted an empirical study with five open source REST APIs. Results showed that our approach clearly outperforms the baseline over all of the five case studies.publishedVersio

Adaptive Hypermutation for Search-Based System Test Generation: A Study on REST APIs with EvoMaster

REST web services are widely popular in industry, and search techniques have been successfully used to automatically generate system-level test cases for those systems. In this article, we propose a novel mutation operator which is designed specifically for test generation at system-level, with a particular focus on REST APIs. In REST API testing, and often in system testing in general, an individual can have a long and complex chromosome. Furthermore, there are two specific issues: (1) fitness evaluation in system testing is highly costly compared with the number of objectives (e.g., testing targets) to optimize for; and (2) a large part of the genotype might have no impact on the phenotype of the individuals (e.g., input data that has no impact on the execution flow in the tested program). Due to these issues, it might be not suitable to apply a typical low mutation rate like 1/n (where n is the number of genes in an individual), which would lead to mutating only one gene on average. Therefore, in this article, we propose an adaptive weight-based hypermutation, which is aware of the different characteristics of the mutated genes. We developed adaptive strategies that enable the selection and mutation of genes adaptively based on their fitness impact and mutation history throughout the search. To assess our novel proposed mutation operator, we implemented it in the EvoMaster tool, integrated in the MIO algorithm, and further conducted an empirical study with three artificial REST APIs and four real-world REST APIs. Results show that our novel mutation operator demonstrates noticeable improvements over the default MIO. It provides a significant improvement in performance for six out of the seven case studies, where the relative improvement is up to +12.09% for target coverage, +12.69% for line coverage, and +32.51% for branch coverage.publishedVersio

Recent Trends in Software Testing Education: A Systematic Literature Review

Testing is a critical aspect of software development. Far too often software is released with critical faults. However, testing is often considered tedious and boring. Unfortunately, many graduates might join the work force without having had any education in software testing, which exacerbates the problem even further. Therefore, teaching software testing as part of a university degree in software engineering and is very important. But it is an open challenge how to teach software testing in an effective way that can successfully motivate students. In this paper, we have carried out a systematic literature review on the topic of teaching software testing. We analysed and reviewed 30 papers that were published between 2013 and 2017. The review points out to a few different trends, like the use of gamification to make the teaching of software testing less tedious

Enhancing Search-based Testing with Testability Transformations for Existing APIs

Search-based software testing (SBST) has been shown to be an effective technique to generate test cases automatically. Its effectiveness strongly depends on the guidance of the fitness function. Unfortunately, a common issue in SBST is the so-called flag problem, where the fitness landscape presents a plateau that provides no guidance to the search. In this paper, we provide a series of novel testability transformations aimed at providing guidance in the context of commonly used API calls (e.g., strings that need to be converted into valid date/time objects). We also provide specific transformations aimed at helping the testing of REST Web Services. We implemented our novel techniques as an extension to EvoMaster, a SBST tool that generates system level test cases. Experiments on nine open-source REST web services, as well as an industrial web service, show that our novel techniques improve performance significantlyacceptedVersio

JavaScript Instrumentation for Search-Based Software Testing: A Study with RESTful APIs

acceptedVersio

JavaScript SBST Heuristics to Enable Effective Fuzzing of NodeJS Web APIs

JavaScript is one of the most popular programming languages. However, its dynamic nature poses several challenges to automated testing techniques. In this paper, we propose an approach and open-source tool support to enable white-box testing of JavaScript applications using Search-Based Software Testing (SBST) techniques. We provide an automated approach to collect search-based heuristics like the common Branch Distance and to enable Testability Transformations. To empirically evaluate our results, we integrated our technique into the EvoMaster test generation tool, and carried out analyses on the automated system testing of RESTful and GraphQL APIs. Experiments on eight Web APIs running on NodeJS show that our technique leads to significantly better results than existing black-box and grey-box testing tools, in terms of code coverage and fault detection.publishedVersio