Automated Test Generation for REST APIs: No Time to Rest Yet

Modern web services routinely provide REST APIs for clients to access their functionality. These APIs present unique challenges and opportunities for automated testing, driving the recent development of many techniques and tools that generate test cases for API endpoints using various strategies. Understanding how these techniques compare to one another is difficult, as they have been evaluated on different benchmarks and using different metrics. To fill this gap, we performed an empirical study aimed to understand the landscape in automated testing of REST APIs and guide future research in this area. We first identified, through a systematic selection process, a set of 10 state-of-the-art REST API testing tools that included tools developed by both researchers and practitioners. We then applied these tools to a benchmark of 20 real-world open-source RESTful services and analyzed their performance in terms of code coverage achieved and unique failures triggered. This analysis allowed us to identify strengths, weaknesses, and limitations of the tools considered and of their underlying strategies, as well as implications of our findings for future research in this area.Comment: 13 pages, 6 figures, In Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) 202

Fault injection testing method of software implemented fault tolerance mechanisms of web service systems

Testing Web Services applications and their Fault Tolerance Mechanisms (FTMs) is crucial for the development of today's applications. The performance and FTMs of composed service systems are hard to measure at design time because service instability is often caused by the nature of the network. Testing in a real internet environment is difficult to set up and control. However, the adequacy of FTMs and the performance of Web Service applications can be tested efficiently by injecting faults and observing how the target system performs under faulty conditions. This thesis investigates what is involved in testing the software-implemented fault tolerance mechanisms of Web Service systems through fault injection. We have developed a fault injection toolkit that emulates a WAN within a LAN environment between composed service components and offers full control over the emulated environments, in addition to the ability to inject communication and specific software faults. The tool also generates background workloads on the tested system for producing more realistic results. The testing method requires that the target system be constructed as a collection of Web Services applications interacting via messages. This enables the insertion of faults into the target system to emulate the incorrect behaviour of faulty conditions by injecting communication faults and manipulating messages. This approach allows the injection of faults while not requiring any significant changes to the target system. This testing method injects two classes of faults, manly communication and interface faults due to their big impact on Web service system dependability. The method differs from the previous work not only by injecting communication faults based on a Wide Area Network emulator, but also in its ability to inject a combination of communication and interface faults, which could cause what are called Byzantine faults (Arbitrary faults) at the application level. The proposed fault injection method has been applied to test a Web Service system deploying what is called a WS-Mediator for improving the system reliability. The WS-Mediator claims to offer comprehensive off-the-shelf fault tolerance mechanisms to cope with various kinds of typical Web Service application scenarios. We chose to use the N-version programming mechanism offered by the WS-Mediator, which has been tested through out tool. The testing demonstrated the usefulness of the method and its capacity to test the target system under different circumstances and faulty conditions.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Automated Realistic Test Input Generation and Cost Reduction in Service-centric System Testing

Service-centric System Testing (ScST) is more challenging than testing traditional software due to the complexity of service technologies and the limitations that are imposed by the SOA environment. One of the most important problems in ScST is the problem of realistic test data generation. Realistic test data is often generated manually or using an existing source, thus it is hard to automate and laborious to generate. One of the limitations that makes ScST challenging is the cost associated with invoking services during testing process. This thesis aims to provide solutions to the aforementioned problems, automated realistic input generation and cost reduction in ScST. To address automation in realistic test data generation, the concept of Service-centric Test Data Generation (ScTDG) is presented, in which existing services used as realistic data sources. ScTDG minimises the need for tester input and dependence on existing data sources by automatically generating service compositions that can generate the required test data. In experimental analysis, our approach achieved between 93% and 100% success rates in generating realistic data while state-of-the-art automated test data generation achieved only between 2% and 34%. The thesis addresses cost concerns at test data generation level by enabling data source selection in ScTDG. Source selection in ScTDG has many dimensions such as cost, reliability and availability. This thesis formulates this problem as an optimisation problem and presents a multi-objective characterisation of service selection in ScTDG, aiming to reduce the cost of test data generation. A cost-aware pareto optimal test suite minimisation approach addressing testing cost concerns during test execution is also presented. The approach adapts traditional multi-objective minimisation approaches to ScST domain by formulating ScST concerns, such as invocation cost and test case reliability. In experimental analysis, the approach achieved reductions between 69% and 98.6% in monetary cost of service invocations during testin

Enterprise Computing: Band 1Einführung in z/OS: Band 2z/OS Internet Integration: Band 3Praktische Übungen unter z/OS

Das vorliegende Buch entstand aus einer zweisemestrigen Vorlesung „Enterprise Computing“, die wir gemeinsam über viele Jahre als Teil des Bachelor- oder Master-Studienganges an der Universität Leipzig gehalten haben. Das Buch führt ein in die Welt des Mainframe und soll dem Leser einen einführenden Überblick geben. Band 1 ist der Einführung in z/OS gewidmet, während sich Band 2 mit der Internet Integration beschäftigt. Ergänzend werden in Band 3 praktische Übungen unter z/OS dargestellt

Generating mock skeletons for lightweight Web service testing : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Computer Science at Massey University, Manawatū New Zealand

Modern application development allows applications to be composed using lightweight HTTP services. Testing such an application requires the availability of services that the application makes requests to. However, continued access to dependent services during testing may be restrained, making adequate testing a significant and non-trivial engineering challenge. The concept of Service Virtualisation is gaining popularity for testing such applications in isolation. It is a practise to simulate the behaviour of dependent services by synthesising responses using semantic models inferred from recorded traffic. Replacing services with their respective mocks is, therefore, useful to address their absence and move on application testing. In reality, however, it is unlikely that fully automated service virtualisation solutions can produce highly accurate proxies. Therefore, we recommend using service virtualisation to infer some attributes of HTTP service responses. We further acknowledge that engineers often want to fine-tune this. This requires algorithms to produce readily interpretable and customisable results. We assume that if service virtualisation is based on simple logical rules, engineers would have the potential to understand and customise rules. In this regard, Symbolic Machine Learning approaches can be investigated because of the high provenance of their results. Accordingly, this thesis examines the appropriateness of symbolic machine learning algorithms to automatically synthesise HTTP services' mock skeletons from network traffic recordings. We consider four commonly used symbolic techniques: the C4.5 decision tree algorithm, the RIPPER and PART rule learners, and the OCEL description logic learning algorithm. The experiments are performed employing network traffic datasets extracted from a few different successful, large-scale HTTP services. The experimental design further focuses on the generation of reproducible results. The chosen algorithms demonstrate the suitability of training highly accurate and human-readable semantic models for predicting the key aspects of HTTP service responses, such as the status and response headers. Having human-readable logics would make interpretation of the response properties simpler. These mock skeletons can then be easily customised to create mocks that can generate service responses suitable for testing

Data-Driven Detection and Diagnosis of System-Level Failures in Middleware-Based Service Compositions

Service-oriented technologies have simplified the development of large, complex software systems that span administrative boundaries. Developers have been enabled to build applications as compositions of services through middleware that hides much of the underlying complexity. The resulting applications inhabit complex, multi-tier operating environments that pose many challenges to their reliable operation and often lead to failures at runtime. Two key aspects of the time to repair a failure are the time to its detection and to the diagnosis of its cause. The prevalent approach to detection and diagnosis is primarily based on ad-hoc monitoring as well as operator experience and intuition. This is inefficient and leads to decreased availability. We propose an approach to data-driven detection and diagnosis in order to decrease the repair time of failures in middleware-based service compositions. Data-driven diagnosis supports system operators with information about the operation and structure of a service composition. We discuss how middleware-based service compositions can be monitored in a comprehensive, yet non-intrusive manner and present a process to discover system structure by processing deployment information that is commonly reified in such systems. We perform a controlled experiment that compares the performance of 22 participants using either a standard or the data-driven approach to diagnose several failures injected into a real-world service composition. We find that system operators using the latter approach are able to achieve significantly higher success rates and lower diagnosis times. Data-driven detection is based on the automation of failure detection through applying an outlier detection technique to multi-variate monitoring data. We evaluate the effectiveness of one-class classification for this purpose and determine a simple approach to select subsets of metrics that afford highly accurate failure detection