Automating property-based testing of evolving web services

Web services are the most widely used service technology that drives the Service-Oriented Computing~(SOC) paradigm. As a result, effective testing of web services is getting increasingly important. In this paper, we present a framework and toolset for testing web services and for evolving test code in sync with the evolution of web services. Our approach to testing web services is based on the Erlang programming language and QuviQ QuickCheck, a property-based testing tool written in Erlang, and our support for test code evolution is added to Wrangler, the Erlang refactoring tool. The key components of our system include the automatic generation of initial test code, the inference of web service interface changes between versions, the provision of a number of domain specific refactorings and the automatic generation of refactoring scripts for evolving the test code. Our framework provides users with a powerful and expressive web service testing framework, while minimising users' effort in creating, maintaining and evolving the test model. The framework presented in this paper can be used by both web service providers and consumers, and can be used to test web services written in whatever language; the approach advocated here could also be adopted in other property-based testing frameworks and refactoring tools

Towards property-based testing of RESTful web services

Developing APIs as Web Services over HTTP implies adding an extra layer to software, compared to the ones that we would need to develop an API distributed as, for example, a library. This additional layer must be included in testing too, but this implies that the software under test has an additional complexity due both to the need to use an intermediate protocol in tests and to the need to test compliance with the constraints imposed by that protocol: in this case the constraints defined by the REST architectural style. On the other hand, these requirements are common to all the Web Services, and because of that, we should be able to abstract this aspect of the testing model so that we can reuse it in testing any Web Service. In this paper, as a first step towards automating the testing of Web Services over HTTP, we describe a practical mechanism and model for testing RESTful Web Services without side effects and give an example of how we successfully adapted that mechanism to test two different existing Web Services: Storage Room by Thriventures and Google Tasks by Google. For this task we have used Erlang together with state machine models in the property-based testing tool Quviq QuickCheck, implemented using the statem module. 1

Leveraging Semantic Web Service Descriptions for Validation by Automated Functional Testing

Recent years have seen the utilisation of Semantic Web Service descriptions for automating a wide range of service-related activities, with a primary focus on service discovery, composition, execution and mediation. An important area which so far has received less attention is service validation, whereby advertised services are proven to conform to required behavioural specifications. This paper proposes a method for validation of service-oriented systems through automated functional testing. The method leverages ontology-based and rule-based descriptions of service inputs, outputs, preconditions and effects (IOPE) for constructing a stateful EFSM specification. The specification is subsequently utilised for functional testing and validation using the proven Stream X-machine (SXM) testing methodology. Complete functional test sets are generated automatically at an abstract level and are then applied to concrete Web services, using test drivers created from the Web service descriptions. The testing method comes with completeness guarantees and provides a strong method for validating the behaviour of Web services

Web services robustness testing

Web services are a new paradigm for building software applications that has many advantages over the previous paradigms; however, Web Services are still not widely used because Service Requesters do not trust services that were built by others. Testing can assuage this problem because it can be used to assess the quality attributes of Web Services. This thesis proposes a framework and presents a proof of concept tool that can be used to test the robustness and other related attributes of a Web Service. The tool can be easily enhanced to assess other quality attributes. The framework is based on analyzing Web Services Description Language (WSDL) documents of Web Services to find what faults could affect the robustness quality attributes. After that using these faults to build test case generation rules to assess the robustness quality attribute of Web Services. This framework will give a better understanding of the faults that may affect the robustness quality attribute of Web Services, how these faults are related to the interface or the contract of a Web Service under test, and what testing techniques can be used to detect such faults. The approach used in this thesis for building test cases for Web Services was used with many examples in order to demonstrate its effectiveness; these examples have shown that the approach and the proof of concept tool are able to assess the robustness of Web Services implementation and Web Services platforms. Four hundred and two test clients were automatically built by the tool, based on the test cases rules, to assess the robustness of these Web Services examples. These test clients detected eleven robustness failures in the Web Services implementations and nine robustness failures in the Web Services platforms. Also the approach was able to help in comparing the robustness of two different Web Services platforms, namely Axis and GLUE. After deploying the same Web Services in both of these platforms; Axis showed less robustness and security failures than GLUE

Metamorphic testing for web services: Framework and a case study

Service Oriented Architecture (SOA) has become a major application development paradigm. As a basic unit of SOA applications, Web services significantly affect the quality of the applications constructed from them. Since the development and consumption of Web services are completely separated under SOA environment, the consumers are normally provided with limited knowledge of the services and thus have little information about test oracles. The lack of source code and the restricted control of Web services limit the testability of Web services. To address the prominent oracle problem when testing Web services, we propose a metamorphic testing framework for Web services taking into account the unique features of SOA. We conduct a case study where the new metamorphic testing framework is employed to test a Web service that implements the electronic payment. The results of case study show the feasibility of the framework for web services, and also the efficiency of metamorphic testing. The work presented in the paper alleviates the test oracle problem when testing Web services under SOA

GRAIL – Grid Access and Instrumentation Tool

Since the release of Globus Toolkit 4 Web services enrich the world of Grid Computing. They provide methods to develop modular Grid applications which can be parallelized easily. The access to Web services is mostly solved by complex command line tools which need a good deal of knowledge of the underlaying Grid technologies. GRAIL is intended to fill the gap between existing Grid access methods and both the developer who wants to utilize the Grid for own developments and the user who wants to access the Grid without much additional knowledge. It simplifies the access and the testing of Web services for the Globus Grid middleware. GRAIL provides an easy to use graphical user interface for executing Web services and enables the user to construct complex relationships between services to realize parallel execution. The underlying framework allows an easy integration of any Web service or other arbitrary task without much additional effort for the developer. Existing technologies, shipped with the Globus Toolkit, are seamlessly integrated into GRAIL

Cloud based testing of business applications and web services

This paper deals with testing of applications based on the principles of cloud computing. It is aimed to describe options of testing business software in clouds (cloud testing). It identifies the needs for cloud testing tools including multi-layer testing; service level agreement (SLA) based testing, large scale simulation, and on-demand test environment. In a cloud-based model, ICT services are distributed and accessed over networks such as intranet or internet, which offer large data centers deliver on demand, resources as a service, eliminating the need for investments in specific hardware, software, or on data center infrastructure. Businesses can apply those new technologies in the contest of intellectual capital management to lower the cost and increase competitiveness and also earnings. Based on comparison of the testing tools and techniques, the paper further investigates future trend of cloud based testing tools research and development. It is also important to say that this comparison and classification of testing tools describes a new area and it has not yet been done

Worst-input mutation approach to web services vulnerability testing based on SOAP messages

The growing popularity and application of Web services have led to an increase in attention to the vulnerability of software based on these services. Vulnerability testing examines the trustworthiness, and reduces the security risks of software systems, however such testing of Web services has become increasing challenging due to the cross-platform and heterogeneous characteristics of their deployment. This paper proposes a worst-input mutation approach for testing Web service vulnerability based on SOAP (Simple Object Access Protocol) messages. Based on characteristics of the SOAP messages, the proposed approach uses the farthest neighbor concept to guide generation of the test suite. The test case generation algorithm is presented, and a prototype Web service vulnerability testing tool described. The tool was applied to the testing of Web services on the Internet, with experimental results indicating that the proposed approach, which found more vulnerability faults than other related approaches, is both practical and effective