An Extensible Technique for High-Precision Testing of Recovery Code

Thorough testing of software systems requires ways to productively employ fault injection. We describe a technique for automatically identifying the errors exposed by shared libraries, finding good injection targets in program binaries, and producing corresponding injection scenarios. We present a framework for writing precise custom triggers that inject the desired faults--in the form of error return codes and corresponding side effects--at the boundary between shared libraries and applications. We incorporated these ideas in the LFI tool chain. With no developer assistance and no access to source code, this new version of LFI found 11 serious, previously unreported bugs in the BIND name server, the Git version control system, the MySQL database server, and the PBFT replication system. LFI achieved entirely automatically 35%-60% improvement in recovery-code coverage, without requiring any new tests. LFI can be downloaded from http://lfi.epfl.ch

Automating Regression Test Selection for Web Services

As Web services grow in maturity and use, so do the methods which are being used to test and maintain them. Regression Testing is a major component of most major testing systems but has only begun to be applied to Web services. The majority of the tools and techniques applying regression test to Web services are focused on test-case generation, thus ignoring the potential savings of regression test selection. Regression test selection optimizes the regression testing process by selecting a subset of all tests, while still maintaining some level of confidence about the system performing no worse than the unmodified system. A safe regression test selection technique implies that after selection, the level of confidence is as high as it would be if no tests were removed. Since safe regression test selection techniques generally involve code-based (white-box) testing, they cannot be directly applied to Web services due to their loosely-coupled, standards-based, and distributed nature. A framework which automates both the regression test selection and regression testing processes for Web services in a decentralized, end-to-end manner is proposed. As part of this approach, special consideration is given to the concurrency issues which may occur in an autonomous and decentralized system. The resulting synchronization method will be presented along with a set of algorithms which manage the regression testing and regression test selection processes throughout the system. A set of empirical results demonstrate the feasibility and benefit of the approach

Parameterized Object Sensitivity for Points-to Analysis for Java

The goal of points-to analysis for Java is to determine the set of objects pointed to by a reference variable or a reference object field. We present object sensitivity, a new form of context sensitivity for flow-insensitive points-to analysis for Java. The key idea of our approach is to analyze a method separately for each of the object names that represent runtime objects on which this method may be invoked. To ensure flexibility and practicality, we propose a parameterization framework that allows analysis designers to control the tradeo#s between cost and precision in the object-sensitive analysis

Web services robustness testing

Web services are a new paradigm for building software applications that has many advantages over the previous paradigms; however, Web Services are still not widely used because Service Requesters do not trust services that were built by others. Testing can assuage this problem because it can be used to assess the quality attributes of Web Services. This thesis proposes a framework and presents a proof of concept tool that can be used to test the robustness and other related attributes of a Web Service. The tool can be easily enhanced to assess other quality attributes. The framework is based on analyzing Web Services Description Language (WSDL) documents of Web Services to find what faults could affect the robustness quality attributes. After that using these faults to build test case generation rules to assess the robustness quality attribute of Web Services. This framework will give a better understanding of the faults that may affect the robustness quality attribute of Web Services, how these faults are related to the interface or the contract of a Web Service under test, and what testing techniques can be used to detect such faults. The approach used in this thesis for building test cases for Web Services was used with many examples in order to demonstrate its effectiveness; these examples have shown that the approach and the proof of concept tool are able to assess the robustness of Web Services implementation and Web Services platforms. Four hundred and two test clients were automatically built by the tool, based on the test cases rules, to assess the robustness of these Web Services examples. These test clients detected eleven robustness failures in the Web Services implementations and nine robustness failures in the Web Services platforms. Also the approach was able to help in comparing the robustness of two different Web Services platforms, namely Axis and GLUE. After deploying the same Web Services in both of these platforms; Axis showed less robustness and security failures than GLUE

Testing of Java Web Services for Robustness

This paper presents a new compile-time analysis that enables a testing methodology for white-box coverage testing of error recovery code (i.e., exception handlers) in Java web services using compilerdirected fault injection. The analysis allows compiler-generated instrumentation to guide the fault injection and to record the recovery code exercised. (An injected fault is experienced as a Java exception.) The analysis (i) identifies the exception-flow 'def-uses' to be tested in this manner, (ii) determines the kind of fault to be requested at a program point, and (iii) finds appropriate locations for code instrumentation. The analysis incorporates refinements that establish sufficient context sensitivity to ensure relatively precise def-use links and to eliminate some spurious def-uses due to demonstrably infeasible control flow. A runtime test harness calculates test coverage of these links using an exception def-catch metric. Experiments with the methodology demonstrate the utility of the increased precision in obtaining good test coverage on a set of moderately-sized Java web services benchmarks