Towards run-time monitoring of web services conformance to business-level agreements

Web service behaviour is currently specified in a mixture of ways, often using methods that are only partially complete. These range from static functional specifications, based on interfaces in WSDL and preconditions in RIF, to business process simulations using executable process-based models such as BPEL, to detailed quality of service (QoS) agreements laid down in a service level agreement (SLA). This paper recognises that something similar to a SLA is required at the higher business level to govern the contract between service producers, brokers and consumers. We call this a business level agreement (BLA) and within this framework, seek to unify disparate aspects of functional specification, QoS and run-time verification. We propose that the method for validating a web service with respect to its advertised BLA should be based on run-time service monitoring. This is a position paper towards defining these goals

Feedback-Based Specification, Coding and Testing with JWalk

JWalk is a lazy systematic unit-testing tool for Java, which supports dynamic inference of specifications from code and systematic testing from the acquired specification. This paper describes the feedback-based development methodology that is possible using the JWalk Editor, an original Java-sensitive editor and compiler coupled to JWalk, which helps programmers to prototype Java class designs, generating novel test cases as they code. Systematic exploratory testing alerts the programmer to unusual consequences in the design; and confirmed test results become part of the evolving specification, which adapts continuously to modified classes and extends to subclasses. The cycle of coding, inferring and testing systematically exposes test cases that are often missed in other test-driven development approaches, which rely on programmer intuition to create test cases

A meta-analysis approach to refactoring and XP

The mechanics of seventy-two different Java refactorings are described fully in Fowler's text. In the same text, Fowler describes seven categories of refactoring, into which each of the seventy-two refactorings can be placed. A current research problem in the refactoring and XP community is assessing the likely time and testing effort for each refactoring, since any single refactoring may use any number of other refactorings as part of its mechanics and, in turn, can be used by many other refactorings. In this paper, we draw on a dependency analysis carried out as part of our research in which we identify the 'Use' and 'Used By' relationships of refactorings in all seven categories. We offer reasons why refactorings in the 'Dealing with Generalisation' category seem to embrace two distinct refactoring sub-categories and how refactorings in the 'Moving Features between Objects' category also exhibit specific characteristics. In a wider sense, our meta-analysis provides a developer with concrete guidelines on which refactorings, due to their explicit dependencies, will prove problematic from an effort and testing perspective

Dynamic Analysis of Algebraic Structure to Optimize Test Generation and Test Case Selection

Where no independent specification is available, object-oriented unit testing is limited to exercising all interleaved method paths, seeking unexpected failures. A recent trend in unit testing, that interleaves dynamic analysis between each test cycle, has brought useful reductions in test-set sizes by pruning redundant prefix paths. This paper describes a dynamic approach to analyzing the algebraic structure of test objects, such that prefix paths ending in observer or transformer operations yielding unchanged, or derived states may be detected and pruned on-the-fly during testing. The fewer retained test cases are so close to the ideal algebraic specification cases that a tester can afford to confirm or reject these cases interactively, which are then used as a test oracle to predict many further test outcomes during automated testing. The algebra-inspired algorithms are incorporated in the latest version of the JWalk lazy systematic unit testing tool suite, which discovers key test cases, while pruning many thousands of redundant test cases

What makes testing work: Nine case studies of software development teams

Recently there has been a focus on test first and test driven development; several empirical studies have tried to assess the advantage that these methods give over testing after development. The results have been mixed. In this paper we investigate nine teams who tested during coding to examine the effect it had on the external quality of their code. Of the top three performing teams two used a documented testing strategy and the other an ad-hoc approach to testing. We conclude that their success appears to be related to a testing culture where the teams proactively test rather than carry out only what is required in a mechanical fashion. © 2009 IEEE

Angels and monsters: An empirical investigation of potential test effectiveness and efficiency improvement from strongly subsuming higher order mutation

We study the simultaneous test effectiveness and efficiency improvement achievable by Strongly Subsuming Higher Order Mutants (SSHOMs), constructed from 15,792 first order mutants in four Java programs. Using SSHOMs in place of the first order mutants they subsume yielded a 35%-45% reduction in the number of mutants required, while simultaneously improving test efficiency by 15% and effectiveness by between 5.6% and 12%. Trivial first order faults often combine to form exceptionally non-trivial higher order faults; apparently innocuous angels can combine to breed monsters. Nevertheless, these same monsters can be recruited to improve automated test effectiveness and efficiency

Higher Order Mutation Testing

Mutation testing is a fault-based software testing technique that has been studied widely for over three decades. To date, work in this field has focused largely on first order mutants because it is believed that higher order mutation testing is too computationally expensive to be practical. This thesis argues that some higher order mutants are potentially better able to simulate real world faults and to reveal insights into programming bugs than the restricted class of first order mutants. This thesis proposes a higher order mutation testing paradigm which combines valuable higher order mutants and non-trivial first order mutants together for mutation testing. To overcome the exponential increase in the number of higher order mutants a search process that seeks fit mutants (both first and higher order) from the space of all possible mutants is proposed. A fault-based higher order mutant classification scheme is introduced. Based on different types of fault interactions, this approach classifies higher order mutants into four categories: expected, worsening, fault masking and fault shifting. A search-based approach is then proposed for locating subsuming and strongly subsuming higher order mutants. These mutants are a subset of fault mask and fault shift classes of higher order mutants that are more difficult to kill than their constituent first order mutants. Finally, a hybrid test data generation approach is introduced, which combines the dynamic symbolic execution and search based software testing approaches to generate strongly adequate test data to kill first and higher order mutants

How validation can help in testing business processes orchestrating web services

Validation and testing are important in developing correct and fault free SOA-basedsystems. BPEL is a high level language that makes it possible to implement business processes asan orchestration of web services. In general, the testing requires much more test scenarios than thevalidation. However, in the case of BPEL processes, which have very simple and well structuredimplementation, test scenarios limited to the validation may also be efficient. The paper describes anexperiment that aims at answering a question whether or not the validation test scenarios are alsoadequate for testing an implementation of BPEL processes. The experiment employs a Software FaultInjector for BPEL Processes that is able to inject faults when the test scenarios are running. Theresults of the experiment seem very promising. Hence, it seems that validation tests might give astrong support for testing

Implementación de pruebas del sistema. Un caso práctico

Las pruebas funcionales del sistema permiten verificar que el sistema en desarrollo satisface sus requisitos funcionales. Existen una amplia cantidad de trabajos y capítulos de libros que proponen cómo obtener objetivos de prueba a partir de requisitos funcionales expresados como casos de uso. Sin embargo, existe una carencia de trabajos que muestren cómo implementar dichos objetivos en pruebas automáticas. Este trabajo presenta un ejemplo, basado en el perfil de pruebas de UML, para la implementación en código ejecutable de objetivos de prueba definidos mediante escenarios y variables operacionale

Generación automática de objetivos de prueba a partir de casos de uso mediante partición de categorías y variables operacionales

Este trabajo complementa y amplia nuestros trabajos anteriores sobre generación de pruebas a partir de casos de uso presentando un proceso que, de manera sistemática y automática, permite generar objetivos de prueba a partir de casos de uso especificados en un lenguaje no formal. Este proceso aplica el método de categoría-partición y el patrón Use Case Test Pattern, el cual usa variables operacionales. Además se presenta los algoritmos necesarios para la automatización del proceso propuesto, acompañados de un caso práctico