Quality metrics for mutation testing with applications to WS-BPEL compositions

Mutation testing is a successful testing technique based on fault injection. However, it can be very costly, and several cost-reduction techniques for reducing the number of mutants have been proposed in the literature. Cost reduction can be aided by an analysis of mutation operators, but this requires the definition of specialized metrics. Several metrics have been proposed before, although their effectiveness and relative merits are not easy to assess. A step ahead in the evaluation of mutation-reduction techniques would be a better metric to determine objectively the quality of a set of mutants with respect to a given test suite. This work introduces such a metric, which is naturally extended to mutation operators and may be used to reduce the number of mutants, particularly of equivalent mutants. Finally, a firm mutation analysis tool for WS-BPEL service compositions is presented, and experimental results obtained by comparing different metrics on several compositions are presented

An empirical study on mutation testing of WS-BPEL programs

Nowadays, applications are increasingly deployed as Web services in the globally distributed cloud computing environment. Multiple services are normally composed to fulfill complex functionalities. Business Process Execution Language for Web Services (WS-BPEL) is an XML-based service composition language that is used to define a complex business process by orchestrating multiple services. Compared with traditional applications, WS-BPEL programs pose many new challenges to the quality assurance, especially testing, of service compositions. A number of techniques have been proposed for testing WS-BPEL programs, but only a few studies have been conducted to systematically evaluate the effectiveness of these techniques. Mutation testing has been widely acknowledged as not only a testing method in its own right but also a popular technique for measuring the fault-detection effectiveness of other testing methods. Several previous studies have proposed a family of mutation operators for generating mutants by seeding various faults into WS-BPEL programs. In this study, we conduct a series of empirical studies to evaluate the applicability and effectiveness of various mutation operators for WS-BPEL programs. The experimental results provide insightful and comprehensive guidance for mutation testing of WS-BPEL programs in practice. In particular, our work is the systematic study in the selection of effective mutation operators specifically for WS-BPEL programs

Automated Testing of WS-BPEL Service Compositions: A Scenario-Oriented Approach

Nowadays, Service Oriented Architecture (SOA) has become one mainstream paradigm for developing distributed applications. As the basic unit in SOA, Web services can be composed to construct complex applications. The quality of Web services and their compositions is critical to the success of SOA applications. Testing, as a major quality assurance technique, is confronted with new challenges in the context of service compositions. In this paper, we propose a scenario-oriented testing approach that can automatically generate test cases for service compositions. Our approach is particularly focused on the service compositions specified by Business Process Execution Language for Web Services (WS-BPEL), a widely recognized executable service composition language. In the approach, a WS-BPEL service composition is first abstracted into a graph model; test scenarios are then derived from the model; finally, test cases are generated according to different scenarios. We also developed a prototype tool implementing the proposed approach, and an empirical study was conducted to demonstrate the applicability and effectiveness of our approach. The experimental results show that the automatic scenario-oriented testing approach is effective in detecting many types of faults seeded in the service compositions

A tool for domain-independent model mutation

Mutation is a systematic technique to create variants of a seed artefact by means of mutation operators. It has many applications in computer science, like software testing, automatic exercise generation and design space exploration. Typically, mutation frameworks are developed ad-hoc by implementing mutation operators and their application strategies from scratch, using general-purpose programming languages. However, this is costly and error-prone. To improve this situation, we propose WODEL: a domain-specific language and tool for model-based mutation that is independent of the domain meta-model. WODELenables the rapid development and application of model mutations. It provides built-in advanced functionalities like automatic generation of seed models, and static and dynamic metrics of operator coverage and applicability. It offers extension points, e.g., to post-process mutants and describe domain-specific equivalence criteria. As an example, we illustrate the usage of WODEL for the mutation of security policies, and present an empirical evaluation of its expressiveness.Work partially funded by project FLEXOR (Spanish MINECO, TIN2014- 52129-R), project DArDOS (Spanish MINECO/FEDER TIN2015-65845-C3-1- R) and the R&D programme of the Madrid Region (S2013/ICE-3006

Using Evolutionary Mutation Testing to improve the quality of test suites

Mutation testing is a method used to assess and improve the fault detection capability of a test suite by creating faulty versions, called mutants, of the system under test. Evolutionary Mutation Testing (EMT), like selective mutation or mutant sampling, was proposed to reduce the computational cost, which is a major concern when applying mutation testing. This technique implements an evolutionary algorithm to produce a reduced subset of mutants but with a high proportion of mutants that can help the tester derive new test cases (strong mutants). In this paper, we go a step further in estimating the ability of this technique to induce the generation of test cases. Instead of measuring the percentage of strong mutants within the subset of generated mutants, we compute how much the test suite is actually improved thanks to those mutants. In our experiments, we have compared the extent to which EMT and the random selection of mutants help to find missing test cases in C++ object-oriented systems. We can conclude from our results that the percentage of mutants generated with EMT is lower than with the random strategy to obtain a test suite of the same size and that the technique scales better for complex programs

Sistema de teste auto-adaptativo baseado em modelo para SOA dinâmico

Orientadores: Eliane Martins, Andrea CeccarelliDissertação (mestrado) - Universidade Estadual de Campinas, Instituto de ComputaçãoResumo: Arquitetura orientada a serviços (SOA) é um padrão de design popular para implemen- tação de serviços web devido à interoperabilidade, escalabilidade e reuso de soluções de software que promove. Os serviços que usam essa arquitetura precisam operar em um am- biente altamente dinâmico, entretanto quanto mais a complexidade desses serviços cresce menos os métodos tradicionais de validação se mostram viáveis. Aplicações baseadas em arquitetura orientada a serviços podem evoluir e mudar du- rante a execução. Por conta disso testes offline não asseguram completamente o compor- tamento correto de um sistema em tempo de execução. Por essa razão, a necessidade de tecnicas diferentes para validar o comportamento adequado de uma aplicação SOA durante o seu ciclo de vida são necessárias, por isso testes online executados durante o funcionamento serão usados nesse projeto. O objetivo do projeto é de aplicar técnicas de testes baseados em modelos para gerar e executar casos de testes relevantes em aplicações SOA durante seu tempo de execu- ção. Para alcançar esse objetivo uma estrura de teste online autoadaptativa baseada em modelos foi idealizada. Testes baseados em modelos podem ser gerados de maneira offline ou online. Nos testes offline, os casos de teste são gerados antes do sistema entrar em execução. Já nos testes online, os casos de teste são gerados e aplicados concomitantemente, e as saídas produzidas pela aplicação em teste definem o próximo passo a ser realizado. Quando uma evolução é detectada em um serviço monitorado uma atualização no modelo da aplicação alvo é executada, seguido pela geração e execução de casos de testes online. Mais precisamente, quatro componentes foram integrados em um circuito autoadap- tativo: um serviço de monitoramento, um serviço de criação de modelos, um serviço de geração de casos de teste baseado em modelos e um serviço de teste. As caracteristicas da estrutura de teste foram testadas em três cenários que foram executados em uma aplicação SOA orquestrada por BPEL, chamada jSeduite. Este trabalho é um esforço para entender as restrições e limitações de teste de soft- ware para aplicações SOA, e apresenta análises e soluções para alguns dos problemas encontrados durante a pesquisaAbstract: Service Oriented Architecture (SOA) is a popular design pattern to build web services be- cause of the interoperability, scalability, and reuse of software solutions that it promotes. The services using this architecture need to operate in a highly dynamic environment, but as the complexity of these services grows, traditional validation processes become less feasible. SOA applications can evolve and change during their execution, and offline tests do not completely assure the correct behavior of the system during its execution. There- fore there is a need of techniques to validate the proper behaviour of SOA applications during the SOA lifecycle. Because of that, in this project online testing will be used. The project goal is to employ model-based testing techniques to generate and execute relevant test cases to SOA applications during runtime. In order to achieve this goal a self-adaptive model-based online testing framework was designed. Tests based on models can be generated offline and online. Offline test are generated before the system execution. Online tests are generated and performed concomitantly, and the output produced by the application under test defines the next step to be performed. when our solution detects that a monitored service evolves, the model of the target service is updated, and online test case generation and execution is performed. More specifically, four components were integrated in a self-adaptive loop: a mon- itoring service, a model generator service, a model based testing service and a testing platform. The testing framework had its features tested in three scenarios that were performed in a SOA application orchestrated by BPEL, called jSeduite. This work is an effort to understand the constraints and limitations of the software testing on SOA applications, and present analysis and solutions to some of the problems found during the researchMestradoCiência da ComputaçãoMestre em Ciência da ComputaçãoCAPE

IN2GESOFT: Innovation and Integration of Methods for the Development and Quantitative Management of Software Projects TIN2004-06689-C03

This coordinated project intends to introduce new methods in software engineering project management, integrating different quantitative and qualitative technologies in the management processes. The underlying goal to all three subprojects participants is the generation of information adapted for the efficient performance in the directing of the project. The topics that are investigated are related to the capture of decisions in dynam ical environments and complex systems, software testing and the analysis of the manage ment strategies for the process assessment of the software in its different phases of the production. The project sets up a methodological, conceptual framework and supporting tools that facilitate the decision making in the software project management. This allows us to eval uate the risk and uncertainty associated to different alternatives of management before leading them to action. Thus, it is necessary to define a taxonomy of software models so that they reflect the current reality of the projects. Since the software testing is one of the most critical and costly processes directed to guarantee the quality and reliability of the software, we undertake the research on the automation of the process of software testing by means of the development of new technologies test case generation, mainly based in metaheuristic and model checking techniques in the domains of database and internet applications. The software system developed will allow the integration of these technologies, and the management information needed, from the first phases of the cycle of life in the construction of a software product up to the last ones such as regression tests and maintenance. The set of technologies that we investigate include the use of statistical analysis and of experimental design for obtaining metrics in the phase of analysis, the application of the bayesian nets to the decision processes, the application of the standards of process eval uation and quality models, the utilization of metaheuristics algorithms and technologies of prediction to optimize resources, the technologies of visualization to construct control dashboards, hybrid models for the simulation of processes and others

Evolutionary mutation testing for IoT with recorded and generated events

Mutation testing is a testing technique that has been applied successfully to several programming languages. Despite its benefits for software testing, the high computational cost of mutation testing has kept it from being widely used. Several refinements have been proposed to reduce its cost by reducing the number of generated mutants; one of those is evolutionary mutation testing (EMT). Evolutionary mutation testing aims at generating a reduced set of mutants with an evolutionary algorithm, which searches for potentially equivalent and difficult to kill mutants that help improve the test suite. Evolutionary mutation testing has been evaluated in two contexts so far, ie, web service compositions and object‐oriented C++ programmes. This study explores its performance when applied to event processing language queries of various domains. This study also considers the impact of the test data, since a lack of events or the need to have specific values in them can hinder testing. The effectiveness of evolutionary mutation testing with the original test data generators and the new internet of things test event generator tool is compared in multiple case studies