Is XML-based test case prioritization for validating WS-BPEL evolution effective in both average and adverse scenarios?

In real life, a tester can only afford to apply one test case prioritization technique to one test suite against a service-oriented workflow application once in the regression testing of the application, even if it results in an adverse scenario such that the actual performance in the test session is far below the average. It is unclear whether the factors of test case prioritization techniques known to be significant in terms of average performance can be extrapolated to adverse scenarios. In this paper, we examine whether such a factor or technique may consistently affect the rate of fault detection in both the average and adverse scenarios. The factors studied include prioritization strategy, artifacts to provide coverage data, ordering direction of a strategy, and the use of executable and non-executable artifacts. The results show that only a minor portion of the 10 studied techniques, most of which are based on the iterative strategy, are consistently effective in both average and adverse scenarios. To the best of our know-ledge, this paper presents the first piece of empirical evidence regarding the consistency in the effectiveness of test case prioritization techniques and factors of service-oriented workflow applications between average and adverse scenarios.published_or_final_versio

Evaluation of alternative design choices for evolutionary mutation testing by means of automated configuration

Mutation testing is a well-established but costly technique to assess and improve the fault detection ability of test suites. This technique consists of introducing subtle changes in the code of a program, which are expected to be detected by the designed test cases. Among the strategies conceived to reduce its cost, evolutionary mutation testing (EMT) has been revealed as a promising approach to select a subset of the whole set of mutants based on a genetic algorithm (GA). However, like any other metaheuristic approach, EMT’s execution depends on a set of parameters (both classical of GAs and context-specific ones), so different configurations can greatly vary its performance. Currently, it is difficult to clarify what are the best values for those parameters by applying manual parameter tuning and whether new design choices could improve its effectiveness with other combinations of values. The experience carried out in this paper applying iterated racing, a well-known automated configuration algorithm, reveals that EMT's performance has been undervalued in previous studies; the new configuration found by iterated racing was able to enhance EMT’s results in all C++ object-oriented programs used in the experiments. This study also confirms alternative design choices as convenient options to improve EMT in this context, namely, detecting and penalizing equivalent mutants by means of Trivial Compiler Equivalence, and learning which mutation operators produced live mutants in the past generations.This work was partially supported by the European Commission (European Regional Development Fund - ERDF), the Spanish Ministry of Science, Innovation and Universities under projects RTI2018-093608-B-C33 and TIN2017-88213-R, the excellence network RED2018-102472-T, the University of Malaga, and Consejería de Economía y Conocimiento de la Junta de Andalucía (grant number UMA18-FEDERJA-003

A subsumption hierarchy of test case prioritization for composite services

published_or_final_versio

Test Reduction for Easing Web Service Integration

Since the irruption of Web Services, in their SOAP and REST flavors, the market has turned from intra-business applications to inter-organizational applications. Nowadays, more organizations have a broad access to the Web and span their frontiers using service-centered applications. In this paper, we review the testing challenges and strategies in Web Services – as the technological weapon-of-choice to implement Business Services. Then we deepen into a possible strategy to address service testing: Test Reduction. Fresh strategies are necessary since Web Services testing is substantially different from legacy systems testing.Sociedad Argentina de Informática e Investigación Operativa (SADIO

Test Reduction for Easing Web Service Integration

Since the irruption of Web Services, in their SOAP and REST flavors, the market has turned from intra-business applications to inter-organizational applications. Nowadays, more organizations have a broad access to the Web and span their frontiers using service-centered applications. In this paper, we review the testing challenges and strategies in Web Services – as the technological weapon-of-choice to implement Business Services. Then we deepen into a possible strategy to address service testing: Test Reduction. Fresh strategies are necessary since Web Services testing is substantially different from legacy systems testing.Sociedad Argentina de Informática e Investigación Operativa (SADIO

Preemptive regression testing of workflow-based web services

published_or_final_versio

Search-Based Mutant Selection for Efficient Test Suite Improvement: Evaluation and Results

Context: Search-based techniques have been applied to almost all areas in software engineering, especially to software testing, seeking to solve hard optimization problems. However, the problem of selecting mutants to improve the test suite at a lower cost has not been explored to the same extent as other problems, such as mutant selection for test suite evaluation or test data generation. Objective: In this paper, we apply search-based mutant selection to enhance the quality of test suites efficiently. Namely, we use the technique known as Evolutionary Mutation Testing (EMT), which allows reducing the number of mutants while preserving the power to refine the test suite. Despite reported benefits of its application, the existing empirical results were derived from a limited number of case studies, a particular set of mutation operators and a vague measure, which currently makes it difficult to determine the real performance of this technique. Method: This paper addresses the shortcomings of previous studies, providing a new methodology to evaluate EMT on the basis of the actual improvement of the test suite achieved by using the evolutionary strategy. We make use of that methodology in new experiments with a carefully selected set of real-world C++ case studies. Results: EMT shows a good performance for most case studies and levels of demand of test suite improvement (around 45% less mutants than random selection in the best case). The results reveal that even a reduced subset of mutants selected with EMT can serve to increase confidence in the test suite, especially in programs with a large set of mutants. Conclusions: These results support the use of search-based techniques to solve the problem of mutant selection for a more efficient test suite refinement. Additionally, we identify some aspects that could foreseeably help enhance EMT

Test Reduction for Easing Web Service Integration

Since the irruption of Web Services, in their SOAP and REST flavors, the market has turned from intra-business applications to inter-organizational applications. Nowadays, more organizations have a broad access to the Web and span their frontiers using service-centered applications. In this paper, we review the testing challenges and strategies in Web Services – as the technological weapon-of-choice to implement Business Services. Then we deepen into a possible strategy to address service testing: Test Reduction. Fresh strategies are necessary since Web Services testing is substantially different from legacy systems testing.Sociedad Argentina de Informática e Investigación Operativa (SADIO

Evolution of security engineering artifacts: a state of the art survey

Security is an important quality aspect of modern open software systems. However, it is challenging to keep such systems secure because of evolution. Security evolution can only be managed adequately if it is considered for all artifacts throughout the software development lifecycle. This article provides state of the art on the evolution of security engineering artifacts. The article covers the state of the art on evolution of security requirements, security architectures, secure code, security tests, security models, and security risks as well as security monitoring. For each of these artifacts the authors give an overview of evolution and security aspects and discuss the state of the art on its security evolution in detail. Based on this comprehensive survey, they summarize key issues and discuss directions of future research

Enhancing coverage adequacy of service compositions after runtime adaptation

Laufzeitüberwachung (engl. runtime monitoring) ist eine wichtige Qualitätssicherungs-Technik für selbstadaptive Service-Komposition. Laufzeitüberwachung überwacht den Betrieb der Service-Komposition. Zur Bestimmung der Genauigkeit von Software-Tests werden häufig Überdeckungskriterien verwendet. Überdeckungskriterien definieren Anforderungen die Software-Tests erfüllen muss. Wegen ihrer wichtigen Rolle im Software-Testen haben Forscher Überdeckungskriterien an die Laufzeitüberwachung von Service-Komposition angepasst. Die passive Art der Laufzeitüberwachung und die adaptive Art der Service-Komposition können die Genauigkeit von Software-Tests zur Laufzeit negativ beeinflussen. Dies kann jedoch die Zuversicht in der Qualität der Service-Komposition begrenzen. Um die Überdeckung selbstadaptiver Service-Komposition zur Laufzeit zu verbessern, untersucht diese Arbeit, wie die Laufzeitüberwachung und Online-Testen kombiniert werden können. Online-Testen bedeutet dass Testen parallel zu der Verwendung einer Service-Komposition erfolgt. Zunächst stellen wir einen Ansatz vor, um gültige Execution-Traces für Service-Komposition zur Laufzeit zu bestimmen. Der Ansatz berücksichtigt die Execution-Traces von Laufzeitüberwachung und (Online)-Testen. Er berücksichtigt Änderungen im Workflow und Software-Services eines Service-Komposition. Zweitens, definieren wir Überdeckungskriterien für Service-Komposition. Die Überdeckungskriterien berücksichtigen Ausführungspläne einer Service-Komposition und berücksichtigen die Überdeckung für Software-Services und die Service-Komposition. Drittens stellen wir Online-Testfälle Priorisierungs Techniken, um die Abdeckungniveau einer Service-Komposition schneller zu erreichen. Die Techniken berücksichtigen die Überdeckung einer Service-Komposition durch beide Laufzeitüberwachung und Online-Tests. Zusätzlich, berücksichtigen sie die Ausführungszeit von Testfällen und das Nutzungsmodell der Service-Komposition. Viertens stellen wir einen Rahmen für die Laufzeitüberwachung und Online-Testen von Software-Services und Service-Komposition, genannt PROSA, vor. PROSA bietet technische Unterstützung für die oben genannten Beiträge. Wir evaluieren die Beiträge anhand einer beispielhaften Service-Komposition, die häufig in dem Forschungsgebiet Service-oriented Computing eingesetzt wird.Runtime monitoring (or monitoring for short) is a key quality assurance technique for self-adaptive service compositions. Monitoring passively observes the runtime behaviour of service compositions. Coverage criteria are extensively used for assessing the adequacy (or thoroughness) of software testing. Coverage criteria specify certain requirements on software testing. The importance of coverage criteria in software testing has motivated researchers to adapt them to the monitoring of service composition. However, the passive nature of monitoring and the adaptive nature of service composition could negatively influence the adequacy of monitoring, thereby limiting the confidence in the quality of the service composition. To enhance coverage adequacy of self-adaptive service compositions at runtime, this thesis investigates how to combine runtime monitoring and online testing. Online testing means testing a service composition in parallel to its actual usage and operation. First, we introduce an approach for determining valid execution traces for service compositions at runtime. The approach considers execution traces of both monitoring and (online) testing. It considers modifications in both workflow and constituent services of a service composition. Second, we define coverage criteria for service compositions. The criteria consider execution plans of a service composition for coverage assessment and consider the coverage of an abstract service and the overall service composition. Third, we introduce online-test-case prioritization techniques to achieve a faster coverage of a service composition. The techniques employ coverage of a service composition from both monitoring and online testing, execution time of test cases, and the usage model of the service composition. Fourth, we introduce a framework for monitoring and online testing of services and service compositions called PROSA. PROSA provides technical support for the aforementioned contributions. We evaluate the contributions of this thesis using service compositions frequently used in service-oriented computing research