Empirical Evaluation of Mutation-based Test Prioritization Techniques

We propose a new test case prioritization technique that combines both mutation-based and diversity-based approaches. Our diversity-aware mutation-based technique relies on the notion of mutant distinguishment, which aims to distinguish one mutant's behavior from another, rather than from the original program. We empirically investigate the relative cost and effectiveness of the mutation-based prioritization techniques (i.e., using both the traditional mutant kill and the proposed mutant distinguishment) with 352 real faults and 553,477 developer-written test cases. The empirical evaluation considers both the traditional and the diversity-aware mutation criteria in various settings: single-objective greedy, hybrid, and multi-objective optimization. The results show that there is no single dominant technique across all the studied faults. To this end, \rev{we we show when and the reason why each one of the mutation-based prioritization criteria performs poorly, using a graphical model called Mutant Distinguishment Graph (MDG) that demonstrates the distribution of the fault detecting test cases with respect to mutant kills and distinguishment

Is XML-based test case prioritization for validating WS-BPEL evolution effective in both average and adverse scenarios?

In real life, a tester can only afford to apply one test case prioritization technique to one test suite against a service-oriented workflow application once in the regression testing of the application, even if it results in an adverse scenario such that the actual performance in the test session is far below the average. It is unclear whether the factors of test case prioritization techniques known to be significant in terms of average performance can be extrapolated to adverse scenarios. In this paper, we examine whether such a factor or technique may consistently affect the rate of fault detection in both the average and adverse scenarios. The factors studied include prioritization strategy, artifacts to provide coverage data, ordering direction of a strategy, and the use of executable and non-executable artifacts. The results show that only a minor portion of the 10 studied techniques, most of which are based on the iterative strategy, are consistently effective in both average and adverse scenarios. To the best of our know-ledge, this paper presents the first piece of empirical evidence regarding the consistency in the effectiveness of test case prioritization techniques and factors of service-oriented workflow applications between average and adverse scenarios.published_or_final_versio

Report from GI-Dagstuhl Seminar 16394: Software Performance Engineering in the DevOps World

This report documents the program and the outcomes of GI-Dagstuhl Seminar 16394 "Software Performance Engineering in the DevOps World". The seminar addressed the problem of performance-aware DevOps. Both, DevOps and performance engineering have been growing trends over the past one to two years, in no small part due to the rise in importance of identifying performance anomalies in the operations (Ops) of cloud and big data systems and feeding these back to the development (Dev). However, so far, the research community has treated software engineering, performance engineering, and cloud computing mostly as individual research areas. We aimed to identify cross-community collaboration, and to set the path for long-lasting collaborations towards performance-aware DevOps. The main goal of the seminar was to bring together young researchers (PhD students in a later stage of their PhD, as well as PostDocs or Junior Professors) in the areas of (i) software engineering, (ii) performance engineering, and (iii) cloud computing and big data to present their current research projects, to exchange experience and expertise, to discuss research challenges, and to develop ideas for future collaborations

How do software architects consider non-functional requirements: an exploratory study

© 2012 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Dealing with non-functional requirements (NFRs) has posed a challenge onto software engineers for many years. Over the years, many methods and techniques have been proposed to improve their elicitation, documentation, and validation. Knowing more about the state of the practice on these topics may benefit both practitioners' and researchers' daily work. A few empirical studies have been conducted in the past, but none under the perspective of software architects, in spite of the great influence that NFRs have on daily architects' practices. This paper presents some of the findings of an empirical study based on 13 interviews with software architects. It addresses questions such as: who decides the NFRs, what types of NFRs matter to architects, how are NFRs documented, and how are NFRs validated. The results are contextualized with existing previous work.Peer ReviewedPostprint (author’s final draft

Improved Annealing-Genetic Algorithm for Test Case Prioritization

Regression testing, which can improve the quality of software systems, is a useful but time consuming method. Many techniques have been introduced to reduce the time cost of regression testing. Among these techniques, test case prioritization is an effective technique which can reduce the time cost by processing relatively more important test cases at an earlier stage. Previous works have demonstrated that some greedy algorithms are effective for regression test case prioritization. Those algorithms, however, have lower stability and scalability. For this reason, this paper proposes a new regression test case prioritization approach based on the improved Annealing-Genetic algorithm which incorporates Simulated Annealing algorithm and Genetic algorithm to explore a bigger potential solution space for the global optimum. Three Java programs and five C programs were employed to evaluate the performance of the new approach with five former approaches such as Greedy, Additional Greedy, GA, etc. The experimental results showed that the proposed approach has relatively better performance as well as higher stability and scalability than those former approaches

Test suite prioritization techniques applied to Web-based applications

Web applications have rapidly gained importance in many businesses. The increased usage of web applications has created a challenging need for efficient and effective web application testing strategies. This thesis examines one aspect of web testing, that of test suite prioritization. We examine new test suite prioritization strategies that may improve the rate of fault detection for user-session based test suites. These techniques consider test-lengths and systematic coverage of parameter-values and their interactions. Experimental results show that some of these prioritization strategies often improve the rate of fault detection of test suites when compared to random ordering of the test cases. In general the most effective prioritization strategies consider the systematic coverage of the combinations of parameter-values as early as possible