Modellbasiertes Regressionstesten von Varianten und Variantenversionen

The quality assurance of software product lines (SPL) achieved via testing is a crucial and challenging activity of SPL engineering. In general, the application of single-software testing techniques for SPL testing is not practical as it leads to the individual testing of a potentially vast number of variants. Testing each variant in isolation further results in redundant testing processes by means of redundant test-case executions due to the shared commonality. Existing techniques for SPL testing cope with those challenges, e.g., by identifying samples of variants to be tested. However, each variant is still tested separately without taking the explicit knowledge about the shared commonality and variability into account to reduce the overall testing effort. Furthermore, due to the increasing longevity of software systems, their development has to face software evolution. Hence, quality assurance has also to be ensured after SPL evolution by testing respective versions of variants. In this thesis, we tackle the challenges of testing redundancy as well as evolution by proposing a framework for model-based regression testing of evolving SPLs. The framework facilitates efficient incremental testing of variants and versions of variants by exploiting the commonality and reuse potential of test artifacts and test results. Our contribution is divided into three parts. First, we propose a test-modeling formalism capturing the variability and version information of evolving SPLs in an integrated fashion. The formalism builds the basis for automatic derivation of reusable test cases and for the application of change impact analysis to guide retest test selection. Second, we introduce two techniques for incremental change impact analysis to identify (1) changing execution dependencies to be retested between subsequently tested variants and versions of variants, and (2) the impact of an evolution step to the variant set in terms of modified, new and unchanged versions of variants. Third, we define a coverage-driven retest test selection based on a new retest coverage criterion that incorporates the results of the change impact analysis. The retest test selection facilitates the reduction of redundantly executed test cases during incremental testing of variants and versions of variants. The framework is prototypically implemented and evaluated by means of three evolving SPLs showing that it achieves a reduction of the overall effort for testing evolving SPLs.Testen ist ein wichtiger Bestandteil der Entwicklung von Softwareproduktlinien (SPL). Aufgrund der potentiell sehr großen Anzahl an Varianten einer SPL ist deren individueller Test im Allgemeinen nicht praktikabel und resultiert zudem in redundanten Testfallausführungen, die durch die Gemeinsamkeiten zwischen Varianten entstehen. Existierende SPL-Testansätze adressieren diese Herausforderungen z.B. durch die Reduktion der Anzahl an zu testenden Varianten. Jedoch wird weiterhin jede Variante unabhängig getestet, ohne dabei das Wissen über Gemeinsamkeiten und Variabilität auszunutzen, um den Testaufwand zu reduzieren. Des Weiteren muss sich die SPL-Entwicklung mit der Evolution von Software auseinandersetzen. Dies birgt weitere Herausforderungen für das SPL-Testen, da nicht nur für Varianten sondern auch für ihre Versionen die Qualität sichergestellt werden muss. In dieser Arbeit stellen wir ein Framework für das modellbasierte Regressionstesten von evolvierenden SPL vor, das die Herausforderungen des redundanten Testens und der Software-Evolution adressiert. Das Framework vereint Testmodellierung, Änderungsauswirkungsanalyse und automatische Testfallselektion, um einen inkrementellen Testprozess zu definieren, der Varianten und Variantenversionen unter Ausnutzung des Wissens über gemeinsame Funktionalität und dem Wiederverwendungspotential von Testartefakten und -resultaten effizient testet. Für die Testmodellierung entwickeln wir einen Ansatz, der Variabilitäts- sowie Versionsinformation von evolvierenden SPL gleichermaßen für die Modellierung einbezieht. Für die Änderungsauswirkungsanalyse definieren wir zwei Techniken, um zum einen Änderungen in Ausführungsabhängigkeiten zwischen zu testenden Varianten und ihren Versionen zu identifizieren und zum anderen die Auswirkungen eines Evolutionsschrittes auf die Variantenmenge zu bestimmen und zu klassifizieren. Für die Testfallselektion schlagen wir ein Abdeckungskriterium vor, das die Resultate der Auswirkungsanalyse einbezieht, um automatisierte Entscheidungen über einen Wiederholungstest von wiederverwendbaren Testfällen durchzuführen. Die abdeckungsgetriebene Testfallselektion ermöglicht somit die Reduktion der redundanten Testfallausführungen während des inkrementellen Testens von Varianten und Variantenversionen. Das Framework ist prototypisch implementiert und anhand von drei evolvierenden SPL evaluiert. Die Resultate zeigen, dass eine Aufwandsreduktion für das Testen evolvierender SPL erreicht wird

Model-based Quality Assurance of Cyber-Physical Systems with Variability in Space, over Time and at Runtime

Cyber-physical systems (CPS) are frequently characterized by three essential properties: CPS perform complex computations, CPS conduct control tasks involving continuous data- and signal-processing, and CPS are (parts of) distributed, and even mobile, communication systems. In addition, modern software systems like CPS have to cope with ever-growing extents of variability, namely variability in space by means of predefined configuration options (e.g., software product lines), variability at runtime by means of preplanned reconfigurations (e.g., runtime-adaptive systems), and variability over time by means of initially unforeseen updates to new versions (e.g., software evolution). Finally, depending on the particular application domain, CPS often constitute safety- and mission-critical parts of socio-technical systems. Thus, novel quality-assurance methodologies are required to systematically cope with the interplay between the different CPS characteristics on the one hand, and the different dimensions of variability on the other hand. This thesis gives an overview on recent research and open challenges in model-based specification and quality-assurance of CPS in the presence of variability. The main focus of this thesis is laid on computation and communication aspects of CPS, utilizing evolving dynamic software product lines as engineering methodology and model-based testing as quality-assurance technique. The research is illustrated and evaluated by means of case studies from different application domains

Cost-effective model-based test case generation and prioritization for software product line

In Software Product Line (SPL), testing is used to manage core assets that comprised variability and commonality in effective ways due to large sizes of products that continue to be developed. SPL testing requires a technique that is capable to manage SPL core assets. Model-based Testing (MBT) is a promising technique that offers automation and reusability in test cases generation. However, there are difficulties to ensure testing in MBT can achieve good test cases generation results based on cost (size of test suite, total execution time) and effectiveness (coverage criteria, fault detection rate) measures. This is due to lack of trade-off between cost and effectiveness in test cases generated in MBT for SPL. This study aims to increase quality of test cases based on cost and effectiveness by using generation and prioritization approaches for MBT in SPL. This study focuses on three parts to enhance quality of test cases. First, test model development based on traceability link. In order to improve test cases quality, this study focused on implementation of hybrid-based and hyper-heuristic based techniques to generate test cases. This is followed by Test Cases Prioritization (TCP) technique that is based on dissimilarity-based technique with string distance. These test cases generation and prioritization approaches are evaluated by using two benchmarks - one test object and one real object. The results are compared with other prominent approaches. The mapping approach showed 10.27% and 32.39% f-measure improvement against existing approach on e-shop object, respectively. For test cases generation using hybrid-based approach, the proposed approach outperformed existing approaches with 11.66% coverage, 17.78% average execution time, and 45.98% average size of test suite on vending machine object. The hyper-heuristic based approach NSGA-II-LHH outperformed other proposed low-level heuristic approaches with 12.00% improvement on coverage, 46.66% average execution time and 42.54% average size of test suite. Furthermore, evaluation of TCP approaches showed fault detection improvement of 21.60%, 10.40% and 12.20% and total execution time improvement of 48.00%, 22.70% and 31.80% in comparison with three existing approaches. The results revealed that proposed model transformations, test cases generation and prioritization approaches significantly improve cost and effectiveness measure in MBT for SPL

A dissimilarity with dice-jaro-winkler test case prioritization approach for model- based testing in software product line

The effectiveness of testing in Model-based Testing (MBT) for Software Product Line (SPL) can be achieved by considering fault detection in test case. The lack of fault consideration caused test case in test suite to be listed randomly. Test Case Prioritization (TCP) is one of regression techniques that is adaptively capable to detect faults as early as possible by reordering test cases based on fault detection rate. However, there is a lack of studies that measured faults in MBT for SPL. This paper proposes a Test Case Prioritization (TCP) approach based on dissimilarity and string based distance called Last Minimal for Local Maximal Distance (LM-LMD) with Dice-Jaro-Winkler Dissimilarity. LM-LMD with Dice-Jaro-Winkler Dissimilarity adopts Local Maximum Distance as the prioritization algorithm and Dice-Jaro-Winkler similarity measure to evaluate distance among test cases. This work is based on the test case generated from statechart in Software Product Line (SPL) domain context. Our results are promising as LM-LMD with Dice-Jaro-Winkler Dissimilarity outperformed the original Local Maximum Distance, Global Maximum Distance and Enhanced All-yes Configuration algorithm in terms of Average Fault Detection Rate (APFD) and average prioritization time

SOFTWARE FAULT DETECTION VIA GRAMMAR-BASED TEST CASE GENERATION

Fault detection is helpful to cut down the failure causes by logically locating and eliminating defects. In this thesis, we present a novel fault detection technique via structured input data which can be represented by a grammar. We take a set of well-distributed test cases as input, each of which has a set of test requirements. We illustrate that test requirements come from structured data can be effectively used as coverage criteria to reduce the test suites. We then propose an automatic fault detection approach to locate software bugs which are shown in failed test cases. This method can be applied in testing data-input-critical software such as compilers, translators, reactive systems etc. Preliminary experimental study proves that our fault detection approach is able to precisely locate the faults of software under test from failed test cases

Supporting the grow-and-prune model for evolving software product lines

207 p.Software Product Lines (SPLs) aim at supporting the development of a whole family of software products through a systematic reuse of shared assets. To this end, SPL development is separated into two interrelated processes: (1) domain engineering (DE), where the scope and variability of the system is defined and reusable core-assets are developed; and (2) application engineering (AE), where products are derived by selecting core assets and resolving variability. Evolution in SPLs is considered to be more challenging than in traditional systems, as both core-assets and products need to co-evolve. The so-called grow-and-prune model has proven great flexibility to incrementally evolve an SPL by letting the products grow, and later prune the product functionalities deemed useful by refactoring and merging them back to the reusable SPL core-asset base. This Thesis aims at supporting the grow-and-prune model as for initiating and enacting the pruning. Initiating the pruning requires SPL engineers to conduct customization analysis, i.e. analyzing how products have changed the core-assets. Customization analysis aims at identifying interesting product customizations to be ported to the core-asset base. However, existing tools do not fulfill engineers needs to conduct this practice. To address this issue, this Thesis elaborates on the SPL engineers' needs when conducting customization analysis, and proposes a data-warehouse approach to help SPL engineers on the analysis. Once the interesting customizations have been identified, the pruning needs to be enacted. This means that product code needs to be ported to the core-asset realm, while products are upgraded with newer functionalities and bug-fixes available in newer core-asset releases. Herein, synchronizing both parties through sync paths is required. However, the state of-the-art tools are not tailored to SPL sync paths, and this hinders synchronizing core-assets and products. To address this issue, this Thesis proposes to leverage existing Version Control Systems (i.e. git/Github) to provide sync operations as first-class construct

Automating Security Risk and Requirements Management for Cyber-Physical Systems

Cyber-physische Systeme ermöglichen zahlreiche moderne Anwendungsfälle und Geschäftsmodelle wie vernetzte Fahrzeuge, das intelligente Stromnetz (Smart Grid) oder das industrielle Internet der Dinge. Ihre Schlüsselmerkmale Komplexität, Heterogenität und Langlebigkeit machen den langfristigen Schutz dieser Systeme zu einer anspruchsvollen, aber unverzichtbaren Aufgabe. In der physischen Welt stellen die Gesetze der Physik einen festen Rahmen für Risiken und deren Behandlung dar. Im Cyberspace gibt es dagegen keine vergleichbare Konstante, die der Erosion von Sicherheitsmerkmalen entgegenwirkt. Hierdurch können sich bestehende Sicherheitsrisiken laufend ändern und neue entstehen. Um Schäden durch böswillige Handlungen zu verhindern, ist es notwendig, hohe und unbekannte Risiken frühzeitig zu erkennen und ihnen angemessen zu begegnen. Die Berücksichtigung der zahlreichen dynamischen sicherheitsrelevanten Faktoren erfordert einen neuen Automatisierungsgrad im Management von Sicherheitsrisiken und -anforderungen, der über den aktuellen Stand der Wissenschaft und Technik hinausgeht. Nur so kann langfristig ein angemessenes, umfassendes und konsistentes Sicherheitsniveau erreicht werden. Diese Arbeit adressiert den dringenden Bedarf an einer Automatisierungsmethodik bei der Analyse von Sicherheitsrisiken sowie der Erzeugung und dem Management von Sicherheitsanforderungen für Cyber-physische Systeme. Das dazu vorgestellte Rahmenwerk umfasst drei Komponenten: (1) eine modelbasierte Methodik zur Ermittlung und Bewertung von Sicherheitsrisiken; (2) Methoden zur Vereinheitlichung, Ableitung und Verwaltung von Sicherheitsanforderungen sowie (3) eine Reihe von Werkzeugen und Verfahren zur Erkennung und Reaktion auf sicherheitsrelevante Situationen. Der Schutzbedarf und die angemessene Stringenz werden durch die Sicherheitsrisikobewertung mit Hilfe von Graphen und einer sicherheitsspezifischen Modellierung ermittelt und bewertet. Basierend auf dem Modell und den bewerteten Risiken werden anschließend fundierte Sicherheitsanforderungen zum Schutz des Gesamtsystems und seiner Funktionalität systematisch abgeleitet und in einer einheitlichen, maschinenlesbaren Struktur formuliert. Diese maschinenlesbare Struktur ermöglicht es, Sicherheitsanforderungen automatisiert entlang der Lieferkette zu propagieren. Ebenso ermöglicht sie den effizienten Abgleich der vorhandenen Fähigkeiten mit externen Sicherheitsanforderungen aus Vorschriften, Prozessen und von Geschäftspartnern. Trotz aller getroffenen Maßnahmen verbleibt immer ein gewisses Restrisiko einer Kompromittierung, worauf angemessen reagiert werden muss. Dieses Restrisiko wird durch Werkzeuge und Prozesse adressiert, die sowohl die lokale und als auch die großräumige Erkennung, Klassifizierung und Korrelation von Vorfällen verbessern. Die Integration der Erkenntnisse aus solchen Vorfällen in das Modell führt häufig zu aktualisierten Bewertungen, neuen Anforderungen und verbessert weitere Analysen. Abschließend wird das vorgestellte Rahmenwerk anhand eines aktuellen Anwendungsfalls aus dem Automobilbereich demonstriert.Cyber-Physical Systems enable various modern use cases and business models such as connected vehicles, the Smart (power) Grid, or the Industrial Internet of Things. Their key characteristics, complexity, heterogeneity, and longevity make the long-term protection of these systems a demanding but indispensable task. In the physical world, the laws of physics provide a constant scope for risks and their treatment. In cyberspace, on the other hand, there is no such constant to counteract the erosion of security features. As a result, existing security risks can constantly change and new ones can arise. To prevent damage caused by malicious acts, it is necessary to identify high and unknown risks early and counter them appropriately. Considering the numerous dynamic security-relevant factors requires a new level of automation in the management of security risks and requirements, which goes beyond the current state of the art. Only in this way can an appropriate, comprehensive, and consistent level of security be achieved in the long term. This work addresses the pressing lack of an automation methodology for the security-risk assessment as well as the generation and management of security requirements for Cyber-Physical Systems. The presented framework accordingly comprises three components: (1) a model-based security risk assessment methodology, (2) methods to unify, deduce and manage security requirements, and (3) a set of tools and procedures to detect and respond to security-relevant situations. The need for protection and the appropriate rigor are determined and evaluated by the security risk assessment using graphs and a security-specific modeling. Based on the model and the assessed risks, well-founded security requirements for protecting the overall system and its functionality are systematically derived and formulated in a uniform, machine-readable structure. This machine-readable structure makes it possible to propagate security requirements automatically along the supply chain. Furthermore, they enable the efficient reconciliation of present capabilities with external security requirements from regulations, processes, and business partners. Despite all measures taken, there is always a slight risk of compromise, which requires an appropriate response. This residual risk is addressed by tools and processes that improve the local and large-scale detection, classification, and correlation of incidents. Integrating the findings from such incidents into the model often leads to updated assessments, new requirements, and improves further analyses. Finally, the presented framework is demonstrated by a recent application example from the automotive domain

Trade-Off Exploration for Acceleration of Continuous Integration

Continuous Integration (CI) is a popular software development practice that allows developers to quickly verify modifications to their projects. To cope with the ever-increasing demand for faster software releases, CI acceleration approaches have been proposed to expedite the feedback that CI provides. However, adoption of CI acceleration is not without cost. The trade-off in duration and trustworthiness of a CI acceleration approach determines the practicality of the CI acceleration process. Indeed, if a CI acceleration approach takes longer to prime than to run the accelerated build, the benefits of acceleration are unlikely to outweigh the costs. Moreover, CI acceleration techniques may mislabel change sets (e.g., a build labelled as failing that passes in an unaccelerated setting or vice versa) or produce results that are inconsistent with an unaccelerated build (e.g., the underlying reason for failure does not match with the unaccelerated build). These inconsistencies call into question the trustworthiness of CI acceleration products. We first evaluate the time trade-off of two CI acceleration products — one based on program analysis (PA) and the other on machine learning (ML). After replaying the CI process of 100,000 builds spanning ten open-source projects, we find that the priming costs (i.e., the extra time spent preparing for acceleration) of the program analysis product are substantially less than that of the machine learning product (e.g., average project-wise median cost difference of 148.25 percentage points). Furthermore, the program analysis product generally provides more time savings than the machine learning product (e.g., average project-wise median savings improvement of 5.03 percentage points). Given their deterministic nature, and our observations about priming costs and benefits, we recommend that organizations consider the adoption of program analysis based acceleration. Next, we study the trustworthiness of the same PA and ML CI acceleration products. We re-execute 50 failing builds from ten open-source projects in non-accelerated (baseline), program analysis accelerated, and machine learning accelerated settings. We find that when applied to known failing builds, program analysis accelerated builds more often (43.83 percentage point difference across ten projects) align with the non-accelerated build results. Accordingly, we conclude that while there is still room for improvement for both CI acceleration products, the selected program analysis product currently provides a more trustworthy signal of build outcomes than the machine learning product. Finally, we propose a mutation testing approach to systematically evaluate the trustworthiness of CI acceleration. We apply our approach to the deterministic PA-based CI acceleration product and uncover issues that hinder its trustworthiness. Our analysis consists of three parts: we first study how often the same build in accelerated and unaccelerated CI settings produce different mutation testing outcomes. We call mutants with different outcomes in the two settings “gap mutants”. Next, we study the code locations where gap mutants appear. Finally, we inspect gap mutants to understand why acceleration causes them to survive. Our analysis of ten thriving open-source projects uncovers 2,237 gap mutants. We find that: (1) the gap in mutation outcomes between accelerated and unaccelerated settings varies from 0.11%–23.50%; (2) 88.95% of gap mutants can be mapped to specific source code functions and classes using the dependency representation of the studied CI acceleration product; (3) 69% of gap mutants survive CI acceleration due to deterministic reasons that can be classified into six fault patterns. Our results show that deterministic CI acceleration suffers from trustworthiness limitations, and highlights the ways in which trustworthiness could be improved in a pragmatic manner. This thesis demonstrates that CI acceleration techniques, whether PA or ML-based, present time trade-offs and can reduce software build trustworthiness. Our findings lead us to encourage users of CI acceleration to carefully weigh both the time costs and trustworthiness of their chosen acceleration technique. This study also demonstrates that the following improvements for PA-based CI acceleration approaches would improve their trustworthiness: (1) depending on the size and complexity of the codebase, it may be necessary to manually refine the dependency graph, especially by concentrating on class properties, global variables, and constructor components; and (2) solutions should be added to detect and bypass flaky test during CI acceleration to minimize the impact of flakiness

Black-Box Testfall-Selektion und -Priorisierung für Software-Varianten und -Versionen

Software testing is a fundamental task in software quality assurance. Especially when dealing with several product variants or software versions under test, testing everything for each variant and version is infeasible due to limited testing resources. To cope with increasing complexity both in time (i.e., versions) and space (i.e., variants), new techniques have to be developed to focus on the most important parts for testing. In the past, regression testing techniques such as test case selection and prioritization have emerged to tackle these issues for single-software systems. However, testing of variants and versions is still a challenging task, especially when no source code is available. Most existing regression testing techniques analyze source code to identify important changes to be retested, i.e., they are likely to reveal a failure. To this end, this thesis contributes different techniques for both, variants and versions, to allow more efficient and effective testing in difficult black-box scenarios by identifying important test cases to be re-executed. Four major contributions in software testing are made. (1) We propose a test case prioritization framework for software product lines based on delta-oriented test models to reduce the redundancy in testing between different product variants.(2) We introduce a risk-based testing technique for software product lines. Our semi-automatic test case prioritization approach is able to compute risk values for test model elements and scales with large numbers of product variants. (3) For black-box software versions, we provide a test case selection technique based on genetic algorithms. In particular, seven different black-box selection objectives are defined, thus, we perform a multi-objective test case selection finding Pareto optimal test sets to reduce the testing effort. (4) We propose a novel test case prioritization technique based on supervised machine learning. It is able to imitate decisions made by experts based on different features, such as natural language test case descriptions and black-box meta-data. All of these techniques have been evaluated using the Body Comfort System case study. For testing of software versions, we also assesses our testing techniques using an industrial system. Our evaluation results indicate that our black-box testing approaches for software variants and versions are able to successfully reduce testing effort compared to existing techniques.Testen ist eine fundamentale Aufgabe zur Qualitätssicherung von modernen Softwaresystemen. Mangels limitierter Ressourcen ist das Testen von vielen Produktvarianten oder Versionen sehr herausfordernd und das wiederholte Ausführen aller Testfälle nicht wirtschaftlich. Um mit der Raum- (Varianten) und Zeitdimension (Versionen) in der Entwicklung umzugehen, wurden in der Vergangenheit verschiedene Testansätze entwickelt. Es existieren jedoch nach wie vor große Herausforderungen, welche es zu lösen gilt. Dies ist vor allem der Fall, wenn der Quellcode der getesteten Softwaresysteme unbekannt ist. Das Testen von Black-Box-Systemen erschwert die Identifikation von zu testenden Unterschieden zu vorher getesteten Varianten oder Versionen. In der Literatur finden sich wenige Ansätze, welche versuchen diese Herausforderungen zu lösen. Daher werden in dieser Dissertation neue Ansätze entwickelt und vorgestellt, welche beim Black-Box Testen von Software-Varianten und -Versionen helfen, wichtige Testfälle zur erneuten Ausführung zu identifizieren. Dies erspart die Ausführung von Testfällen, welche weder neues Verhalten testen noch mit hoher Wahrscheinlichkeit neue Fehler zu finden. Insgesamt leistet diese Dissertation die folgenden vier wissenschaftlichen Beiträge: (1) Ein modell-basiertes Framework zur Definition von Testfallpriorisierungsfunktionen für variantenreiche Systeme. Das Framework ermöglicht eine flexible Priorisierung von Testfällen für individuelle Produktvarianten. (2) Einen risiko-basierten Testfallpriorisierungsansatz für variantenreiche Systeme. Das Verfahren ermöglicht eine semi-automatisierte Berechnung von Risikowerten für Elemente von Produktvarianten und skaliert mit großen Produktzahlen. (3) Ein multi-kriterielles Testfallselektionsverfahren für den Regressionstest von Black-Box Software-Versionen. Es werden Black-Box Testkriterien aufgestellt und mittels eines genetischen Algorithmus optimiert um Pareto-optimale Testsets zu berechnen. (4) Ein Testfallpriorisierungsverfahren für Black-Box Regressionstests mit Hilfe von Machine Learning. Der verwendete Algorithmus imitiert Entscheidungen von Testexperten um wichtige Testfälle zu identifizieren. Diese Ansätze wurden alle mit Hilfe von Fallstudien evaluiert. Die resultierenden Ergebnisse zeigen, dass die Ansätze die gewünschten Ziele erreichen und helfen, wichtige Testfälle effektiv zu identifizieren. Insgesamt wird der Testaufwand im Vergleich zu existierenden Techniken verringert

Multi-Objective Search-Based Software Microbenchmark Prioritization

Ensuring that software performance does not degrade after a code change is paramount. A potential solution, particularly for libraries and frameworks, is regularly executing software microbenchmarks, a performance testing technique similar to (functional) unit tests. This often becomes infeasible due to the extensive runtimes of microbenchmark suites, however. To address that challenge, research has investigated regression testing techniques, such as test case prioritization (TCP), which reorder the execution within a microbenchmark suite to detect larger performance changes sooner. Such techniques are either designed for unit tests and perform sub-par on microbenchmarks or require complex performance models, reducing their potential application drastically. In this paper, we propose a search-based technique based on multi-objective evolutionary algorithms (MOEAs) to improve the current state of microbenchmark prioritization. The technique utilizes three objectives, i.e., coverage to maximize, coverage overlap to minimize, and historical performance change detection to maximize. We find that our technique improves over the best coverage-based, greedy baselines in terms of average percentage of fault-detection on performance (APFD-P) and Top-3 effectiveness by 26 percentage points (pp) and 43 pp (for Additional) and 17 pp and 32 pp (for Total) to 0.77 and 0.24, respectively. Employing the Indicator-Based Evolutionary Algorithm (IBEA) as MOEA leads to the best effectiveness among six MOEAs. Finally, the technique's runtime overhead is acceptable at 19% of the overall benchmark suite runtime, if we consider the enormous runtimes often spanning multiple hours. The added overhead compared to the greedy baselines is miniscule at 1%.These results mark a step forward for universally applicable performance regression testing techniques.Comment: 17 pages, 5 figure