Supporting Stress Testing in Real-Time Systems with Constraint Programming

Failures in safety-critical Real-Time Embedded Systems (RTES) could result in catastrophic consequences for the system itself, its users, and the environment. Therefore, these systems are subject to strict performance requirements specifying constraints on real-time properties such as task deadlines, response time and CPU usage. Lately, RTES have been shifting towards multi-threaded application design, highly configurable operating systems, and multi-core architectures for computing platforms. The concurrent nature of their operating environment also entails that the order of external events triggering RTES tasks is often unpredictable. Such complexity in the system architecture, concurrency, and environment renders performance testing increasingly challenging. Specifically, computing input combinations that are intended to violate performance requirements, i.e., stress testing, is one of the preferred ways for verifying RTES performance. These input combinations are referred to as stress test cases, and, upon execution, are predicted to result in worst-case scenarios with respect to a performance requirement. In RTES, stress test cases are usually characterized by sequences of arrival times for aperiodic tasks in the subject system. Generating stress test cases is challenging because it is hard to predict how a particular sequence of arrival times will affect the system performance, and because the set of all arrival times for aperiodic tasks quickly grows as the system size increases. For this reason, search strategies based on Genetic Algorithms (GA) have been used to find stress test cases with high chances of violating performance requirements. For practical use, software testing has to accommodate time and budget constraints. In the context of stress testing, it is essential to investigate the trade-off between the time needed to generate test cases (efficiency), their capability to reveal scenarios that violate performance requirements (effectiveness), and to cover different scenarios where these violations arise (diversity). Even though GA are efficient, and capable of finding diverse solutions, they explore only part of the search space, and their effectiveness depends on configuration parameters. This aspect justifies considering alternative strategies, such as Constraint Programming (CP), that explore the search space completely. Furthermore, to enable effective industrial application, stress testing has to be capable of seamless integration in the development cycle of companies. Therefore, it is both important to capture specific system and contextual properties in a conceptual model, and to map such conceptual model in a standard Model Driven Engineering (MDE) language such as UML/MARTE. In this thesis, we address the challenges above by presenting a practical approach, based on CP, to support performance stress testing in RTES. Specifically, we make the following contributions: (1) a conceptual model, mapped to UML/MARTE, which captures the abstractions required to generate stress test cases, (2) a constraint optimization model to generate such test cases, and (3) a combined GA+CP stress testing strategy that achieves a practical trade-off between efficiency, effectiveness and diversity. The validation of our work shows that (1) the conceptual model can be applied with a reasonable overhead in an industrial settings, (2) CP is able to effectively identify worst-case scenarios with respect to task deadlines, response time, and CPU usage, and (3) the combined GA+CP strategy is more likely than GA and CP in isolation to scale to large and complex systems. The work in this thesis opens up the exploration of further directions, involving the use of multi-objective optimization to generate stress test cases that simultaneously exercise different performance properties of the system, and of MiniMax analysis to derive design and configuration guidelines that minimize the risk to violate performance requirements at runtime

Combining Genetic Algorithms and Constraint Programming to Support Stress Testing of Task Deadlines

Tasks in Real Time Embedded Systems (RTES) are often subject to hard deadlines, that constrain how quickly the system must react to external inputs. These inputs and their timing vary in a large domain depending on the environment state, and can never be fully predicted prior to system execution. Therefore, approaches for stress testing must be developed to uncover possible deadline misses of tasks for different input arrival times. In this paper, we describe stress test case generation as a search problem over the space of task arrival times. Specifically, we search for worst case scenarios maximizing deadline misses where each scenario characterizes a test case. In order to scale our search to large industrial-size problems, we combine two state-of-the-art search strategies, namely Genetic Algorithms (GA) and Constraint Programming (CP). Our experimental results show that, in comparison with GA and CP in isolation, GA+CP achieves nearly the same effectiveness as CP and the same efficiency and solution diversity as GA, thus combining the advantages of the two strategies. In light of these results, we conclude that a combined GA+CP approach to stress testing is more likely to scale to large and complex systems

Die funktionelle Relevanz von natürlich vorkommenden Varianten des Adhäsions-G-Protein-gekoppelten Rezeptors GPR133 (ADGRD1)

Eine Vielzahl von Krankheiten und Phänotypen im Menschen werden durch Mutationen in G-Protein-gekoppelten Rezeptoren (GPCR) verursacht. Die Klasse der Adhäsions-GPCR (aGPCR) ist bisher wenig erforscht, wird aber mit diversen Funktionen in der Immunität, Nervenentwicklung, Embryonalentwicklung und im Tumorwachstum in Zusammenhang gebracht. Ein Vertreter der Gruppe V der aGPCR ist der GPR133 (ADGRD1). Einige genomweite Assoziationsstudien konnten den gpr133-Lokus mit Veränderungen im Metabolismus, in der Körpergröße und der Herzfrequenz verknüpfen. Für diesen aGPCR wurde bereits die Signaltransduktion über Gαs und Gαi gezeigt, sodass eine funktionelle Charakterisierung des GPR133 und dessen Varianten über cAMP-, cAMP response element- (CRE) und CRE binding protein- (CREB) Assays in vitro möglich ist. Systematische Untersuchungen der Struktur des GPR133 konnten die gebundene agonistische „Stachelsequenz“ aufzeigen. Dies legt den Grundstein für die funktionelle Untersuchung von Mutationen im GPR133. Eine Analyse von mehr als 1000 sequenzierten humanen Genomen ergab über 9000 Einzelnukleotidpolymorphismen (SNP) im gpr133-Gen. Ungefähr 2,4 % der SNP liegen in kodierenden Genabschnitten und resultieren in 129 nicht-synonymen SNP (nsSNP) an 119 Aminosäurepositionen. Die funktionelle Relevanz dieser Missense-Varianten war unbekannt. Tiefergehende Analysen konnten nsSNP identifizieren, die zu einem vollständigen bzw. partiellen Funktionsverlust (A448D, Q600stop, C632fs [frame shift], A761E, N795K) oder zu einer erhöhten Basalaktivität (F383S, D453N) führen. Ein Vergleich der aGPCR Subklassen basierend auf diversen Orthologsequenzen konnte zudem stark konservierte Bereiche aufzeigen, deren Änderungen durch nsSNP im GPR133 in Funktionsänderungen münden. Das große im Menschen vorhandene funktionelle Spektrum von GPR133-Varianten könnte für klinisch relevante Phänotypen verantwortlich sein, auch wenn die bisher erfassten heterozygoten Individuen lebensfähig sind

A Goal-Based Approach for Qualification of New Technologies: Foundations, Tool Support, and Industrial Validation

New technologies typically involve innovative aspects that are not addressed by the existing normative standards and hence are not assessable through common certification procedures. To ensure that new technologies can be implemented in a safe and reliable manner, a specific kind of assessment is performed, which in many industries, e.g., the energy sector, is known as Technology Qualification (TQ). TQ aims at demonstrating with an acceptable level of confidence that a new technology will function within specified limits. Expert opinion plays an important role in TQ, both to identify the safety and reliability evidence that needs to be developed and to interpret the evidence provided. Since there are often multiple experts involved in TQ, it is crucial to apply a structured process for eliciting expert opinions, and to use this information systematically when analyzing the satisfaction of the technology's safety and reliability objectives. In this paper, we present a go al-based approach for TQ. Our approach enables analysts to quantitatively reason about the satisfaction of the technology's overall goals and further to identify the aspects that must be improved to increase goal satisfaction. The approach is founded on three main components: goal models, expert elicitation, and probabilistic simulation. We describe a tool, named Modus, that we have developed in support of our approach. We provide an extensive empirical validation of our approach through two industrial case studies and a survey