Differential Performance Debugging with Discriminant Regression Trees

Differential performance debugging is a technique to find performance problems. It applies in situations where the performance of a program is (unexpectedly) different for different classes of inputs. The task is to explain the differences in asymptotic performance among various input classes in terms of program internals. We propose a data-driven technique based on discriminant regression tree (DRT) learning problem where the goal is to discriminate among different classes of inputs. We propose a new algorithm for DRT learning that first clusters the data into functional clusters, capturing different asymptotic performance classes, and then invokes off-the-shelf decision tree learning algorithms to explain these clusters. We focus on linear functional clusters and adapt classical clustering algorithms (K-means and spectral) to produce them. For the K-means algorithm, we generalize the notion of the cluster centroid from a point to a linear function. We adapt spectral clustering by defining a novel kernel function to capture the notion of linear similarity between two data points. We evaluate our approach on benchmarks consisting of Java programs where we are interested in debugging performance. We show that our algorithm significantly outperforms other well-known regression tree learning algorithms in terms of running time and accuracy of classification.Comment: To Appear in AAAI 201

Non-Intrusive Program Tracing of Non-Preemptive Multitasking Systems Using Power Consumption

System tracing, runtime monitoring, execution reconstruction are useful techniques for protecting the safety and integrity of systems. Furthermore, with time-aware or overhead-aware techniques being available, these techniques can also be used to monitor and secure production systems. As operating systems gain in popularity, even in deeply embedded systems, these techniques face the challenge to support multitasking. In this thesis, we propose a novel non-intrusive technique, which efficiently reconstructs the execution trace of non-preemptive multitasking system by observing power consumption characteristics. Our technique uses the Control Flow Graph (CFG) of the application program to identify the most likely block of code that the system is executing at any given point in time. For the purpose of the experimental evaluation, we first instrument the source code to obtain power consumption information of each Basic Block (BB), which is used as the training data for our Dynamic Time Warping (DTW) and k-Nearest Neighbors (k-NN) classifier. Once the system is trained, this technique is used to identify live code-block execution (LCBE). We show that the technique can reconstruct the execution flow of programs in a multi-tasking environment with high accuracy. To aid the classification process, we analyze eight widely used machine learning algorithms with time-series power-traces data and show the comparison of time and computational resources for all the algorithms

Direct and indirect classifiers

Automated classification rules are often required tools in medical research. In screening programs, an automated classification rule is desired which can accurately identify the subjects as being ``healthy'' or ``affected'', allowing expensive and experienced specialised staff to be replaced by cheaper assistants. In the current thesis we examine how to make use of medical examinations and a priori knowledge, which are given for learning samples but not for later test samples, in order to improve an automated classification rule. The starting point of all considerations was glaucoma, an affection of the retinal nerve fibre layer. Glaucoma is an irreversible disease and focus of recent research. Several medical approaches lead to learning samples in the Erlanger Eye Registry. The registry includes a magnitude of medical examinations, although the diagnosis is usually based on two conventional examination tools.At the beginning, we describe the approach of indirect classification. The conventional indirect classifier distinguishes between explanatory variables, i.e. variables which are available for recent and future observations, and intermediate variables, i.e. variables the diagnosis is based on. This approach of indirect classification can be applied in situations where detailed a priori knowledge, including a diagnostic rule based on the intermediate variables, is given. In the following, we extend indirect classification to situations where such a diagnostic tool is not known. We define an indirect classifier more generally, as a classification rule, which makes use of all variables given in the learning sample. We make the algorithmic proposal ``indirect subagging''. Indirect subagging is a generalised indirect classification approach which combines an arbitrary number of prediction models for intermediate variables, which are not collected for future observations. In contrast to the conventional indirect approach, we can apply indirect subagging in situations where only little a priori knowledge is given. In contrast to the framework of indirect classification we define a direct classifier as a classifier which only uses the set of explanatory variables. A given diagnostic function, incorporated into the conventional indirect classification approach, enables the distinction between an observed class membership following the diagnostic function and a true class membership. We distinguish between these two possible states of the disease in following investigations.Furthermore, we examine asymptotic properties of indirect classification and show that the conventional indirect approach is Bayes consistent with respect to the observed class membership and under certain model assumptions, while indirect subagging is Bayes consistent under more general assumptions. An artificial simulation model leads to the conclusion, that a correct specification of the fixed diagnostic function is crucial for the performance of the conventional indirect classifier. All in all, the indirect classifiers outperform direct ones within this simulation framework.Moreover, we develop a complex simulation setup, which generates the data structure as given by the task of glaucoma classification. We investigate the performance of direct and indirect classifiers for different variances of explanatory and intermediate variables here.Application to a case-control study of glaucoma and healthy subjects show the gain of indirect classification. The application to two additional datasets indicate that indirect subagging performs comparably or even better than the corresponding direct classifiers. The performance of the classifier seems to depend on the diagnostic value of the intermediate variables.Finally, we demonstrate the application of indirect classification, using the add-on package ipred in the programming environment R.In der medizinischen Forschung finden automatische Klassifikationsregeln ihre Anwendung häufig als Hilfsmittel zur untersucherunabhängigen Entscheidungsfindung. So kann in Screeningprogrammen mit einer solchen Klassifikationsvorschrift, welche die Probanden automatisch in ,,krank'' und ,,gesund'' einteilt, teures und erfahrenes Fachpersonal durch weniger teure Hilfskräfte entlastet werden. In der vorliegenden Arbeit wird diskutiert, wie sowohl das a priori Wissen eines Arztes über eine Erkrankung, als auch die medizinischen Messungen, welche nur für den Lerndatensatz, nicht jedoch für spätere Testdatensätze erhoben werden, zur Verbesserung einer solchen automatischen Klassifikationsvorschrift genutzt werden können.Ausgangspunkt aller Überlegungen ist hierbei eine irreversible Erkrankung der retinalen Nervenfaserschicht, genannt der grüne Star (Glaukom). Diese Erkrankung ist derzeit Gegenstand aktueller medizinischer Forschung. Daher sind durch verschiedenste Forschungsansätze im Erlanger Glaukomregister Lerndatensätze entstanden, welche Ergebnisse vieler medizinischer Untersuchungen beinhalten, obwohl die eigentliche medizinische Diagnose dieser Erkrankung meist auf nur zwei konventionellen Untersuchungsverfahren basiert.In der vorliegenden Arbeit wird zunächst der Ansatz der indirekten Klassifikation beschrieben. Der konventionelle indirekte Klassifikator unterscheidet zwischen so genannten ,,erklärenden'' Variablen, welche sowohl für derzeitige, als auch für zukünftige Beobachtungen erhoben werden, und ,,intermediären'' Variablen, auf denen die Diagnose basiert. Dieser konventionelle Ansatz der indirekten Klassifikation kann in Situationen, in denen ein detailliertes a priori Wissen einschließlich einer medizinischen Diagnosevorschrift, basierend auf intermediären Variablen, bekannt ist, angewendet werden. In einem weiteren Schritt wird die indirekte Klassifikation auf Situationen, in denen weniger Wissen über einen gegebenen Lerndatensatz vorhanden ist, ausgeweitet. Allgemeiner wird ein indirekter Klassifikator als ein Klassifikator definiert, der alle im Lerndatensatz vorhandenen Variablen in einer gewissen Form für die Klassifikation nutzt. Der algorithmische Vorschlag ,,indirect subagging'' kombiniert eine beliebige Anzahl von Vorhersagemodellen für intermediäre Variablen, welche für zukünftige Probanden nicht erfaßt werden. Im Gegensatz zur konventionellen indirekten Klassifikation, kann ,,indirect subagging'' auch in Situationen angewendet werden in denen nur wenig a priori Wissen vorhanden ist. Ein direkter Klassifikator nutzt ausschließlich die erklärenden Variablen und die Klassenvariable zur Erstellung einer automatischen Entscheidungsregel.Eine bekannte medizinische Diagnosevorschrift, wie sie im konventionellen Ansatz der indirekten Klassifikation einbezogen wird, ermöglicht die Unterscheidung zwischen einer diagnostizierten Klassenzugehörigkeit bzgl. der Diagnosevorschrift und einer wahren Klassenzugehörigkeit. Es wird im folgenden zwischen diesen beiden möglichen Erkrankungszuständen unterschieden. Asymptotische Eigenschaften der indirekten Klassifikation werden untersucht, und es wird gezeigt, daß der konventionelle indirekte Ansatz unter bestimmten Modellannahmen Bayes konsistent bzgl. des diagnostizierten Erkrankungszustandes ist, während ,,indirect subagging'' auch unter allgemeineren Annahmen Bayes konsistent ist.Ein abstraktes Simulationsmodell führt zu der Erkenntnis, daß die korrekte Formulierung der Diagnosevorschrift im konventionellen indirekten Ansatz ausschlaggebend für dessen Misklassifikationsrate ist. Insgesamt erreichten die indirekten Klassifikatoren niedrigere Fehlerraten, als die direkten. Im weiteren wird ein komplexes Simulationsmodell beschrieben, welches eine ähnliche Datenstruktur, wie sie bei der Glaukomklassifikation gegeben ist, generiert. Es wird untersucht, wie indirekte und direkte Klassifikatoren auf verschiedene Varianzen der erklärenden bzw. der intermediären Variablen reagieren.Die Anwendung der diskutierten Verfahren auf eine Fall-Kontroll-Studie von Glaukompatienten und gesunden Probanden macht den Nutzen der indirekten Klassifikation bei realen Fragestellungen deutlich. Anwendungen auf zwei weitere Datensätze weisen darauf hin, daß der Ansatz ,,indirect subagging'' vergleichbar gute oder bessere Misklassifikationsraten wie die entsprechenden direkten Klassifikatoren zu erreichen scheint. Die Güte der indirekten Klassifikation scheint auch von dem Informationsgehalt der intermediären Variablen abzuhängen. Abschließend wird die Durchführung der indirekten Klassifikation mit Hilfe des Zusatzpaketes ipred in der Programmierumgebung R demonstriert

FlaKat: A Machine Learning-Based Categorization Framework for Flaky Tests

Flaky tests can pass or fail non-deterministically, without alterations to a software system. Such tests are frequently encountered by developers and hinder the credibility of test suites. Thus, flaky tests have caught the attention of researchers in recent years. Numerous approaches have been published on defining, locating, and categorizing flaky tests, along with auto-repairing strategies for specific types of flakiness. Practitioners have developed several techniques to detect flaky tests automatically. The most traditional approaches adopt repeated execution of test suites accompanied by techniques such as shuffled execution order, and random distortion of environment. State-of-the-art research also incorporates machine learning solutions into flaky test detection and achieves reasonably good accuracy. Moreover, strategies for repairing flaky tests have also been published for specific flaky test categories and the process has been automated as well. However, there is a research gap between flaky test detection and category-specific flakiness repair. To address the aforementioned gap, this thesis proposes a novel categorization framework, called FlaKat, which uses machine-learning classifiers for fast and accurate categorization of a given flaky test case. FlaKat first parses and converts raw flaky tests into vector embeddings. The dimensionality of embeddings is reduced and then used for training machine learning classifiers. Sampling techniques are applied to address the imbalance between flaky test categories in the dataset. The evaluation of FlaKat was conducted to determine its performance with different combinations of configurations using known flaky tests from 108 open-source Java projects. Notably, Implementation-Dependent and Order-Dependent flaky tests, which represent almost 75% of the total dataset, achieved F1 scores (harmonic mean of precision and recall) of 0.94 and 0.90 respectively while the overall macro average (no weight difference between categories) is at 0.67. This research work also proposes a new evaluation metric, called Flakiness Detection Capacity (FDC), for measuring the accuracy of classifiers from the perspective of information theory and provides proof for its effectiveness. The final obtained results for FDC also aligns with F1 score regarding which classifier yields the best flakiness classification

Hierarchical Cluster-based Partial Least Squares Regression (HC-PLSR) is an efficient tool for metamodelling of nonlinear dynamic models

<p>Abstract</p> <p>Background</p> <p>Deterministic dynamic models of complex biological systems contain a large number of parameters and state variables, related through nonlinear differential equations with various types of feedback. A metamodel of such a dynamic model is a statistical approximation model that maps variation in parameters and initial conditions (inputs) to variation in features of the trajectories of the state variables (outputs) throughout the entire biologically relevant input space. A sufficiently accurate mapping can be exploited both instrumentally and epistemically. Multivariate regression methodology is a commonly used approach for emulating dynamic models. However, when the input-output relations are highly nonlinear or non-monotone, a standard linear regression approach is prone to give suboptimal results. We therefore hypothesised that a more accurate mapping can be obtained by locally linear or locally polynomial regression. We present here a new method for local regression modelling, Hierarchical Cluster-based PLS regression (HC-PLSR), where fuzzy <it>C</it>-means clustering is used to separate the data set into parts according to the structure of the response surface. We compare the metamodelling performance of HC-PLSR with polynomial partial least squares regression (PLSR) and ordinary least squares (OLS) regression on various systems: six different gene regulatory network models with various types of feedback, a deterministic mathematical model of the mammalian circadian clock and a model of the mouse ventricular myocyte function.</p> <p>Results</p> <p>Our results indicate that multivariate regression is well suited for emulating dynamic models in systems biology. The hierarchical approach turned out to be superior to both polynomial PLSR and OLS regression in all three test cases. The advantage, in terms of explained variance and prediction accuracy, was largest in systems with highly nonlinear functional relationships and in systems with positive feedback loops.</p> <p>Conclusions</p> <p>HC-PLSR is a promising approach for metamodelling in systems biology, especially for highly nonlinear or non-monotone parameter to phenotype maps. The algorithm can be flexibly adjusted to suit the complexity of the dynamic model behaviour, inviting automation in the metamodelling of complex systems.</p

CBR and MBR techniques: review for an application in the emergencies domain

The purpose of this document is to provide an in-depth analysis of current reasoning engine practice and the integration strategies of Case Based Reasoning and Model Based Reasoning that will be used in the design and development of the RIMSAT system. RIMSAT (Remote Intelligent Management Support and Training) is a European Commission funded project designed to: a.. Provide an innovative, 'intelligent', knowledge based solution aimed at improving the quality of critical decisions b.. Enhance the competencies and responsiveness of individuals and organisations involved in highly complex, safety critical incidents - irrespective of their location. In other words, RIMSAT aims to design and implement a decision support system that using Case Base Reasoning as well as Model Base Reasoning technology is applied in the management of emergency situations. This document is part of a deliverable for RIMSAT project, and although it has been done in close contact with the requirements of the project, it provides an overview wide enough for providing a state of the art in integration strategies between CBR and MBR technologies.Postprint (published version