Using Metamorphic Testing at Runtime to Detect Defects in Applications without Test Oracles

First, we will present an approach called Automated Metamorphic System Testing. This will involve automating system-level metamorphic testing by treating the application as a black box and checking that the metamorphic properties of the entire application hold after execution. This will allow for metamorphic testing to be conducted in the production environment without affecting the user, and will not require the tester to have access to the source code. The tests do not require an oracle upon their creation; rather, the metamorphic properties act as built-in test oracles. We will also introduce an implementation framework called Amsterdam. Second, we will present a new type of testing called Metamorphic Runtime Checking. This involves the execution of metamorphic tests from within the application, i.e., the application launches its own tests, within its current context. The tests execute within the application's current state, and in particular check a function's metamorphic properties. We will also present a system called Columbus that supports the execution of the Metamorphic Runtime Checking from within the context of the running application. Like Amsterdam, it will conduct the tests with acceptable performance overhead, and will ensure that the execution of the tests does not affect the state of the original application process from the users' perspective; however, the implementation of Columbus will be more challenging in that it will require more sophisticated mechanisms for conducting the tests without pre-empting the rest of the application, and for comparing the results which may conceivably be in different processes or environments. Third, we will describe a set of metamorphic testing guidelines that can be followed to assist in the formulation and specification of metamorphic properties that can be used with the above approaches. These will categorize the different types of properties exhibited by many applications in the domain of machine learning and data mining in particular (as a result of the types of applications we will investigate), but we will demonstrate that they are also generalizable to other domains as well. This set of guidelines will also correlate to the different types of defects that we expect the approaches will be able to find

Levels of feline infectious peritonitis virus in blood, effusions, and various tissues and the role of lymphopenia in disease outcome following experimental infection.

Twenty specific pathogen free cats were experimentally infected with a virulent cat-passaged type I field strain of FIPV. Eighteen cats succumbed within 2-4 weeks to effusive abdominal FIP, one survived for 6 weeks, and one seroconverted without outward signs of disease. A profound drop in the absolute count of blood lymphocytes occurred around 2 weeks post-infection (p.i.) in cats with rapid disease, while the decrease was delayed in the one cat that survived for 6 weeks. The absolute lymphocyte count of the surviving cat remained within normal range. Serum antibodies as measured by indirect immunofluorescence appeared after 2 weeks p.i. and correlated with the onset of disease signs. Viral genomic RNA was either not detectable by reverse transcription quantitative real-time PCR (RT-qPCR) or detectable only at very low levels in terminal tissues not involved directly in the infection, including hepatic and renal parenchyma, cardiac muscle, lung or popliteal lymph node. High tissue virus loads were measured in severely affected tissues such as the omentum, mesenteric lymph nodes and spleen. High levels of viral genomic RNA were also detected in whole ascitic fluid, with the cellular fraction containing 10-1000 times more viral RNA than the supernatant. Replicating virus was strongly associated with macrophages by immunohistochemistry. Virus was usually detected at relatively low levels in feces and there was no evidence of enterocyte infection. Viral genomic RNA was not detected at the level of test sensitivity in whole blood, plasma, or the white cell fraction in terminal samples from the 19 cats that succumbed or in the single survivor. These studies reconfirmed the effect of lymphopenia on disease outcome. FIPV genomic RNA was also found to be highly macrophage associated within diseased tissues and effusions as determined by RT-qPCR and immunohistochemistry but was not present in blood

Is a Rigorous Agile Methodology the Best Development Strategy for Small Scale Tech Startups?

Recently, Agile development processes have become popular in the software development community, and have been shown to be effective in large organizations. However, given that the communication and cooperation dynamics in startup companies are very different from that of larger, more established companies, and the fact that the initial focus of a startup might be significantly different from its ultimate goal, it is questionable whether a rigid process model that works for larger companies is appropriate in tackling the problems faced by a startup. When we scale down even further and observe the small scale startup with only a few members, many of the same problems that Agile methodology sets out to solve do not even exist. Then, for a small scale startup, is it still worth putting the resources into establishing a process model? Do the benefits of adopting an Agile methodology outweigh the opportunity cost of spending the resources elsewhere? This paper examines the advantages and disadvantages of adopting an Agile methodology in a small scale tech startup and compares it to other process models, such as the Waterfall model and Lean Startup. In determining whether a rigorous agile methodology is the best development strategy for small scale tech startups, we consider the metrics of cost, time, quality, and scope in light of the particular needs of small startup organizations, and present a case study of a company that has needed to answer this very question

Metamorphic Runtime Checking of Non-Testable Programs

Challenges arise in assuring the quality of applications that do not have test oracles, i.e., for which it is impossible to know what the correct output should be for arbitrary input. Metamorphic testing has been shown to be a simple yet effective technique in addressing the quality assurance of these "non-testable programs". In metamorphic testing, if test input x produces output f(x), specified "metamorphic properties" are used to create a transformation function t, which can be applied to the input to produce t(x); this transformation then allows the output f(t(x)) to be predicted based on the already-known value of f(x). If the output is not as expected, then a defect must exist. Previously we investigated the effectiveness of testing based on metamorphic properties of the entire application. Here, we improve upon that work by presenting a new technique called Metamorphic Runtime Checking, a testing approach that automatically conducts metamorphic testing of individual functions during the program's execution. We also describe an implementation framework called Columbus, and discuss the results of empirical studies that demonstrate that checking the metamorphic properties of individual functions increases the effectiveness of the approach in detecting defects, with minimal performance impact

Perspectives on Allyship in Academia

Allyship in academia is critical for creating inclusive communities that are welcoming to all students, but the perception of its benefits and challenges can vary depending on a number of factors. This session will explore perspectives of allyship in academia by bringing together a diverse group of faculty and students who can share a wide range of experiences and insights, and aims to facilitate a discussion among all attendees that leads to an exchange of ideas, the strengthening of our community, and progress toward our common goal of inclusion in computing

Improving the Dependability of Machine Learning Applications

As machine learning (ML) applications become prevalent in various aspects of everyday life, their dependability takes on increasing importance. It is challenging to test such applications, however, because they are intended to learn properties of data sets where the correct answers are not already known. Our work is not concerned with testing how well an ML algorithm learns, but rather seeks to ensure that an application using the algorithm implements the specification correctly and fulfills the users' expectations. These are critical to ensuring the application's dependability. This paper presents three approaches to testing these types of applications. In the first, we create a set of limited test cases for which it is, in fact, possible to predict what the correct output should be. In the second approach, we use random testing to generate large data sets according to parameterization based on the application's equivalence classes. Our third approach is based on metamorphic testing, in which properties of the application are exploited to define transformation functions on the input, such that the new output can easily be predicted based on the original output. Here we discuss these approaches, and our findings from testing the dependability of three real-world ML applications

Using Runtime Testing to Detect Defects in Applications without Test Oracles

It is typically infeasible to test a large, complex software system in all its possible configurations and system states prior to deployment. Moreover, some such applications have no test oracles to indicate their correctness. In my thesis, we will address these problems in two ways. First, we suggest that executing tests within the context of an application running in the field can reveal defects that would not ordinarily otherwise be found. Second, we believe that this approach can further be extended to applications for which there is no test oracle by using a variant of metamorphic testing at runtime

Automatic Detection of Defects in Applications without Test Oracles

In application domains that do not have a test oracle, such as machine learning and scientific computing, quality assurance is a challenge because it is difficult or impossible to know in advance what the correct output should be for general input. Previously, metamorphic testing has been shown to be a simple yet effective technique in detecting defects, even without an oracle. In metamorphic testing, the application's ``metamorphic properties'' are used to modify existing test case input to produce new test cases in such a manner that, when given the new input, the new output can easily be computed based on the original output. If the new output is not as expected, then a defect must exist. In practice, however, metamorphic testing can be a manually intensive technique for all but the simplest cases. The transformation of input data can be laborious for large data sets, and errors can occur in comparing the outputs when they are very complex. In this paper, we present a tool called Amsterdam that automates metamorphic testing by allowing the tester to easily set up and conduct metamorphic tests with little manual intervention, merely by specifying the properties to check, configuring the framework, and running the software. Additionally, we describe an approach called Heuristic Metamorphic Testing, which addresses issues related to false positives and non-determinism, and we present the results of new empirical studies that demonstrate the effectiveness of metamorphic testing techniques at detecting defects in real-world programs without test oracles

Itaconate modulates tricarboxylic acid and redox metabolism to mitigate reperfusion injury.

ObjectivesCerebral ischemia/reperfusion (IR) drives oxidative stress and injurious metabolic processes that lead to redox imbalance, inflammation, and tissue damage. However, the key mediators of reperfusion injury remain unclear, and therefore, there is considerable interest in therapeutically targeting metabolism and the cellular response to oxidative stress.MethodsThe objective of this study was to investigate the molecular, metabolic, and physiological impact of itaconate treatment to mitigate reperfusion injuries in in vitro and in vivo model systems. We conducted metabolic flux and bioenergetic studies in response to exogenous itaconate treatment in cultures of primary rat cortical neurons and astrocytes. In addition, we administered itaconate to mouse models of cerebral reperfusion injury with ischemia or traumatic brain injury followed by hemorrhagic shock resuscitation. We quantitatively characterized the metabolite levels, neurological behavior, markers of redox stress, leukocyte adhesion, arterial blood flow, and arteriolar diameter in the brains of the treated/untreated mice.ResultsWe demonstrate that the "immunometabolite" itaconate slowed tricarboxylic acid (TCA) cycle metabolism and buffered redox imbalance via succinate dehydrogenase (SDH) inhibition and induction of anti-oxidative stress response in primary cultures of astrocytes and neurons. The addition of itaconate to reperfusion fluids after mouse cerebral IR injury increased glutathione levels and reduced reactive oxygen/nitrogen species (ROS/RNS) to improve neurological function. Plasma organic acids increased post-reperfusion injury, while administration of itaconate normalized these metabolites. In mouse cranial window models, itaconate significantly improved hemodynamics while reducing leukocyte adhesion. Further, itaconate supplementation increased survival in mice experiencing traumatic brain injury (TBI) and hemorrhagic shock.ConclusionsWe hypothesize that itaconate transiently inhibits SDH to gradually "awaken" mitochondrial function upon reperfusion that minimizes ROS and tissue damage. Collectively, our data indicate that itaconate acts as a mitochondrial regulator that controls redox metabolism to improve physiological outcomes associated with IR injury

Genomic and epigenomic mapping of leptin-responsive neuronal populations involved in body weight regulation.

Genome wide association studies (GWAS) in obesity have identified a large number of noncoding loci located near genes expressed in the central nervous system. However, due to the difficulties in isolating and characterizing specific neuronal subpopulations, few obesity-associated SNPs have been functionally characterized. Leptin responsive neurons in the hypothalamus are essential in controlling energy homeostasis and body weight. Here, we combine FACS-sorting of leptin-responsive hypothalamic neuron nuclei with genomic and epigenomic approaches (RNA-seq, ChIP-seq, ATAC-seq) to generate a comprehensive map of leptin-response specific regulatory elements, several of which overlap obesity-associated GWAS variants. We demonstrate the usefulness of our leptin-response neuron regulome, by functionally characterizing a novel enhancer near Socs3, a leptin response-associated transcription factor. We envision our data to serve as a useful resource and a blueprint for functionally characterizing obesity-associated SNPs in the hypothalamus