Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testing

We have recently witnessed tremendous success of Machine Learning (ML) in practical applications. Computer vision, speech recognition and language translation have all seen a near human level performance. We expect, in the near future, most business applications will have some form of ML. However, testing such applications is extremely challenging and would be very expensive if we follow today's methodologies. In this work, we present an articulation of the challenges in testing ML based applications. We then present our solution approach, based on the concept of Metamorphic Testing, which aims to identify implementation bugs in ML based image classifiers. We have developed metamorphic relations for an application based on Support Vector Machine and a Deep Learning based application. Empirical validation showed that our approach was able to catch 71% of the implementation bugs in the ML applications.Comment: Published at 27th ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2018

Synthetic Gene Circuits: Design with Directed Evolution

Synthetic circuits offer great promise for generating insights into nature's underlying design principles or forward engineering novel biotechnology applications. However, construction of these circuits is not straightforward. Synthetic circuits generally consist of components optimized to function in their natural context, not in the context of the synthetic circuit. Combining mathematical modeling with directed evolution offers one promising means for addressing this problem. Modeling identifies mutational targets and limits the evolutionary search space for directed evolution, which alters circuit performance without the need for detailed biophysical information. This review examines strategies for integrating modeling and directed evolution and discusses the utility and limitations of available methods

Transcription factor target prediction using multiple short expression time series from Arabidopsis thaliana

BACKGROUND: The central role of transcription factors (TFs) in higher eukaryotes has led to much interest in deciphering transcriptional regulatory interactions. Even in the best case, experimental identification of TF target genes is error prone, and has been shown to be improved by considering additional forms of evidence such as expression data. Previous expression based methods have not explicitly tried to associate TFs with their targets and therefore largely ignored the treatment specific and time dependent nature of transcription regulation. RESULTS: In this study we introduce CERMT, Covariance based Extraction of Regulatory targets using Multiple Time series. Using simulated and real data we show that using multiple expression time series, selecting treatments in which the TF responds, allowing time shifts between TFs and their targets and using covariance to identify highly responding genes appear to be a good strategy. We applied our method to published TF - target gene relationships determined using expression profiling on TF mutants and show that in most cases we obtain significant target gene enrichment and in half of the cases this is sufficient to deliver a usable list of high-confidence target genes. CONCLUSION: CERMT could be immediately useful in refining possible target genes of candidate TFs using publicly available data, particularly for organisms lacking comprehensive TF binding data. In the future, we believe its incorporation with other forms of evidence may improve integrative genome-wide predictions of transcriptional networks

A Computational-Experimental Approach Identifies Mutations That Enhance Surface Expression of an Oseltamivir-Resistant Influenza Neuraminidase

The His274 → Tyr (H274Y) oseltamivir (Tamiflu) resistance mutation causes a substantial decrease in the total levels of surface-expressed neuraminidase protein and activity in early isolates of human seasonal H1N1 influenza, and in the swine-origin pandemic H1N1. In seasonal H1N1, H274Y only became widespread after the occurrence of secondary mutations that counteracted this decrease. H274Y is currently rare in pandemic H1N1, and it remains unclear whether secondary mutations exist that might similarly counteract the decreased neuraminidase surface expression associated with this resistance mutation in pandemic H1N1. Here we investigate the possibility of predicting such secondary mutations. We first test the ability of several computational approaches to retrospectively identify the secondary mutations that enhanced levels of surface-expressed neuraminidase protein and activity in seasonal H1N1 shortly before the emergence of oseltamivir resistance. We then use the most successful computational approach to predict a set of candidate secondary mutations to the pandemic H1N1 neuraminidase. We experimentally screen these mutations, and find that several of them do indeed partially counteract the decrease in neuraminidase surface expression caused by H274Y. Two of the secondary mutations together restore surface-expressed neuraminidase activity to wildtype levels, and also eliminate the very slight decrease in viral growth in tissue-culture caused by H274Y. Our work therefore demonstrates a combined computational-experimental approach for identifying mutations that enhance neuraminidase surface expression, and describes several specific mutations with the potential to be of relevance to the spread of oseltamivir resistance in pandemic H1N1

Is the Stack Distance Between Test Case and Method Correlated With Test Effectiveness?

Mutation testing is a means to assess the effectiveness of a test suite and its outcome is considered more meaningful than code coverage metrics. However, despite several optimizations, mutation testing requires a significant computational effort and has not been widely adopted in industry. Therefore, we study in this paper whether test effectiveness can be approximated using a more light-weight approach. We hypothesize that a test case is more likely to detect faults in methods that are close to the test case on the call stack than in methods that the test case accesses indirectly through many other methods. Based on this hypothesis, we propose the minimal stack distance between test case and method as a new test measure, which expresses how close any test case comes to a given method, and study its correlation with test effectiveness. We conducted an empirical study with 21 open-source projects, which comprise in total 1.8 million LOC, and show that a correlation exists between stack distance and test effectiveness. The correlation reaches a strength up to 0.58. We further show that a classifier using the minimal stack distance along with additional easily computable measures can predict the mutation testing result of a method with 92.9% precision and 93.4% recall. Hence, such a classifier can be taken into consideration as a light-weight alternative to mutation testing or as a preceding, less costly step to that.Comment: EASE 201

Spectrum-Based Fault Localization in Model Transformations

Model transformations play a cornerstone role in Model-Driven Engineering (MDE), as they provide the essential mechanisms for manipulating and transforming models. The correctness of software built using MDE techniques greatly relies on the correctness of model transformations. However, it is challenging and error prone to debug them, and the situation gets more critical as the size and complexity of model transformations grow, where manual debugging is no longer possible. Spectrum-Based Fault Localization (SBFL) uses the results of test cases and their corresponding code coverage information to estimate the likelihood of each program component (e.g., statements) of being faulty. In this article we present an approach to apply SBFL for locating the faulty rules in model transformations. We evaluate the feasibility and accuracy of the approach by comparing the effectiveness of 18 different stateof- the-art SBFL techniques at locating faults in model transformations. Evaluation results revealed that the best techniques, namely Kulcynski2, Mountford, Ochiai, and Zoltar, lead the debugger to inspect a maximum of three rules to locate the bug in around 74% of the cases. Furthermore, we compare our approach with a static approach for fault localization in model transformations, observing a clear superiority of the proposed SBFL-based method.Comisión Interministerial de Ciencia y Tecnología TIN2015-70560-RJunta de Andalucía P12-TIC-186

Vesicular stomatitis virus glycoprotein is necessary for H-2-restricted lysis of infected cells by cytotoxic T lymphocytes

Vesicular stomatitis virus (VSV) elicited cytotoxic thymus-derived lymphocytes (CTLs) in mice of the BALB/c and three congenic strains (BALB.b, BALB.k, BALB.HTG). CTL lysis of VSV-infected fibroblasts from the four strains was restricted by the target cells' major histocompatibility complex (H-2). Target cells were also infected with two temperature-sensitive mutants of VSV, tsM and tsG in which, respectively, the viral matrix protein and glycoprotein are not expressed at 39 degrees (restrictive temperature) on the infected cell's surface membrane. At the restrictive temperature, cells infected with wild-type VSV or tsM were lysed by CTLs, but cells infected with tsG were not. The requirement for the glycoprotein on the target cell was also evident from the ability of antisera to the glycoprotein to block completely CTL lysis of VSV-infected cells