Formal Analysis and Redesign of a Neural Network-Based Aircraft Taxiing System with VerifAI

We demonstrate a unified approach to rigorous design of safety-critical autonomous systems using the VerifAI toolkit for formal analysis of AI-based systems. VerifAI provides an integrated toolchain for tasks spanning the design process, including modeling, falsification, debugging, and ML component retraining. We evaluate all of these applications in an industrial case study on an experimental autonomous aircraft taxiing system developed by Boeing, which uses a neural network to track the centerline of a runway. We define runway scenarios using the Scenic probabilistic programming language, and use them to drive tests in the X-Plane flight simulator. We first perform falsification, automatically finding environment conditions causing the system to violate its specification by deviating significantly from the centerline (or even leaving the runway entirely). Next, we use counterexample analysis to identify distinct failure cases, and confirm their root causes with specialized testing. Finally, we use the results of falsification and debugging to retrain the network, eliminating several failure cases and improving the overall performance of the closed-loop system.Comment: Full version of a CAV 2020 pape

Methods for cost-sensitive learning

Many approaches for achieving intelligent behavior of automated (computer) systems involve components that learn from past experience. This dissertation studies computational methods for learning from examples, for classification and for decision making, when the decisions have different non-zero costs associated with them. Many practical applications of learning algorithms, including transaction monitoring, fraud detection, intrusion detection, and medical diagnosis, have such non-uniform costs, and there is a great need for new methods that can handle them. This dissertation discusses two approaches to cost-sensitive classification: input data weighting and conditional density estimation. The first method assigns a weight to each training example in order to force the learning algorithm (which is otherwise unchanged) to pay more attention to examples with higher misclassification costs. The dissertation discusses several different weighting methods and concludes that a method that gives higher weight to examples from rarer classes works quite well. Another algorithm that gave good results was a wrapper method that applies Powell's gradient-free algorithm to optimize the input weights. The second approach to cost-sensitive classification is conditional density estimation. In this approach, the output of the learning algorithm is a classifier that estimates, for a new data point, the probability that it belongs to each of the classes. These probability estimates can be combined with a cost matrix to make decisions that minimize the expected cost. The dissertation presents a new algorithm, bagged lazy option trees (B-LOTs), that gives better probability estimates than any previous method based on decision trees. In order to evaluate cost-sensitive classification methods, appropriate statistical methods are needed. The dissertation presents two new statistical procedures: BLOTs provides a confidence interval on the expected cost of a classifier, and BDELTACOST provides a confidence interval on the difference in expected costs of two classifiers. These methods are applied to a large set of experimental studies to evaluate and compare the cost-sensitive methods presented in this dissertation. Finally, the dissertation describes the application of the B-LOTs to a problem of predicting the stability of river channels. In this study, B-LOTs were shown to be superior to other methods in cases where the classes have very different frequencies a situation that arises frequently in cost-sensitive classification problems

Detection of mitochondrial DNA (mtDNA) mutations

The maternally inherited mitochondrial DNA (mtDNA) is a circular 16,569-bp double stranded DNA that encodes 37 genes, twenty-four of which (2 rRNA and 22 tRNA) are necessary for transcription and translation of 13 polypeptides that are all subunits of respiratory chain. Pathogenic mutations of mtDNA cause respiratory chain dysfunction, and are the underlying defect in an ever-increasing number of mtDNA-related encephalomyopathies with distinct phenotypes. In this chapter, we present an overview of mtDNA mutations and describe the molecular techniques currently employed in our laboratory to detect two types of mtDNA mutations: Single-large scale rearrangements and point mutations

The metabolome regulates the epigenetic landscape during naive-to-primed human embryonic stem cell transition.

For nearly a century developmental biologists have recognized that cells from embryos can differ in their potential to differentiate into distinct cell types. Recently, it has been recognized that embryonic stem cells derived from both mice and humans exhibit two stable yet epigenetically distinct states of pluripotency: naive and primed. We now show that nicotinamide N-methyltransferase (NNMT) and the metabolic state regulate pluripotency in human embryonic stem cells (hESCs).  Specifically, in naive hESCs, NNMT and its enzymatic product 1-methylnicotinamide are highly upregulated, and NNMT is required for low S-adenosyl methionine (SAM) levels and the H3K27me3 repressive state. NNMT consumes SAM in naive cells, making it unavailable for histone methylation that represses Wnt and activates the HIF pathway in primed hESCs. These data support the hypothesis that the metabolome regulates the epigenetic landscape of the earliest steps in human development

Culling sick mitochondria from the herd

The PINK1–Parkin pathway plays a critical role in mitochondrial quality control by selectively targeting damaged mitochondria for autophagy. In this issue, Tanaka et al. (2010. J. Cell Biol. doi: 10.1083/jcb.201007013) demonstrate that the AAA-type ATPase p97 acts downstream of PINK1 and Parkin to segregate fusion-incompetent mitochondria for turnover. p97 acts by targeting the mitochondrial fusion-promoting factor mitofusin for degradation through an endoplasmic reticulum–associated degradation (ERAD)-like mechanism

Bootstrap methods for the cost-sensitive evaluation of classifiers

Many machine learning applications require classifiers that minimize an asymmetric cost function rather than the misclassification rate, and several recent papers have addressed this problem. However, these papers have either applied no statistical testing or have applied statistical methods that are not appropriate for the cost-sensitive setting. Without good statistical methods, it is difficult to tell whether these new cost-sensitive methods are better than existing methods that ignore costs, and it is also difficult to tell whether one cost-sensitive method is better than another. To rectify this problem, this paper presents two statistical methods for the cost-sensitive setting. The first constructs a confidence interval for the expected cost of a single classifier. The second constructs a confidence interval for the expected difference in costs of two classifiers. In both cases, the basic idea is to separate the problem of estimating the probabilities of each cell in the confusion matrix (which is independent of the cost matrix) from the problem of computing the expected cost. We show experimentally that these bootstrap tests work better than applying standard z tests based on the normal distribution

Transmission of mitochondrial DNA following assisted reproduction and nuclear transfer

Review of the articleMitochondria are the organelles responsible for producing the majority of a cell's ATP and also play an essential role in gamete maturation and embryo development. ATP production within the mitochondria is dependent on proteins encoded by both the nuclear and the mitochondrial genomes, therefore co-ordination between the two genomes is vital for cell survival. To assist with this co-ordination, cells normally contain only one type of mitochondrial DNA (mtDNA) termed homoplasmy. Occasionally, however, two or more types of mtDNA are present termed heteroplasmy. This can result from a combination of mutant and wild-type mtDNA molecules or from a combination of wild-type mtDNA variants. As heteroplasmy can result in mitochondrial disease, various mechanisms exist in the natural fertilization process to ensure the maternal-only transmission of mtDNA and the maintenance of homoplasmy in future generations. However, there is now an increasing use of invasive oocyte reconstruction protocols, which tend to bypass mechanisms for the maintenance of homoplasmy, potentially resulting in the transmission of either form of mtDNA heteroplasmy. Indeed, heteroplasmy caused by combinations of wild-type variants has been reported following cytoplasmic transfer (CT) in the human and following nuclear transfer (NT) in various animal species. Other techniques, such as germinal vesicle transfer and pronuclei transfer, have been proposed as methods of preventing transmission of mitochondrial diseases to future generations. However, resulting embryos and offspring may contain mtDNA heteroplasmy, which itself could result in mitochondrial disease. It is therefore essential that uniparental transmission of mtDNA is ensured before these techniques are used therapeutically

BoostingTree: parallel selection of weak learners in boosting, with application to ranking

Boosting algorithms have been found successful in many areas of machine learning and, in particular, in ranking. For typical classes of weak learners used in boosting (such as decision stumps or trees), a large feature space can slow down the training, while a long sequence of weak hypotheses combined by boosting can result in a computationally expensive model. In this paper we propose a strategy that builds several sequences of weak hypotheses in parallel, and extends the ones that are likely to yield a good model. The weak hypothesis sequences are arranged in a boosting tree, and new weak hypotheses are added to promising nodes (both leaves and inner nodes) of the tree using some randomized method. Theoretical results show that the proposed algorithm asymptotically achieves the performance of the base boosting algorithm applied. Experiments are provided in ranking web documents and move ordering in chess, and the results indicate that the new strategy yields better performance when the length of the sequence is limited, and converges to similar performance as the original boosting algorithms otherwise. © 2013 The Author(s)