Explainable and Advisable Learning for Self-driving Vehicles

Deep neural perception and control networks are likely to be a key component of self-driving vehicles. These models need to be explainable - they should provide easy-to-interpret rationales for their behavior - so that passengers, insurance companies, law enforcement, developers, etc., can understand what triggered a particular behavior. Explanations may be triggered by the neural controller, namely introspective explanations, or informed by the neural controller's output, namely rationalizations. Our work has focused on the challenge of generating introspective explanations of deep models for self-driving vehicles. In Chapter 3, we begin by exploring the use of visual explanations. These explanations take the form of real-time highlighted regions of an image that causally influence the network's output (steering control). In the first stage, we use a visual attention model to train a convolution network end-to-end from images to steering angle. The attention model highlights image regions that potentially influence the network's output. Some of these are true influences, but some are spurious. We then apply a causal filtering step to determine which input regions actually influence the output. This produces more succinct visual explanations and more accurately exposes the network's behavior. In Chapter 4, we add an attention-based video-to-text model to produce textual explanations of model actions, e.g. "the car slows down because the road is wet". The attention maps of controller and explanation model are aligned so that explanations are grounded in the parts of the scene that mattered to the controller. We explore two approaches to attention alignment, strong- and weak-alignment. These explainable systems represent an externalization of tacit knowledge. The network's opaque reasoning is simplified to a situation-specific dependence on a visible object in the image. This makes them brittle and potentially unsafe in situations that do not match training data. In Chapter 5, we propose to address this issue by augmenting training data with natural language advice from a human. Advice includes guidance about what to do and where to attend. We present the first step toward advice-giving, where we train an end-to-end vehicle controller that accepts advice. The controller adapts the way it attends to the scene (visual attention) and the control (steering and speed). Further, in Chapter 6, we propose a new approach that learns vehicle control with the help of long-term (global) human advice. Specifically, our system learns to summarize its visual observations in natural language, predict an appropriate action response (e.g. "I see a pedestrian crossing, so I stop"), and predict the controls, accordingly

Optimal Pose and Shape Estimation for Category-level 3D Object Perception

We consider a category-level perception problem, where one is given 3D sensor data picturing an object of a given category (e.g. a car), and has to reconstruct the pose and shape of the object despite intra-class variability (i.e. different car models have different shapes). We consider an active shape model, where -- for an object category -- we are given a library of potential CAD models describing objects in that category, and we adopt a standard formulation where pose and shape estimation are formulated as a non-convex optimization. Our first contribution is to provide the first certifiably optimal solver for pose and shape estimation. In particular, we show that rotation estimation can be decoupled from the estimation of the object translation and shape, and we demonstrate that (i) the optimal object rotation can be computed via a tight (small-size) semidefinite relaxation, and (ii) the translation and shape parameters can be computed in closed-form given the rotation. Our second contribution is to add an outlier rejection layer to our solver, hence making it robust to a large number of misdetections. Towards this goal, we wrap our optimal solver in a robust estimation scheme based on graduated non-convexity. To further enhance robustness to outliers, we also develop the first graph-theoretic formulation to prune outliers in category-level perception, which removes outliers via convex hull and maximum clique computations; the resulting approach is robust to 70%-90% outliers. Our third contribution is an extensive experimental evaluation. Besides providing an ablation study on a simulated dataset and on the PASCAL3D+ dataset, we combine our solver with a deep-learned keypoint detector, and show that the resulting approach improves over the state of the art in vehicle pose estimation in the ApolloScape datasets

AnEnPi: Identification and annotation of analogous enzymes

Enzymes are responsible for the catalysis of the biochemical reactions in metabolic pathways. Analogous enzymes are able to catalyze the same reactions, but they present no significant sequence similarity at the primary level, and possibly different tertiary structures as well. They are thought to have arisen as the result of independent evolutionary events. A detailed study of analogous enzymes may reveal new catalytic mechanisms, add information about the origin and evolution of biochemical pathways and disclose potential targets for drug development. Results: In this work, we have constructed and implemented a new approach, AnEnPi (the Analogous Enzyme Pipeline), using a combination of bioinformatics tools like BLAST, HMMer, and in-house scripts, to assist in the identification, annotation, comparison and study of analogous and homologous enzymes. The algorithm for the detection of analogy is based i) on the construction of groups of homologous enzymes and ii) on the identification of cases where a given enzymatic activity is performed by two or more proteins without significant similarity between their primary structures. We applied this approach to a dataset obtained from KEGG Comprising all annotated enzymes, which resulted in the identification of 986 EC classes where putative analogy was detected (40.5% of all EC classes). AnEnPi is of considerable value in the construction of initial datasets that can be further curated, particularly in gene and genome annotation, in studies involving molecular evolution and metabolism and in the identification of new potential drug targets. Conclusion: AnEnPi is an efficient tool for detection and annotation of analogous enzymes and other enzymes in whole genomes

Austin also must be remembered. The Augustinian legacy in Milton's work

When I started working on this project, with a limited knowledge of Augustine, but determined to spot his presence in Miltonâs poetry, I was little aware of the intricacy of the relationship between the two authors. At this stage of my research, I do subscribe to Savoyeâs opinion, that this relationship is pervasive. However, one could safely add, it is as pervasive as it is hidden, primarily because of changed cultural paradigms, so that Miltonâs references are no longer familiar to the reader. As I have pointed out in my presentation of the state of the art, these articulations are hardly made explicit in Miltonâs Oeuvre and also in critical literature they are hardly brought to the surface. My objective has been to make them a little more visible. I have started my own process of discovery from the works where Milton more openly (but not completely) acknowledges his Augustinian sources, although arguably mediated. As concerns Samson Agonistes, I have presented a reading through Augustinian lenses. I am by no means claiming that mine is the best of all possible readings, but through those lenses I have been able to see a coherence, in Miltonâs dramatic poem, that is not generally recognized. On the other hand, I thoroughly agree that âone cannot simply take any English poet and turn the post-structuralist critical machine loose on him or her in good faithâ. In particular, I am aware that I have read Miltonâs works against the current critical grain which, with a powerful turn impressed by Empsonâs Miltonâs God, is continually surfacing Miltonâs idiosyncrasies in order to cancel the received picture of a Christian author. Rather, I agree with Cirillo that Miltonâs perspective is that of âa professed Christian poet whose Christian consciousness, no matter how heterodox, colored virtually everything he wrote.â.We may ask, echoing Febvre on Rabelais, âMais de quel christianisme? In accordance with very traditional, even traditionalist Milton Criticism, I think it can safely be stated that Milton is a post-Reformation religious author, and one whose endeavour to âjustify the ways of God to menâ had to come to terms with the difficult task to find signs of providential history in the aftermath of a civil war and in the adverse context of the Restoration. His last published poems deal with this problem in different terms. As readers, we can come to different conclusions as to the texts. Behind them there is the man, âest abyssus humanae conscientiae,â in front of which, after Augustine, I can only say: "nescio"

Coinductive subtyping for abstract compilation of object-oriented languages into Horn formulas

In recent work we have shown how it is possible to define very precise type systems for object-oriented languages by abstractly compiling a program into a Horn formula f. Then type inference amounts to resolving a certain goal w.r.t. the coinductive (that is, the greatest) Herbrand model of f. Type systems defined in this way are idealized, since in the most interesting instantiations both the terms of the coinductive Herbrand universe and goal derivations cannot be finitely represented. However, sound and quite expressive approximations can be implemented by considering only regular terms and derivations. In doing so, it is essential to introduce a proper subtyping relation formalizing the notion of approximation between types. In this paper we study a subtyping relation on coinductive terms built on union and object type constructors. We define an interpretation of types as set of values induced by a quite intuitive relation of membership of values to types, and prove that the definition of subtyping is sound w.r.t. subset inclusion between type interpretations. The proof of soundness has allowed us to simplify the notion of contractive derivation and to discover that the previously given definition of subtyping did not cover all possible representations of the empty type

Using Interactive Slides and Videos to Engage Students

We learn best by interacting with the material we are trying to learn. I have developed a suite of materials for engaging students in learning digital design and computer architecture. These materials include a textbook, exercises, hands-on labs, and lectures that use interactive ink annotations. I have also developed online materials (video lectures, practice exercises, quizzes, etc.) to support online teaching of the material. Instructors may use one, several, or all of these materials. During lecture, students benefit from pre-drawn figures on the slides so that they can focus on understanding (instead of just copying down) the material. However, with the annotation tool, the instructor and students can add ink-based notations during the lecture to interact with and modify the provided designs and circuits in real-time. This interactive classroom learning is supplemented with readings, written exercises, and hands-on labs where they can explore and practice the principles they learned during lecture at their own pace.https://digitalscholarship.unlv.edu/btp_expo/1098/thumbnail.jp