HAct: Out-of-Distribution Detection with Neural Net Activation Histograms

We propose a simple, efficient, and accurate method for detecting out-of-distribution (OOD) data for trained neural networks, a potential first step in methods for OOD generalization. We propose a novel descriptor, HAct - activation histograms, for OOD detection, that is, probability distributions (approximated by histograms) of output values of neural network layers under the influence of incoming data. We demonstrate that HAct is significantly more accurate than state-of-the-art on multiple OOD image classification benchmarks. For instance, our approach achieves a true positive rate (TPR) of 95% with only 0.05% false-positives using Resnet-50 on standard OOD benchmarks, outperforming previous state-of-the-art by 20.66% in the false positive rate (at the same TPR of 95%). The low computational complexity and the ease of implementation make HAct suitable for online implementation in monitoring deployed neural networks in practice at scale

Tubular Surface Evolution for Segmentation of the Cingulum Bundle From DW-MRI

Presented at the 2nd MICCAI Workshop on Mathematical Foundations of Computational Anatomy: Geometrical and Statistical Methods for Biological Shape Variability Modeling, September 6th, 2008, Kimmel Center, New York, USA.This work provides a framework for modeling and extracting the Cingulum Bundle (CB) from Diffusion-Weighted Imagery (DW-MRI) of the brain. The CB is a tube-like structure in the brain that is of potentially of tremendous importance to clinicians since it may be helpful in diagnosing Schizophrenia. This structure consists of a collection of fibers in the brain that have locally similar diffusion patterns, but vary globally. Standard region-based segmentation techniques adapted to DW-MRI are not suitable here because the diffusion pattern of the CB cannot be described by a global set of simple statistics. Active surface models extended to DW-MRI are not suitable since they allow for arbitrary deformations that give rise to unlikely shapes, which do not respect the tubular geometry of the CB. In this work, we explicitly model the CB as a tube-like surface and construct a general class of energies defined on tube-like surfaces. An example energy of our framework is optimized by a tube that encloses a region that has locally similar diffusion patterns, which differ from the diffusion patterns immediately outside. Modeling the CB as a tube-like surface is a natural shape prior. Since a tube is characterized by a center-line and a radius function, the method is reduced to a 4D (center-line plus radius) curve evolution that is computationally much less costly than an arbitrary surface evolution. The method also provides the center-line of CB, which is potentially of clinical significance

Surprising Instabilities in Training Deep Networks and a Theoretical Analysis

We discover restrained numerical instabilities in current training practices of deep networks with SGD. We show numerical error (on the order of the smallest floating point bit) induced from floating point arithmetic in training deep nets can be amplified significantly and result in significant test accuracy variance, comparable to the test accuracy variance due to stochasticity in SGD. We show how this is likely traced to instabilities of the optimization dynamics that are restrained, i.e., localized over iterations and regions of the weight tensor space. We do this by presenting a theoretical framework using numerical analysis of partial differential equations (PDE), and analyzing the gradient descent PDE of a simplified convolutional neural network (CNN). We show that it is stable only under certain conditions on the learning rate and weight decay. We reproduce the localized instabilities in the PDE for the simplified network, which arise when the conditions are violated