Are Out-of-Distribution Detection Methods Effective on Large-Scale Datasets?

Supervised classification methods often assume the train and test data distributions are the same and that all classes in the test set are present in the training set. However, deployed classifiers often require the ability to recognize inputs from outside the training set as unknowns. This problem has been studied under multiple paradigms including out-of-distribution detection and open set recognition. For convolutional neural networks, there have been two major approaches: 1) inference methods to separate knowns from unknowns and 2) feature space regularization strategies to improve model robustness to outlier inputs. There has been little effort to explore the relationship between the two approaches and directly compare performance on anything other than small-scale datasets that have at most 100 categories. Using ImageNet-1K and Places-434, we identify novel combinations of regularization and specialized inference methods that perform best across multiple outlier detection problems of increasing difficulty level. We found that input perturbation and temperature scaling yield the best performance on large scale datasets regardless of the feature space regularization strategy. Improving the feature space by regularizing against a background class can be helpful if an appropriate background class can be found, but this is impractical for large scale image classification datasets

SIESTA: Efficient Online Continual Learning with Sleep

In supervised continual learning, a deep neural network (DNN) is updated with an ever-growing data stream. Unlike the offline setting where data is shuffled, we cannot make any distributional assumptions about the data stream. Ideally, only one pass through the dataset is needed for computational efficiency. However, existing methods are inadequate and make many assumptions that cannot be made for real-world applications, while simultaneously failing to improve computational efficiency. In this paper, we propose a novel continual learning method, SIESTA based on wake/sleep framework for training, which is well aligned to the needs of on-device learning. The major goal of SIESTA is to advance compute efficient continual learning so that DNNs can be updated efficiently using far less time and energy. The principal innovations of SIESTA are: 1) rapid online updates using a rehearsal-free, backpropagation-free, and data-driven network update rule during its wake phase, and 2) expedited memory consolidation using a compute-restricted rehearsal policy during its sleep phase. For memory efficiency, SIESTA adapts latent rehearsal using memory indexing from REMIND. Compared to REMIND and prior arts, SIESTA is far more computationally efficient, enabling continual learning on ImageNet-1K in under 2 hours on a single GPU; moreover, in the augmentation-free setting it matches the performance of the offline learner, a milestone critical to driving adoption of continual learning in real-world applications.Comment: Accepted to TMLR 202

Thorium isotopes tracing the iron cycle at the Hawaii Ocean Time-series Station ALOHA

The role of iron as a limiting micronutrient motivates an effort to understand the supply and removal of lithogenic trace metals in the ocean. The long-lived thorium isotopes (²³²Th and ²³⁰Th) in seawater can be used to quantify the input of lithogenic metals attributable to the partial dissolution of aerosol dust. Thus, Th can help in disentangling the Fe cycle by providing an estimate of its ultimate supply and turnover rate. Here we present time-series (1994–2014) data on thorium isotopes and iron concentrations in seawater from the Hawaii Ocean Time-series Station ALOHA. By comparing Th-based dissolved Fe fluxes with measured dissolved Fe inventories, we derive Fe residence times of 6–12 months for the surface ocean. Therefore, Fe inventories in the surface ocean are sensitive to seasonal changes in dust input. Ultrafiltration results further reveal that Th has a much lower colloidal content than Fe does, despite a common source. On this basis, we suggest Fe colloids may be predominantly organic in composition, at least at Station ALOHA. In the deep ocean (>2 km), Fe approaches a solubility limit while Th, surprisingly, is continually leached from lithogenic particles. This distinction has implications for the relevance of Fe ligand availability in the deep ocean, but also suggests Th is not a good tracer for Fe in deep waters. While uncovering divergent behavior of these elements in the water column, this study finds that dissolved Th flux is a suitable proxy for the supply of Fe from dust in the remote surface ocean.National Science Foundation (U.S.) (Grant NS-OIA E-0424599