2,534 research outputs found
Human uncertainty makes classification more robust
The classification performance of deep neural networks has begun to asymptote
at near-perfect levels. However, their ability to generalize outside the
training set and their robustness to adversarial attacks have not. In this
paper, we make progress on this problem by training with full label
distributions that reflect human perceptual uncertainty. We first present a new
benchmark dataset which we call CIFAR10H, containing a full distribution of
human labels for each image of the CIFAR10 test set. We then show that, while
contemporary classifiers fail to exhibit human-like uncertainty on their own,
explicit training on our dataset closes this gap, supports improved
generalization to increasingly out-of-training-distribution test datasets, and
confers robustness to adversarial attacks.Comment: In Proceedings of the 2019 IEEE International Conference on Computer
Vision (ICCV
Inherent Weight Normalization in Stochastic Neural Networks
Multiplicative stochasticity such as Dropout improves the robustness and
generalizability of deep neural networks. Here, we further demonstrate that
always-on multiplicative stochasticity combined with simple threshold neurons
are sufficient operations for deep neural networks. We call such models Neural
Sampling Machines (NSM). We find that the probability of activation of the NSM
exhibits a self-normalizing property that mirrors Weight Normalization, a
previously studied mechanism that fulfills many of the features of Batch
Normalization in an online fashion. The normalization of activities during
training speeds up convergence by preventing internal covariate shift caused by
changes in the input distribution. The always-on stochasticity of the NSM
confers the following advantages: the network is identical in the inference and
learning phases, making the NSM suitable for online learning, it can exploit
stochasticity inherent to a physical substrate such as analog non-volatile
memories for in-memory computing, and it is suitable for Monte Carlo sampling,
while requiring almost exclusively addition and comparison operations. We
demonstrate NSMs on standard classification benchmarks (MNIST and CIFAR) and
event-based classification benchmarks (N-MNIST and DVS Gestures). Our results
show that NSMs perform comparably or better than conventional artificial neural
networks with the same architecture
Interoceptive robustness through environment-mediated morphological development
Typically, AI researchers and roboticists try to realize intelligent behavior
in machines by tuning parameters of a predefined structure (body plan and/or
neural network architecture) using evolutionary or learning algorithms. Another
but not unrelated longstanding property of these systems is their brittleness
to slight aberrations, as highlighted by the growing deep learning literature
on adversarial examples. Here we show robustness can be achieved by evolving
the geometry of soft robots, their control systems, and how their material
properties develop in response to one particular interoceptive stimulus
(engineering stress) during their lifetimes. By doing so we realized robots
that were equally fit but more robust to extreme material defects (such as
might occur during fabrication or by damage thereafter) than robots that did
not develop during their lifetimes, or developed in response to a different
interoceptive stimulus (pressure). This suggests that the interplay between
changes in the containing systems of agents (body plan and/or neural
architecture) at different temporal scales (evolutionary and developmental)
along different modalities (geometry, material properties, synaptic weights)
and in response to different signals (interoceptive and external perception)
all dictate those agents' abilities to evolve or learn capable and robust
strategies
Sparsity-based Defense against Adversarial Attacks on Linear Classifiers
Deep neural networks represent the state of the art in machine learning in a
growing number of fields, including vision, speech and natural language
processing. However, recent work raises important questions about the
robustness of such architectures, by showing that it is possible to induce
classification errors through tiny, almost imperceptible, perturbations.
Vulnerability to such "adversarial attacks", or "adversarial examples", has
been conjectured to be due to the excessive linearity of deep networks. In this
paper, we study this phenomenon in the setting of a linear classifier, and show
that it is possible to exploit sparsity in natural data to combat
-bounded adversarial perturbations. Specifically, we demonstrate
the efficacy of a sparsifying front end via an ensemble averaged analysis, and
experimental results for the MNIST handwritten digit database. To the best of
our knowledge, this is the first work to show that sparsity provides a
theoretically rigorous framework for defense against adversarial attacks.Comment: Published in IEEE International Symposium on Information Theory
(ISIT) 201
- …