18,393 research outputs found
Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
Current learning machines have successfully solved hard application problems,
reaching high accuracy and displaying seemingly "intelligent" behavior. Here we
apply recent techniques for explaining decisions of state-of-the-art learning
machines and analyze various tasks from computer vision and arcade games. This
showcases a spectrum of problem-solving behaviors ranging from naive and
short-sighted, to well-informed and strategic. We observe that standard
performance evaluation metrics can be oblivious to distinguishing these diverse
problem solving behaviors. Furthermore, we propose our semi-automated Spectral
Relevance Analysis that provides a practically effective way of characterizing
and validating the behavior of nonlinear learning machines. This helps to
assess whether a learned model indeed delivers reliably for the problem that it
was conceived for. Furthermore, our work intends to add a voice of caution to
the ongoing excitement about machine intelligence and pledges to evaluate and
judge some of these recent successes in a more nuanced manner.Comment: Accepted for publication in Nature Communication
Learning a local-variable model of aromatic and conjugated systems
A collection of new
approaches to building and training neural
networks, collectively referred to as deep learning, are attracting
attention in theoretical chemistry. Several groups aim to replace
computationally expensive <i>ab initio</i> quantum mechanics
calculations with learned estimators. This raises questions about
the representability of complex quantum chemical systems with neural
networks. Can local-variable models efficiently approximate nonlocal
quantum chemical features? Here, we find that convolutional architectures,
those that only aggregate information locally, cannot efficiently
represent aromaticity and conjugation in large systems. They cannot
represent long-range nonlocality known to be important in quantum
chemistry. This study uses aromatic and conjugated systems computed
from molecule graphs, though reproducing quantum simulations is the
ultimate goal. This task, by definition, is both computable and known
to be important to chemistry. The failure of convolutional architectures
on this focused task calls into question their use in modeling quantum
mechanics. To remedy this heretofore unrecognized deficiency, we introduce
a new architecture that propagates information back and forth in waves
of nonlinear computation. This architecture is still a local-variable
model, and it is both computationally and representationally efficient,
processing molecules in sublinear time with far fewer parameters than
convolutional networks. Wave-like propagation models aromatic and
conjugated systems with high accuracy, and even models the impact
of small structural changes on large molecules. This new architecture
demonstrates that some nonlocal features of quantum chemistry can
be efficiently represented in local variable models
Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition
The primate visual system achieves remarkable visual object recognition
performance even in brief presentations and under changes to object exemplar,
geometric transformations, and background variation (a.k.a. core visual object
recognition). This remarkable performance is mediated by the representation
formed in inferior temporal (IT) cortex. In parallel, recent advances in
machine learning have led to ever higher performing models of object
recognition using artificial deep neural networks (DNNs). It remains unclear,
however, whether the representational performance of DNNs rivals that of the
brain. To accurately produce such a comparison, a major difficulty has been a
unifying metric that accounts for experimental limitations such as the amount
of noise, the number of neural recording sites, and the number trials, and
computational limitations such as the complexity of the decoding classifier and
the number of classifier training examples. In this work we perform a direct
comparison that corrects for these experimental limitations and computational
considerations. As part of our methodology, we propose an extension of "kernel
analysis" that measures the generalization accuracy as a function of
representational complexity. Our evaluations show that, unlike previous
bio-inspired models, the latest DNNs rival the representational performance of
IT cortex on this visual object recognition task. Furthermore, we show that
models that perform well on measures of representational performance also
perform well on measures of representational similarity to IT and on measures
of predicting individual IT multi-unit responses. Whether these DNNs rely on
computational mechanisms similar to the primate visual system is yet to be
determined, but, unlike all previous bio-inspired models, that possibility
cannot be ruled out merely on representational performance grounds.Comment: 35 pages, 12 figures, extends and expands upon arXiv:1301.353
- …