5,897 research outputs found
Sparse Coding Predicts Optic Flow Specificities of Zebrafish Pretectal Neurons
Zebrafish pretectal neurons exhibit specificities for large-field optic flow
patterns associated with rotatory or translatory body motion. We investigate
the hypothesis that these specificities reflect the input statistics of natural
optic flow. Realistic motion sequences were generated using computer graphics
simulating self-motion in an underwater scene. Local retinal motion was
estimated with a motion detector and encoded in four populations of
directionally tuned retinal ganglion cells, represented as two signed input
variables. This activity was then used as input into one of two learning
networks: a sparse coding network (competitive learning) and backpropagation
network (supervised learning). Both simulations develop specificities for optic
flow which are comparable to those found in a neurophysiological study (Kubo et
al. 2014), and relative frequencies of the various neuronal responses are best
modeled by the sparse coding approach. We conclude that the optic flow neurons
in the zebrafish pretectum do reflect the optic flow statistics. The predicted
vectorial receptive fields show typical optic flow fields but also "Gabor" and
dipole-shaped patterns that likely reflect difference fields needed for
reconstruction by linear superposition.Comment: Published Conference Paper from ICANN 2018, Rhode
Neural Architecture Search using Deep Neural Networks and Monte Carlo Tree Search
Neural Architecture Search (NAS) has shown great success in automating the
design of neural networks, but the prohibitive amount of computations behind
current NAS methods requires further investigations in improving the sample
efficiency and the network evaluation cost to get better results in a shorter
time. In this paper, we present a novel scalable Monte Carlo Tree Search (MCTS)
based NAS agent, named AlphaX, to tackle these two aspects. AlphaX improves the
search efficiency by adaptively balancing the exploration and exploitation at
the state level, and by a Meta-Deep Neural Network (DNN) to predict network
accuracies for biasing the search toward a promising region. To amortize the
network evaluation cost, AlphaX accelerates MCTS rollouts with a distributed
design and reduces the number of epochs in evaluating a network by transfer
learning, which is guided with the tree structure in MCTS. In 12 GPU days and
1000 samples, AlphaX found an architecture that reaches 97.84\% top-1 accuracy
on CIFAR-10, and 75.5\% top-1 accuracy on ImageNet, exceeding SOTA NAS methods
in both the accuracy and sampling efficiency. Particularly, we also evaluate
AlphaX on NASBench-101, a large scale NAS dataset; AlphaX is 3x and 2.8x more
sample efficient than Random Search and Regularized Evolution in finding the
global optimum. Finally, we show the searched architecture improves a variety
of vision applications from Neural Style Transfer, to Image Captioning and
Object Detection.Comment: To appear in the Thirty-Fourth AAAI conference on Artificial
Intelligence (AAAI-2020
Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods
Training neural networks is a challenging non-convex optimization problem,
and backpropagation or gradient descent can get stuck in spurious local optima.
We propose a novel algorithm based on tensor decomposition for guaranteed
training of two-layer neural networks. We provide risk bounds for our proposed
method, with a polynomial sample complexity in the relevant parameters, such as
input dimension and number of neurons. While learning arbitrary target
functions is NP-hard, we provide transparent conditions on the function and the
input for learnability. Our training method is based on tensor decomposition,
which provably converges to the global optimum, under a set of mild
non-degeneracy conditions. It consists of simple embarrassingly parallel linear
and multi-linear operations, and is competitive with standard stochastic
gradient descent (SGD), in terms of computational complexity. Thus, we propose
a computationally efficient method with guaranteed risk bounds for training
neural networks with one hidden layer.Comment: The tensor decomposition analysis is expanded, and the analysis of
ridge regression is added for recovering the parameters of last layer of
neural networ
- …