97,153 research outputs found
Deep supervised learning using local errors
Error backpropagation is a highly effective mechanism for learning
high-quality hierarchical features in deep networks. Updating the features or
weights in one layer, however, requires waiting for the propagation of error
signals from higher layers. Learning using delayed and non-local errors makes
it hard to reconcile backpropagation with the learning mechanisms observed in
biological neural networks as it requires the neurons to maintain a memory of
the input long enough until the higher-layer errors arrive. In this paper, we
propose an alternative learning mechanism where errors are generated locally in
each layer using fixed, random auxiliary classifiers. Lower layers could thus
be trained independently of higher layers and training could either proceed
layer by layer, or simultaneously in all layers using local error information.
We address biological plausibility concerns such as weight symmetry
requirements and show that the proposed learning mechanism based on fixed,
broad, and random tuning of each neuron to the classification categories
outperforms the biologically-motivated feedback alignment learning technique on
the MNIST, CIFAR10, and SVHN datasets, approaching the performance of standard
backpropagation. Our approach highlights a potential biological mechanism for
the supervised, or task-dependent, learning of feature hierarchies. In
addition, we show that it is well suited for learning deep networks in custom
hardware where it can drastically reduce memory traffic and data communication
overheads
End-to-End Navigation in Unknown Environments using Neural Networks
We investigate how a neural network can learn perception actions loops for
navigation in unknown environments. Specifically, we consider how to learn to
navigate in environments populated with cul-de-sacs that represent convex local
minima that the robot could fall into instead of finding a set of feasible
actions that take it to the goal. Traditional methods rely on maintaining a
global map to solve the problem of over coming a long cul-de-sac. However, due
to errors induced from local and global drift, it is highly challenging to
maintain such a map for long periods of time. One way to mitigate this problem
is by using learning techniques that do not rely on hand engineered map
representations and instead output appropriate control policies directly from
their sensory input. We first demonstrate that such a problem cannot be solved
directly by deep reinforcement learning due to the sparse reward structure of
the environment. Further, we demonstrate that deep supervised learning also
cannot be used directly to solve this problem. We then investigate network
models that offer a combination of reinforcement learning and supervised
learning and highlight the significance of adding fully differentiable memory
units to such networks. We evaluate our networks on their ability to generalize
to new environments and show that adding memory to such networks offers huge
jumps in performanceComment: Workshop on Learning Perception and Control for Autonomous Flight:
Safety, Memory and Efficiency, Robotics Science and Systems 201
A Semi-Supervised Two-Stage Approach to Learning from Noisy Labels
The recent success of deep neural networks is powered in part by large-scale
well-labeled training data. However, it is a daunting task to laboriously
annotate an ImageNet-like dateset. On the contrary, it is fairly convenient,
fast, and cheap to collect training images from the Web along with their noisy
labels. This signifies the need of alternative approaches to training deep
neural networks using such noisy labels. Existing methods tackling this problem
either try to identify and correct the wrong labels or reweigh the data terms
in the loss function according to the inferred noisy rates. Both strategies
inevitably incur errors for some of the data points. In this paper, we contend
that it is actually better to ignore the labels of some of the data points than
to keep them if the labels are incorrect, especially when the noisy rate is
high. After all, the wrong labels could mislead a neural network to a bad local
optimum. We suggest a two-stage framework for the learning from noisy labels.
In the first stage, we identify a small portion of images from the noisy
training set of which the labels are correct with a high probability. The noisy
labels of the other images are ignored. In the second stage, we train a deep
neural network in a semi-supervised manner. This framework effectively takes
advantage of the whole training set and yet only a portion of its labels that
are most likely correct. Experiments on three datasets verify the effectiveness
of our approach especially when the noisy rate is high
- …