1,945 research outputs found
Neural Network Memory Architectures for Autonomous Robot Navigation
This paper highlights the significance of including memory structures in
neural networks when the latter are used to learn perception-action loops for
autonomous robot navigation. Traditional navigation approaches rely on global
maps of the environment to overcome cul-de-sacs and plan feasible motions. Yet,
maintaining an accurate global map may be challenging in real-world settings. A
possible way to mitigate this limitation is to use learning techniques that
forgo hand-engineered map representations and infer appropriate control
responses directly from sensed information. An important but unexplored aspect
of such approaches is the effect of memory on their performance. This work is a
first thorough study of memory structures for deep-neural-network-based robot
navigation, and offers novel tools to train such networks from supervision and
quantify their ability to generalize to unseen scenarios. We analyze the
separation and generalization abilities of feedforward, long short-term memory,
and differentiable neural computer networks. We introduce a new method to
evaluate the generalization ability by estimating the VC-dimension of networks
with a final linear readout layer. We validate that the VC estimates are good
predictors of actual test performance. The reported method can be applied to
deep learning problems beyond robotics
Asynchronous spiking neurons, the natural key to exploit temporal sparsity
Inference of Deep Neural Networks for stream signal (Video/Audio) processing in edge devices is still challenging. Unlike the most state of the art inference engines which are efficient for static signals, our brain is optimized for real-time dynamic signal processing. We believe one important feature of the brain (asynchronous state-full processing) is the key to its excellence in this domain. In this work, we show how asynchronous processing with state-full neurons allows exploitation of the existing sparsity in natural signals. This paper explains three different types of sparsity and proposes an inference algorithm which exploits all types of sparsities in the execution of already trained networks. Our experiments in three different applications (Handwritten digit recognition, Autonomous Steering and Hand-Gesture recognition) show that this model of inference reduces the number of required operations for sparse input data by a factor of one to two orders of magnitudes. Additionally, due to fully asynchronous processing this type of inference can be run on fully distributed and scalable neuromorphic hardware platforms
The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism
We present scalable hybrid-parallel algorithms for training large-scale 3D
convolutional neural networks. Deep learning-based emerging scientific
workflows often require model training with large, high-dimensional samples,
which can make training much more costly and even infeasible due to excessive
memory usage. We solve these challenges by extensively applying hybrid
parallelism throughout the end-to-end training pipeline, including both
computations and I/O. Our hybrid-parallel algorithm extends the standard data
parallelism with spatial parallelism, which partitions a single sample in the
spatial domain, realizing strong scaling beyond the mini-batch dimension with a
larger aggregated memory capacity. We evaluate our proposed training algorithms
with two challenging 3D CNNs, CosmoFlow and 3D U-Net. Our comprehensive
performance studies show that good weak and strong scaling can be achieved for
both networks using up 2K GPUs. More importantly, we enable training of
CosmoFlow with much larger samples than previously possible, realizing an
order-of-magnitude improvement in prediction accuracy.Comment: 12 pages, 10 figure
- …