1,783 research outputs found
Direct Feedback Alignment with Sparse Connections for Local Learning
Recent advances in deep neural networks (DNNs) owe their success to training
algorithms that use backpropagation and gradient-descent. Backpropagation,
while highly effective on von Neumann architectures, becomes inefficient when
scaling to large networks. Commonly referred to as the weight transport
problem, each neuron's dependence on the weights and errors located deeper in
the network require exhaustive data movement which presents a key problem in
enhancing the performance and energy-efficiency of machine-learning hardware.
In this work, we propose a bio-plausible alternative to backpropagation drawing
from advances in feedback alignment algorithms in which the error computation
at a single synapse reduces to the product of three scalar values. Using a
sparse feedback matrix, we show that a neuron needs only a fraction of the
information previously used by the feedback alignment algorithms. Consequently,
memory and compute can be partitioned and distributed whichever way produces
the most efficient forward pass so long as a single error can be delivered to
each neuron. Our results show orders of magnitude improvement in data movement
and improvement in multiply-and-accumulate operations over
backpropagation. Like previous work, we observe that any variant of feedback
alignment suffers significant losses in classification accuracy on deep
convolutional neural networks. By transferring trained convolutional layers and
training the fully connected layers using direct feedback alignment, we
demonstrate that direct feedback alignment can obtain results competitive with
backpropagation. Furthermore, we observe that using an extremely sparse
feedback matrix, rather than a dense one, results in a small accuracy drop
while yielding hardware advantages. All the code and results are available
under https://github.com/bcrafton/ssdfa.Comment: 15 pages, 8 figure
Convolutional Sparse Kernel Network for Unsupervised Medical Image Analysis
The availability of large-scale annotated image datasets and recent advances
in supervised deep learning methods enable the end-to-end derivation of
representative image features that can impact a variety of image analysis
problems. Such supervised approaches, however, are difficult to implement in
the medical domain where large volumes of labelled data are difficult to obtain
due to the complexity of manual annotation and inter- and intra-observer
variability in label assignment. We propose a new convolutional sparse kernel
network (CSKN), which is a hierarchical unsupervised feature learning framework
that addresses the challenge of learning representative visual features in
medical image analysis domains where there is a lack of annotated training
data. Our framework has three contributions: (i) We extend kernel learning to
identify and represent invariant features across image sub-patches in an
unsupervised manner. (ii) We initialise our kernel learning with a layer-wise
pre-training scheme that leverages the sparsity inherent in medical images to
extract initial discriminative features. (iii) We adapt a multi-scale spatial
pyramid pooling (SPP) framework to capture subtle geometric differences between
learned visual features. We evaluated our framework in medical image retrieval
and classification on three public datasets. Our results show that our CSKN had
better accuracy when compared to other conventional unsupervised methods and
comparable accuracy to methods that used state-of-the-art supervised
convolutional neural networks (CNNs). Our findings indicate that our
unsupervised CSKN provides an opportunity to leverage unannotated big data in
medical imaging repositories.Comment: Accepted by Medical Image Analysis (with a new title 'Convolutional
Sparse Kernel Network for Unsupervised Medical Image Analysis'). The
manuscript is available from following link
(https://doi.org/10.1016/j.media.2019.06.005
- …