164 research outputs found
Chaos-guided Input Structuring for Improved Learning in Recurrent Neural Networks
Anatomical studies demonstrate that brain reformats input information to
generate reliable responses for performing computations. However, it remains
unclear how neural circuits encode complex spatio-temporal patterns. We show
that neural dynamics are strongly influenced by the phase alignment between the
input and the spontaneous chaotic activity. Input structuring along the
dominant chaotic projections causes the chaotic trajectories to become stable
channels (or attractors), hence, improving the computational capability of a
recurrent network. Using mean field analysis, we derive the impact of input
structuring on the overall stability of attractors formed. Our results indicate
that input alignment determines the extent of intrinsic noise suppression and
hence, alters the attractor state stability, thereby controlling the network's
inference ability.Comment: 11 pages with 5 figures including supplementary materia
STDP Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy Efficient Recognition
Spiking Neural Networks (SNNs) with a large number of weights and varied
weight distribution can be difficult to implement in emerging in-memory
computing hardware due to the limitations on crossbar size (implementing dot
product), the constrained number of conductance levels in non-CMOS devices and
the power budget. We present a sparse SNN topology where non-critical
connections are pruned to reduce the network size and the remaining critical
synapses are weight quantized to accommodate for limited conductance levels.
Pruning is based on the power law weight-dependent Spike Timing Dependent
Plasticity (STDP) model; synapses between pre- and post-neuron with high spike
correlation are retained, whereas synapses with low correlation or uncorrelated
spiking activity are pruned. The weights of the retained connections are
quantized to the available number of conductance levels. The process of pruning
non-critical connections and quantizing the weights of critical synapses is
performed at regular intervals during training. We evaluated our sparse and
quantized network on MNIST dataset and on a subset of images from Caltech-101
dataset. The compressed topology achieved a classification accuracy of 90.1%
(91.6%) on the MNIST (Caltech-101) dataset with 3.1x (2.2x) and 4x (2.6x)
improvement in energy and area, respectively. The compressed topology is energy
and area efficient while maintaining the same classification accuracy of a
2-layer fully connected SNN topology.Comment: 9 pages, 8 figure
Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognition
Deep learning neural networks have emerged as one of the most powerful
classification tools for vision related applications. However, the
computational and energy requirements associated with such deep nets can be
quite high, and hence their energy-efficient implementation is of great
interest. Although traditionally the entire network is utilized for the
recognition of all inputs, we observe that the classification difficulty varies
widely across inputs in real-world datasets; only a small fraction of inputs
require the full computational effort of a network, while a large majority can
be classified correctly with very low effort. In this paper, we propose
Conditional Deep Learning (CDL) where the convolutional layer features are used
to identify the variability in the difficulty of input instances and
conditionally activate the deeper layers of the network. We achieve this by
cascading a linear network of output neurons for each convolutional layer and
monitoring the output of the linear network to decide whether classification
can be terminated at the current stage or not. The proposed methodology thus
enables the network to dynamically adjust the computational effort depending
upon the difficulty of the input data while maintaining competitive
classification accuracy. We evaluate our approach on the MNIST dataset. Our
experiments demonstrate that our proposed CDL yields 1.91x reduction in average
number of operations per input, which translates to 1.84x improvement in
energy. In addition, our results show an improvement in classification accuracy
from 97.5% to 98.9% as compared to the original network.Comment: 6 pages, 10 figures, 2 algorithms < Accepted for Design and
Automation Test in Europe (DATE) conference, 2016
A Low Effort Approach to Structured CNN Design Using PCA
Deep learning models hold state of the art performance in many fields, yet
their design is still based on heuristics or grid search methods that often
result in overparametrized networks. This work proposes a method to analyze a
trained network and deduce an optimized, compressed architecture that preserves
accuracy while keeping computational costs tractable. Model compression is an
active field of research that targets the problem of realizing deep learning
models in hardware. However, most pruning methodologies tend to be
experimental, requiring large compute and time intensive iterations of
retraining the entire network. We introduce structure into model design by
proposing a single shot analysis of a trained network that serves as a first
order, low effort approach to dimensionality reduction, by using PCA (Principal
Component Analysis). The proposed method simultaneously analyzes the
activations of each layer and considers the dimensionality of the space
described by the filters generating these activations. It optimizes the
architecture in terms of number of layers, and number of filters per layer
without any iterative retraining procedures, making it a viable, low effort
technique to design efficient networks. We demonstrate the proposed methodology
on AlexNet and VGG style networks on the CIFAR-10, CIFAR-100 and ImageNet
datasets, and successfully achieve an optimized architecture with a reduction
of up to 3.8X and 9X in the number of operations and parameters respectively,
while trading off less than 1% accuracy. We also apply the method to MobileNet,
and achieve 1.7X and 3.9X reduction in the number of operations and parameters
respectively, while improving accuracy by almost one percentage point.Comment: To be Published in IEEE Access, Volume 8, 202
Voltage-Driven Domain-Wall Motion based Neuro-Synaptic Devices for Dynamic On-line Learning
Conventional von-Neumann computing models have achieved remarkable feats for
the past few decades. However, they fail to deliver the required efficiency for
certain basic tasks like image and speech recognition when compared to
biological systems. As such, taking cues from biological systems, novel
computing paradigms are being explored for efficient hardware implementations
of recognition/classification tasks. The basic building blocks of such
neuromorphic systems are neurons and synapses. Towards that end, we propose a
leaky-integrate-fire (LIF) neuron and a programmable non-volatile synapse using
domain wall motion induced by magneto-electric effect. Due to a strong elastic
pinning between the ferro-magnetic domain wall (FM-DW) and the underlying
ferro-electric domain wall (FE-DW), the FM-DW gets dragged by the FE-DW on
application of a voltage pulse. The fact that FE materials are insulators
allows for pure voltage-driven FM-DW motion, which in turn can be used to mimic
the behaviors of biological spiking neurons and synapses. The voltage driven
nature of the proposed devices allows energy-efficient operation. A detailed
device to system level simulation framework based on micromagnetic simulations
has been developed to analyze the feasibility of the proposed neuro-synaptic
devices. We also demonstrate that the energy-efficient voltage-controlled
behavior of the proposed devices make them suitable for dynamic on-line and
lifelong learning in spiking neural networks (SNNs)
ASP: Learning to Forget with Adaptive Synaptic Plasticity in Spiking Neural Networks
A fundamental feature of learning in animals is the "ability to forget" that
allows an organism to perceive, model and make decisions from disparate streams
of information and adapt to changing environments. Against this backdrop, we
present a novel unsupervised learning mechanism ASP (Adaptive Synaptic
Plasticity) for improved recognition with Spiking Neural Networks (SNNs) for
real time on-line learning in a dynamic environment. We incorporate an adaptive
weight decay mechanism with the traditional Spike Timing Dependent Plasticity
(STDP) learning to model adaptivity in SNNs. The leak rate of the synaptic
weights is modulated based on the temporal correlation between the spiking
patterns of the pre- and post-synaptic neurons. This mechanism helps in gradual
forgetting of insignificant data while retaining significant, yet old,
information. ASP, thus, maintains a balance between forgetting and immediate
learning to construct a stable-plastic self-adaptive SNN for continuously
changing inputs. We demonstrate that the proposed learning methodology
addresses catastrophic forgetting while yielding significantly improved
accuracy over the conventional STDP learning method for digit recognition
applications. Additionally, we observe that the proposed learning model
automatically encodes selective attention towards relevant features in the
input data while eliminating the influence of background noise (or denoising)
further improving the robustness of the ASP learning.Comment: 14 pages, 14 figure
Implicit Generative Modeling of Random Noise during Training for Adversarial Robustness
We introduce a Noise-based prior Learning (NoL) approach for training neural
networks that are intrinsically robust to adversarial attacks. We find that the
implicit generative modeling of random noise with the same loss function used
during posterior maximization, improves a model's understanding of the data
manifold furthering adversarial robustness. We evaluate our approach's efficacy
and provide a simplistic visualization tool for understanding adversarial data,
using Principal Component Analysis. Our analysis reveals that adversarial
robustness, in general, manifests in models with higher variance along the
high-ranked principal components. We show that models learnt with our approach
perform remarkably well against a wide-range of attacks. Furthermore, combining
NoL with state-of-the-art adversarial training extends the robustness of a
model, even beyond what it is adversarially trained for, in both white-box and
black-box attack scenarios.Comment: Preliminary version of this work accepted at ICML 2019 (Workshop on
Uncertainty and Robustness in Deep Learning
Magnetic Tunnel Junction Mimics Stochastic Cortical Spiking Neurons
Brain-inspired computing architectures attempt to mimic the computations
performed in the neurons and the synapses in the human brain in order to
achieve its efficiency in learning and cognitive tasks. In this work, we
demonstrate the mapping of the probabilistic spiking nature of pyramidal
neurons in the cortex to the stochastic switching behavior of a Magnetic Tunnel
Junction in presence of thermal noise. We present results to illustrate the
efficiency of neuromorphic systems based on such probabilistic neurons for
pattern recognition tasks in presence of lateral inhibition and homeostasis.
Such stochastic MTJ neurons can also potentially provide a direct mapping to
the probabilistic computing elements in Belief Networks for performing
regenerative tasks.Comment: The article will appear in Scientific Report
Energy-Efficient Object Detection using Semantic Decomposition
Machine-learning algorithms offer immense possibilities in the development of
several cognitive applications. In fact, large scale machine-learning
classifiers now represent the state-of-the-art in a wide range of object
detection/classification problems. However, the network complexities of
large-scale classifiers present them as one of the most challenging and energy
intensive workloads across the computing spectrum. In this paper, we present a
new approach to optimize energy efficiency of object detection tasks using
semantic decomposition to build a hierarchical classification framework. We
observe that certain semantic information like color/texture are common across
various images in real-world datasets for object detection applications. We
exploit these common semantic features to distinguish the objects of interest
from the remaining inputs (non-objects of interest) in a dataset at a lower
computational effort. We propose a 2-stage hierarchical classification
framework, with increasing levels of complexity, wherein the first stage is
trained to recognize the broad representative semantic features relevant to the
object of interest. The first stage rejects the input instances that do not
have the representative features and passes only the relevant instances to the
second stage. Our methodology thus allows us to reject certain information at
lower complexity and utilize the full computational effort of a network only on
a smaller fraction of inputs to perform detection. We use color and texture as
distinctive traits to carry out several experiments for object detection. Our
experiments on the Caltech101/CIFAR10 dataset show that the proposed method
yields 1.93x/1.46x improvement in average energy, respectively, over the
traditional single classifier model.Comment: 10 pages, 13 figures, 3 algorithms, Submitted to IEEE TVLSI(Under
Review
Synthesizing Images from Spatio-Temporal Representations using Spike-based Backpropagation
Spiking neural networks (SNNs) offer a promising alternative to current
artificial neural networks to enable low-power event-driven neuromorphic
hardware. Spike-based neuromorphic applications require processing and
extracting meaningful information from spatio-temporal data, represented as
series of spike trains over time. In this paper, we propose a method to
synthesize images from multiple modalities in a spike-based environment. We use
spiking auto-encoders to convert image and audio inputs into compact
spatio-temporal representations that is then decoded for image synthesis. For
this, we use a direct training algorithm that computes loss on the membrane
potential of the output layer and back-propagates it by using a sigmoid
approximation of the neuron's activation function to enable differentiability.
The spiking autoencoders are benchmarked on MNIST and Fashion-MNIST and achieve
very low reconstruction loss, comparable to ANNs. Then, spiking autoencoders
are trained to learn meaningful spatio-temporal representations of the data,
across the two modalities - audio and visual. We synthesize images from audio
in a spike-based environment by first generating, and then utilizing such
shared multi-modal spatio-temporal representations. Our audio to image
synthesis model is tested on the task of converting TI-46 digits audio samples
to MNIST images. We are able to synthesize images with high fidelity and the
model achieves competitive performance against ANNs.Comment: 17 pages, 10 Figures, 1 tabl
- …