51,138 research outputs found
Neural computation at the thermal limit
Although several measurements and analyses support the idea that the brain is
energy-optimized, there is one disturbing, contradictory observation: In
theory, computation limited by thermal noise can occur as cheaply as ~ joules per bit (kTln2). Unfortunately, for a neuron the ostensible
discrepancy from this minimum is startling - ignoring inhibition the
discrepancy is times this amount and taking inhibition into account
. Here we point out that what has been defined as neural computation is
actually a combination of computation and neural communication: the
communication costs, transmission from each excitatory postsynaptic activation
to the S4-gating-charges of the fast Na+ channels of the initial segment
(fNa's), dominate the joule-costs. Making this distinction between
communication to the initial segment and computation at the initial segment
(i.e., adding up of the activated fNa's) implies that the size of the average
synaptic event reaching the fNa's is the size of the standard deviation of the
thermal noise. Moreover, defining computation as the addition of activated
fNa's, yields a biophysically plausible mechanism for approaching the desired
minimum. This mechanism, requiring something like the electrical engineer's
equalizer (not much more than the action potential generating conductances),
only operates at threshold. This active filter modifies the last few synaptic
excitations, providing barely enough energy to allow the last sub-threshold
gating charge to transport. That is, the last, threshold-achieving S4-subunit
activation requires an energy that matches the information being provided by
the last few synaptic events, a ratio that is near kTln2 joules per bit.Comment: 2 figure
Algorithm and Hardware Design of Discrete-Time Spiking Neural Networks Based on Back Propagation with Binary Activations
We present a new back propagation based training algorithm for discrete-time
spiking neural networks (SNN). Inspired by recent deep learning algorithms on
binarized neural networks, binary activation with a straight-through gradient
estimator is used to model the leaky integrate-fire spiking neuron, overcoming
the difficulty in training SNNs using back propagation. Two SNN training
algorithms are proposed: (1) SNN with discontinuous integration, which is
suitable for rate-coded input spikes, and (2) SNN with continuous integration,
which is more general and can handle input spikes with temporal information.
Neuromorphic hardware designed in 40nm CMOS exploits the spike sparsity and
demonstrates high classification accuracy (>98% on MNIST) and low energy
(48.4-773 nJ/image).Comment: 2017 IEEE Biomedical Circuits and Systems (BioCAS
AX-DBN: An Approximate Computing Framework for the Design of Low-Power Discriminative Deep Belief Networks
The power budget for embedded hardware implementations of Deep Learning
algorithms can be extremely tight. To address implementation challenges in such
domains, new design paradigms, like Approximate Computing, have drawn
significant attention. Approximate Computing exploits the innate
error-resilience of Deep Learning algorithms, a property that makes them
amenable for deployment on low-power computing platforms. This paper describes
an Approximate Computing design methodology, AX-DBN, for an architecture
belonging to the class of stochastic Deep Learning algorithms known as Deep
Belief Networks (DBNs). Specifically, we consider procedures for efficiently
implementing the Discriminative Deep Belief Network (DDBN), a stochastic neural
network which is used for classification tasks, extending Approximation
Computing from the analysis of deterministic to stochastic neural networks. For
the purpose of optimizing the DDBN for hardware implementations, we explore the
use of: (a)Limited precision of neurons and functional approximations of
activation functions; (b) Criticality analysis to identify nodes in the network
which can operate at reduced precision while allowing the network to maintain
target accuracy levels; and (c) A greedy search methodology with incremental
retraining to determine the optimal reduction in precision for all neurons to
maximize power savings. Using the AX-DBN methodology proposed in this paper, we
present experimental results across several network architectures that show
significant power savings under a user-specified accuracy loss constraint with
respect to ideal full precision implementations
Deep Sparse Coding for Invariant Multimodal Halle Berry Neurons
Deep feed-forward convolutional neural networks (CNNs) have become ubiquitous
in virtually all machine learning and computer vision challenges; however,
advancements in CNNs have arguably reached an engineering saturation point
where incremental novelty results in minor performance gains. Although there is
evidence that object classification has reached human levels on narrowly
defined tasks, for general applications, the biological visual system is far
superior to that of any computer. Research reveals there are numerous missing
components in feed-forward deep neural networks that are critical in mammalian
vision. The brain does not work solely in a feed-forward fashion, but rather
all of the neurons are in competition with each other; neurons are integrating
information in a bottom up and top down fashion and incorporating expectation
and feedback in the modeling process. Furthermore, our visual cortex is working
in tandem with our parietal lobe, integrating sensory information from various
modalities.
In our work, we sought to improve upon the standard feed-forward deep
learning model by augmenting them with biologically inspired concepts of
sparsity, top-down feedback, and lateral inhibition. We define our model as a
sparse coding problem using hierarchical layers. We solve the sparse coding
problem with an additional top-down feedback error driving the dynamics of the
neural network. While building and observing the behavior of our model, we were
fascinated that multimodal, invariant neurons naturally emerged that mimicked,
"Halle Berry neurons" found in the human brain. Furthermore, our sparse
representation of multimodal signals demonstrates qualitative and quantitative
superiority to the standard feed-forward joint embedding in common vision and
machine learning tasks
New acceleration technique for the backpropagation algorithm
Artificial neural networks have been studied for many years in the hope of achieving human like performance in the area of pattern recognition, speech synthesis and higher level of cognitive process. In the connectionist model there are several interconnected processing elements called the neurons that have limited processing capability. Even though the rate of information transmitted between these elements is limited, the complex interconnection and the cooperative interaction between these elements results in a vastly increased computing power; The neural network models are specified by an organized network topology of interconnected neurons. These networks have to be trained in order them to be used for a specific purpose. Backpropagation is one of the popular methods of training the neural networks. There has been a lot of improvement over the speed of convergence of standard backpropagation algorithm in the recent past. Herein we have presented a new technique for accelerating the existing backpropagation without modifying it. We have used the fourth order interpolation method for the dominant eigen values, by using these we change the slope of the activation function. And by doing so we increase the speed of convergence of the backpropagation algorithm; Our experiments have shown significant improvement in the convergence time for problems widely used in benchmarKing Three to ten fold decrease in convergence time is achieved. Convergence time decreases as the complexity of the problem increases. The technique adjusts the energy state of the system so as to escape from local minima
Autonomous learning and chaining of motor primitives using the Free Energy Principle
In this article, we apply the Free-Energy Principle to the question of motor
primitives learning. An echo-state network is used to generate motor
trajectories. We combine this network with a perception module and a controller
that can influence its dynamics. This new compound network permits the
autonomous learning of a repertoire of motor trajectories. To evaluate the
repertoires built with our method, we exploit them in a handwriting task where
primitives are chained to produce long-range sequences
A Reconfigurable Low Power High Throughput Architecture for Deep Network Training
General purpose computing systems are used for a large variety of
applications. Extensive supports for flexibility in these systems limit their
energy efficiencies. Neural networks, including deep networks, are widely used
for signal processing and pattern recognition applications. In this paper we
propose a multicore architecture for deep neural network based processing.
Memristor crossbars are utilized to provide low power high throughput execution
of neural networks. The system has both training and recognition (evaluation of
new input) capabilities. The proposed system could be used for classification,
dimensionality reduction, feature extraction, and anomaly detection
applications. The system level area and power benefits of the specialized
architecture is compared with the NVIDIA Telsa K20 GPGPU. Our experimental
evaluations show that the proposed architecture can provide up to five orders
of magnitude more energy efficiency over GPGPUs for deep neural network
processing.Comment: 9 page
Distilling Spikes: Knowledge Distillation in Spiking Neural Networks
Spiking Neural Networks (SNN) are energy-efficient computing architectures
that exchange spikes for processing information, unlike classical Artificial
Neural Networks (ANN). Due to this, SNNs are better suited for real-life
deployments. However, similar to ANNs, SNNs also benefit from deeper
architectures to obtain improved performance. Furthermore, like the deep ANNs,
the memory, compute and power requirements of SNNs also increase with model
size, and model compression becomes a necessity. Knowledge distillation is a
model compression technique that enables transferring the learning of a large
machine learning model to a smaller model with minimal loss in performance. In
this paper, we propose techniques for knowledge distillation in spiking neural
networks for the task of image classification. We present ways to distill
spikes from a larger SNN, also called the teacher network, to a smaller one,
also called the student network, while minimally impacting the classification
accuracy. We demonstrate the effectiveness of the proposed method with detailed
experiments on three standard datasets while proposing novel distillation
methodologies and loss functions. We also present a multi-stage knowledge
distillation technique for SNNs using an intermediate network to obtain higher
performance from the student network. Our approach is expected to open up new
avenues for deploying high performing large SNN models on resource-constrained
hardware platforms.Comment: Preprint: Manuscript under revie
Efficient Computation in Adaptive Artificial Spiking Neural Networks
Artificial Neural Networks (ANNs) are bio-inspired models of neural
computation that have proven highly effective. Still, ANNs lack a natural
notion of time, and neural units in ANNs exchange analog values in a
frame-based manner, a computationally and energetically inefficient form of
communication. This contrasts sharply with biological neurons that communicate
sparingly and efficiently using binary spikes. While artificial Spiking Neural
Networks (SNNs) can be constructed by replacing the units of an ANN with
spiking neurons, the current performance is far from that of deep ANNs on hard
benchmarks and these SNNs use much higher firing rates compared to their
biological counterparts, limiting their efficiency. Here we show how spiking
neurons that employ an efficient form of neural coding can be used to construct
SNNs that match high-performance ANNs and exceed state-of-the-art in SNNs on
important benchmarks, while requiring much lower average firing rates. For
this, we use spike-time coding based on the firing rate limiting adaptation
phenomenon observed in biological spiking neurons. This phenomenon can be
captured in adapting spiking neuron models, for which we derive the effective
transfer function. Neural units in ANNs trained with this transfer function can
be substituted directly with adaptive spiking neurons, and the resulting
Adaptive SNNs (AdSNNs) can carry out inference in deep neural networks using up
to an order of magnitude fewer spikes compared to previous SNNs. Adaptive
spike-time coding additionally allows for the dynamic control of neural coding
precision: we show how a simple model of arousal in AdSNNs further halves the
average required firing rate and this notion naturally extends to other forms
of attention. AdSNNs thus hold promise as a novel and efficient model for
neural computation that naturally fits to temporally continuous and
asynchronous applications
Automation of Processor Verification Using Recurrent Neural Networks
When considering simulation-based verification of processors, the current
trend is to generate stimuli using pseudorandom generators (PRGs), apply them
to the processor inputs and monitor the achieved coverage of its functionality
in order to determine verification completeness. Stimuli can have different
forms, for example, they can be represented by bit vectors applied to the input
ports of the processor or by programs that are loaded directly into the program
memory. In this paper, we propose a new technique dynamically altering
constraints for PRG via recurrent neural network, which receives a coverage
feedback from the simulation of design under verification. For the
demonstration purposes we used processors provided by Codasip as their coverage
state space is reasonably big and differs for various kinds of processors.
Nevertheless, techniques presented in this paper are widely applicable. The
results of experiments show that not only the coverage closure is achieved much
sooner, but we are able to isolate a small set of stimuli with high coverage
that can be used for running regression tests.Comment: Paper contains 6 pages, 6 figures. Presented on MTVCon 2017. Soon to
be released by IEE
- …