96 research outputs found
Learning the Pseudoinverse Solution to Network Weights
The last decade has seen the parallel emergence in computational neuroscience
and machine learning of neural network structures which spread the input signal
randomly to a higher dimensional space; perform a nonlinear activation; and
then solve for a regression or classification output by means of a mathematical
pseudoinverse operation. In the field of neuromorphic engineering, these
methods are increasingly popular for synthesizing biologically plausible neural
networks, but the "learning method" - computation of the pseudoinverse by
singular value decomposition - is problematic both for biological plausibility
and because it is not an online or an adaptive method. We present an online or
incremental method of computing the pseudoinverse, which we argue is
biologically plausible as a learning method, and which can be made adaptable
for non-stationary data streams. The method is significantly more
memory-efficient than the conventional computation of pseudoinverses by
singular value decomposition.Comment: 13 pages, 3 figures; in submission to Neural Network
Synthesis of neural networks for spatio-temporal spike pattern recognition and processing
The advent of large scale neural computational platforms has highlighted the
lack of algorithms for synthesis of neural structures to perform predefined
cognitive tasks. The Neural Engineering Framework offers one such synthesis,
but it is most effective for a spike rate representation of neural information,
and it requires a large number of neurons to implement simple functions. We
describe a neural network synthesis method that generates synaptic connectivity
for neurons which process time-encoded neural signals, and which makes very
sparse use of neurons. The method allows the user to specify, arbitrarily,
neuronal characteristics such as axonal and dendritic delays, and synaptic
transfer functions, and then solves for the optimal input-output relationship
using computed dendritic weights. The method may be used for batch or online
learning and has an extremely fast optimization process. We demonstrate its use
in generating a network to recognize speech which is sparsely encoded as spike
times.Comment: In submission to Frontiers in Neuromorphic Engineerin
Adaptive Online Sequential ELM for Concept Drift Tackling
A machine learning method needs to adapt to over time changes in the
environment. Such changes are known as concept drift. In this paper, we propose
concept drift tackling method as an enhancement of Online Sequential Extreme
Learning Machine (OS-ELM) and Constructive Enhancement OS-ELM (CEOS-ELM) by
adding adaptive capability for classification and regression problem. The
scheme is named as adaptive OS-ELM (AOS-ELM). It is a single classifier scheme
that works well to handle real drift, virtual drift, and hybrid drift. The
AOS-ELM also works well for sudden drift and recurrent context change type. The
scheme is a simple unified method implemented in simple lines of code. We
evaluated AOS-ELM on regression and classification problem by using concept
drift public data set (SEA and STAGGER) and other public data sets such as
MNIST, USPS, and IDS. Experiments show that our method gives higher kappa value
compared to the multiclassifier ELM ensemble. Even though AOS-ELM in practice
does not need hidden nodes increase, we address some issues related to the
increasing of the hidden nodes such as error condition and rank values. We
propose taking the rank of the pseudoinverse matrix as an indicator parameter
to detect underfitting condition.Comment: Hindawi Publishing. Computational Intelligence and Neuroscience
Volume 2016 (2016), Article ID 8091267, 17 pages Received 29 January 2016,
Accepted 17 May 2016. Special Issue on "Advances in Neural Networks and
Hybrid-Metaheuristics: Theory, Algorithms, and Novel Engineering
Applications". Academic Editor: Stefan Hauf
Hamiltonian Monte Carlo Acceleration Using Surrogate Functions with Random Bases
For big data analysis, high computational cost for Bayesian methods often
limits their applications in practice. In recent years, there have been many
attempts to improve computational efficiency of Bayesian inference. Here we
propose an efficient and scalable computational technique for a
state-of-the-art Markov Chain Monte Carlo (MCMC) methods, namely, Hamiltonian
Monte Carlo (HMC). The key idea is to explore and exploit the structure and
regularity in parameter space for the underlying probabilistic model to
construct an effective approximation of its geometric properties. To this end,
we build a surrogate function to approximate the target distribution using
properly chosen random bases and an efficient optimization process. The
resulting method provides a flexible, scalable, and efficient sampling
algorithm, which converges to the correct target distribution. We show that by
choosing the basis functions and optimization process differently, our method
can be related to other approaches for the construction of surrogate functions
such as generalized additive models or Gaussian process models. Experiments
based on simulated and real data show that our approach leads to substantially
more efficient sampling algorithms compared to existing state-of-the art
methods
ADIC: Anomaly Detection Integrated Circuit in 65nm CMOS utilizing Approximate Computing
In this paper, we present a low-power anomaly detection integrated circuit
(ADIC) based on a one-class classifier (OCC) neural network. The ADIC achieves
low-power operation through a combination of (a) careful choice of algorithm
for online learning and (b) approximate computing techniques to lower average
energy. In particular, online pseudoinverse update method (OPIUM) is used to
train a randomized neural network for quick and resource efficient learning. An
additional 42% energy saving can be achieved when a lighter version of OPIUM
method is used for training with the same number of data samples lead to no
significant compromise on the quality of inference. Instead of a single
classifier with large number of neurons, an ensemble of K base learner approach
is chosen to reduce learning memory by a factor of K. This also enables
approximate computing by dynamically varying the neural network size based on
anomaly detection. Fabricated in 65nm CMOS, the ADIC has K = 7 Base Learners
(BL) with 32 neurons in each BL and dissipates 11.87pJ/OP and 3.35pJ/OP during
learning and inference respectively at Vdd = 0.75V when all 7 BLs are enabled.
Further, evaluated on the NASA bearing dataset, approximately 80% of the chip
can be shut down for 99% of the lifetime leading to an energy efficiency of
0.48pJ/OP, an 18.5 times reduction over full-precision computing running at Vdd
= 1.2V throughout the lifetime.Comment: 1
- …