2,041 research outputs found
Learning the Pseudoinverse Solution to Network Weights
The last decade has seen the parallel emergence in computational neuroscience
and machine learning of neural network structures which spread the input signal
randomly to a higher dimensional space; perform a nonlinear activation; and
then solve for a regression or classification output by means of a mathematical
pseudoinverse operation. In the field of neuromorphic engineering, these
methods are increasingly popular for synthesizing biologically plausible neural
networks, but the "learning method" - computation of the pseudoinverse by
singular value decomposition - is problematic both for biological plausibility
and because it is not an online or an adaptive method. We present an online or
incremental method of computing the pseudoinverse, which we argue is
biologically plausible as a learning method, and which can be made adaptable
for non-stationary data streams. The method is significantly more
memory-efficient than the conventional computation of pseudoinverses by
singular value decomposition.Comment: 13 pages, 3 figures; in submission to Neural Network
Synthesis of neural networks for spatio-temporal spike pattern recognition and processing
The advent of large scale neural computational platforms has highlighted the
lack of algorithms for synthesis of neural structures to perform predefined
cognitive tasks. The Neural Engineering Framework offers one such synthesis,
but it is most effective for a spike rate representation of neural information,
and it requires a large number of neurons to implement simple functions. We
describe a neural network synthesis method that generates synaptic connectivity
for neurons which process time-encoded neural signals, and which makes very
sparse use of neurons. The method allows the user to specify, arbitrarily,
neuronal characteristics such as axonal and dendritic delays, and synaptic
transfer functions, and then solves for the optimal input-output relationship
using computed dendritic weights. The method may be used for batch or online
learning and has an extremely fast optimization process. We demonstrate its use
in generating a network to recognize speech which is sparsely encoded as spike
times.Comment: In submission to Frontiers in Neuromorphic Engineerin
OCReP: An Optimally Conditioned Regularization for Pseudoinversion Based Neural Training
In this paper we consider the training of single hidden layer neural networks
by pseudoinversion, which, in spite of its popularity, is sometimes affected by
numerical instability issues. Regularization is known to be effective in such
cases, so that we introduce, in the framework of Tikhonov regularization, a
matricial reformulation of the problem which allows us to use the condition
number as a diagnostic tool for identification of instability. By imposing
well-conditioning requirements on the relevant matrices, our theoretical
analysis allows the identification of an optimal value for the regularization
parameter from the standpoint of stability. We compare with the value derived
by cross-validation for overfitting control and optimisation of the
generalization performance. We test our method for both regression and
classification tasks. The proposed method is quite effective in terms of
predictivity, often with some improvement on performance with respect to the
reference cases considered. This approach, due to analytical determination of
the regularization parameter, dramatically reduces the computational load
required by many other techniques.Comment: Published on Neural Network
- …