2,290 research outputs found
Nonlinear Hebbian learning as a unifying principle in receptive field formation
The development of sensory receptive fields has been modeled in the past by a
variety of models including normative models such as sparse coding or
independent component analysis and bottom-up models such as spike-timing
dependent plasticity or the Bienenstock-Cooper-Munro model of synaptic
plasticity. Here we show that the above variety of approaches can all be
unified into a single common principle, namely Nonlinear Hebbian Learning. When
Nonlinear Hebbian Learning is applied to natural images, receptive field shapes
were strongly constrained by the input statistics and preprocessing, but
exhibited only modest variation across different choices of nonlinearities in
neuron models or synaptic plasticity rules. Neither overcompleteness nor sparse
network activity are necessary for the development of localized receptive
fields. The analysis of alternative sensory modalities such as auditory models
or V2 development lead to the same conclusions. In all examples, receptive
fields can be predicted a priori by reformulating an abstract model as
nonlinear Hebbian learning. Thus nonlinear Hebbian learning and natural
statistics can account for many aspects of receptive field formation across
models and sensory modalities
Comparison between Oja's and BCM neural networks models in finding useful projections in high-dimensional spaces
This thesis presents the concept of a neural network starting from its corresponding biological model, paying particular attention to the learning algorithms proposed by Oja and Bienenstock Cooper & Munro. A brief introduction to Data Analysis is then performed, with particular reference to the Principal Components Analysis and Singular Value Decomposition.
The two previously introduced algorithms are then dealt with more thoroughly, going to study in particular their connections with data analysis. Finally, it is proposed to use the Singular Value Decomposition as a method for obtaining stationary points in the BCM algorithm, in the case of linearly dependent inputs
Generating functionals for computational intelligence: the Fisher information as an objective function for self-limiting Hebbian learning rules
Generating functionals may guide the evolution of a dynamical system and
constitute a possible route for handling the complexity of neural networks as
relevant for computational intelligence. We propose and explore a new objective
function, which allows to obtain plasticity rules for the afferent synaptic
weights. The adaption rules are Hebbian, self-limiting, and result from the
minimization of the Fisher information with respect to the synaptic flux. We
perform a series of simulations examining the behavior of the new learning
rules in various circumstances. The vector of synaptic weights aligns with the
principal direction of input activities, whenever one is present. A linear
discrimination is performed when there are two or more principal directions;
directions having bimodal firing-rate distributions, being characterized by a
negative excess kurtosis, are preferred. We find robust performance and full
homeostatic adaption of the synaptic weights results as a by-product of the
synaptic flux minimization. This self-limiting behavior allows for stable
online learning for arbitrary durations. The neuron acquires new information
when the statistics of input activities is changed at a certain point of the
simulation, showing however, a distinct resilience to unlearn previously
acquired knowledge. Learning is fast when starting with randomly drawn synaptic
weights and substantially slower when the synaptic weights are already fully
adapted
An Adaptive Locally Connected Neuron Model: Focusing Neuron
This paper presents a new artificial neuron model capable of learning its
receptive field in the topological domain of inputs. The model provides
adaptive and differentiable local connectivity (plasticity) applicable to any
domain. It requires no other tool than the backpropagation algorithm to learn
its parameters which control the receptive field locations and apertures. This
research explores whether this ability makes the neuron focus on informative
inputs and yields any advantage over fully connected neurons. The experiments
include tests of focusing neuron networks of one or two hidden layers on
synthetic and well-known image recognition data sets. The results demonstrated
that the focusing neurons can move their receptive fields towards more
informative inputs. In the simple two-hidden layer networks, the focusing
layers outperformed the dense layers in the classification of the 2D spatial
data sets. Moreover, the focusing networks performed better than the dense
networks even when 70 of the weights were pruned. The tests on
convolutional networks revealed that using focusing layers instead of dense
layers for the classification of convolutional features may work better in some
data sets.Comment: 45 pages, a national patent filed, submitted to Turkish Patent
Office, No: -2017/17601, Date: 09.11.201
Stochastic trapping in a solvable model of on-line independent component analysis
Previous analytical studies of on-line Independent Component Analysis (ICA)
learning rules have focussed on asymptotic stability and efficiency. In
practice the transient stages of learning will often be more significant in
determining the success of an algorithm. This is demonstrated here with an
analysis of a Hebbian ICA algorithm which can find a small number of
non-Gaussian components given data composed of a linear mixture of independent
source signals. An idealised data model is considered in which the sources
comprise a number of non-Gaussian and Gaussian sources and a solution to the
dynamics is obtained in the limit where the number of Gaussian sources is
infinite. Previous stability results are confirmed by expanding around optimal
fixed points, where a closed form solution to the learning dynamics is
obtained. However, stochastic effects are shown to stabilise otherwise unstable
sub-optimal fixed points. Conditions required to destabilise one such fixed
point are obtained for the case of a single non-Gaussian component, indicating
that the initial learning rate \eta required to successfully escape is very low
(\eta = O(N^{-2}) where N is the data dimension) resulting in very slow
learning typically requiring O(N^3) iterations. Simulations confirm that this
picture holds for a finite system.Comment: 17 pages, 3 figures. To appear in Neural Computatio
Quantum Hopfield neural network
Quantum computing allows for the potential of significant advancements in
both the speed and the capacity of widely used machine learning techniques.
Here we employ quantum algorithms for the Hopfield network, which can be used
for pattern recognition, reconstruction, and optimization as a realization of a
content-addressable memory system. We show that an exponentially large network
can be stored in a polynomial number of quantum bits by encoding the network
into the amplitudes of quantum states. By introducing a classical technique for
operating the Hopfield network, we can leverage quantum algorithms to obtain a
quantum computational complexity that is logarithmic in the dimension of the
data. We also present an application of our method as a genetic sequence
recognizer.Comment: 13 pages, 3 figures, final versio
- …