816 research outputs found
Online representation learning with single and multi-layer Hebbian networks for image classification
Unsupervised learning permits the development of algorithms that are able to adapt to a variety of different datasets using the same underlying rules thanks to the autonomous discovery of discriminating features during training. Recently, a new class of Hebbian-like and local unsupervised learning rules for neural networks have been developed that minimise a similarity matching costfunction. These have been shown to perform sparse representation learning. This
study tests the effectiveness of one such learning rule for learning features from
images. The rule implemented is derived from a nonnegative classical multidimensional
scaling cost-function, and is applied to both single and multi-layer architectures. The features learned by the algorithm are then used as input to an SVM to test their effectiveness in classification on the established CIFAR-10 image dataset. The algorithm performs well in comparison to other unsupervised learning algorithms and multi-layer networks, thus suggesting its validity in the design of a new class of compact, online learning networks
On the Origin of Deep Learning
This paper is a review of the evolutionary history of deep learning models.
It covers from the genesis of neural networks when associationism modeling of
the brain is studied, to the models that dominate the last decade of research
in deep learning like convolutional neural networks, deep belief networks, and
recurrent neural networks. In addition to a review of these models, this paper
primarily focuses on the precedents of the models above, examining how the
initial ideas are assembled to construct the early models and how these
preliminary models are developed into their current forms. Many of these
evolutionary paths last more than half a century and have a diversity of
directions. For example, CNN is built on prior knowledge of biological vision
system; DBN is evolved from a trade-off of modeling power and computation
complexity of graphical models and many nowadays models are neural counterparts
of ancient linear models. This paper reviews these evolutionary paths and
offers a concise thought flow of how these models are developed, and aims to
provide a thorough background for deep learning. More importantly, along with
the path, this paper summarizes the gist behind these milestones and proposes
many directions to guide the future research of deep learning.Comment: 70 pages, 200 reference
Locally Connected Spiking Neural Networks for Unsupervised Feature Learning
In recent years, Spiking Neural Networks (SNNs) have demonstrated great
successes in completing various Machine Learning tasks. We introduce a method
for learning image features by \textit{locally connected layers} in SNNs using
spike-timing-dependent plasticity (STDP) rule. In our approach, sub-networks
compete via competitive inhibitory interactions to learn features from
different locations of the input space. These \textit{Locally-Connected SNNs}
(LC-SNNs) manifest key topological features of the spatial interaction of
biological neurons. We explore biologically inspired n-gram classification
approach allowing parallel processing over various patches of the the image
space. We report the classification accuracy of simple two-layer LC-SNNs on two
image datasets, which match the state-of-art performance and are the first
results to date. LC-SNNs have the advantage of fast convergence to a dataset
representation, and they require fewer learnable parameters than other SNN
approaches with unsupervised learning. Robustness tests demonstrate that
LC-SNNs exhibit graceful degradation of performance despite the random deletion
of large amounts of synapses and neurons.Comment: 22 pages, 7 figures, and 4 table
Memory Aware Synapses: Learning what (not) to forget
Humans can learn in a continuous manner. Old rarely utilized knowledge can be
overwritten by new incoming information while important, frequently used
knowledge is prevented from being erased. In artificial learning systems,
lifelong learning so far has focused mainly on accumulating knowledge over
tasks and overcoming catastrophic forgetting. In this paper, we argue that,
given the limited model capacity and the unlimited new information to be
learned, knowledge has to be preserved or erased selectively. Inspired by
neuroplasticity, we propose a novel approach for lifelong learning, coined
Memory Aware Synapses (MAS). It computes the importance of the parameters of a
neural network in an unsupervised and online manner. Given a new sample which
is fed to the network, MAS accumulates an importance measure for each parameter
of the network, based on how sensitive the predicted output function is to a
change in this parameter. When learning a new task, changes to important
parameters can then be penalized, effectively preventing important knowledge
related to previous tasks from being overwritten. Further, we show an
interesting connection between a local version of our method and Hebb's
rule,which is a model for the learning process in the brain. We test our method
on a sequence of object recognition tasks and on the challenging problem of
learning an embedding for predicting triplets.
We show state-of-the-art performance and, for the first time, the ability to
adapt the importance of the parameters based on unlabeled data towards what the
network needs (not) to forget, which may vary depending on test conditions.Comment: ECCV 201
Training Convolutional Neural Networks With Hebbian Principal Component Analysis
Recent work has shown that biologically plausible Hebbian learning can be
integrated with backpropagation learning (backprop), when training deep
convolutional neural networks. In particular, it has been shown that Hebbian
learning can be used for training the lower or the higher layers of a neural
network. For instance, Hebbian learning is effective for re-training the higher
layers of a pre-trained deep neural network, achieving comparable accuracy
w.r.t. SGD, while requiring fewer training epochs, suggesting potential
applications for transfer learning. In this paper we build on these results and
we further improve Hebbian learning in these settings, by using a nonlinear
Hebbian Principal Component Analysis (HPCA) learning rule, in place of the
Hebbian Winner Takes All (HWTA) strategy used in previous work. We test this
approach in the context of computer vision. In particular, the HPCA rule is
used to train Convolutional Neural Networks in order to extract relevant
features from the CIFAR-10 image dataset. The HPCA variant that we explore
further improves the previous results, motivating further interest towards
biologically plausible learning algorithms.Comment: 12 pages, 3 figures, 2 table
Lifelong Neural Predictive Coding: Learning Cumulatively Online without Forgetting
In lifelong learning systems, especially those based on artificial neural
networks, one of the biggest obstacles is the severe inability to retain old
knowledge as new information is encountered. This phenomenon is known as
catastrophic forgetting. In this article, we propose a new kind of
connectionist architecture, the Sequential Neural Coding Network, that is
robust to forgetting when learning from streams of data points and, unlike
networks of today, does not learn via the immensely popular back-propagation of
errors. Grounded in the neurocognitive theory of predictive processing, our
model adapts its synapses in a biologically-plausible fashion, while another,
complementary neural system rapidly learns to direct and control this
cortex-like structure by mimicking the task-executive control functionality of
the basal ganglia. In our experiments, we demonstrate that our self-organizing
system experiences significantly less forgetting as compared to standard neural
models and outperforms a wide swath of previously proposed methods even though
it is trained across task datasets in a stream-like fashion. The promising
performance of our complementary system on benchmarks, e.g., SplitMNIST, Split
Fashion MNIST, and Split NotMNIST, offers evidence that by incorporating
mechanisms prominent in real neuronal systems, such as competition, sparse
activation patterns, and iterative input processing, a new possibility for
tackling the grand challenge of lifelong machine learning opens up.Comment: Key updates including results on standard benchmarks, e.g., split
mnist/fmnist/not-mnist. Task selection/basal ganglia model has been
integrate
An investigation of the cortical learning algorithm
Pattern recognition and machine learning fields have revolutionized countless industries and applications from biometric security to modern industrial assembly lines. The fields continue to accelerate as faster, more efficient processing hardware becomes commercially available. Despite the accelerated growth of the pattern recognition and machine learning fields, computers still are unable to learn, reason, and perform rudimentary tasks that humans and animals find routine. Animals are able to move fluidly, understand their environment, and maximize their chances of survival through adaptation - animals demonstrate intelligence. A primary argument in this thesis that we have not yet achieved a level of intelligence similar to humans and animals in the pattern recognition and machine learning fields, not due to a lack of computational power but, rather, due to lack of understanding of how the cortical structures of mammalian brain interact and operate.
This thesis describes a cortical learning algorithm (CLA) that models how the cortical structures in the mammalian neocortex operate. Furthermore, a high level understanding of how the cortical structures in the mammalian brain interact, store semantic patterns, and auto-recall these patterns for future predictions are discussed. Finally, we demonstrate that the algorithm can build and maintain a model of its environment and provide feedback for actions and/or classification in a similar fashion to our understanding of cortical operation
Neuroscience-inspired online unsupervised learning algorithms
Although the currently popular deep learning networks achieve unprecedented
performance on some tasks, the human brain still has a monopoly on general
intelligence. Motivated by this and biological implausibility of deep learning
networks, we developed a family of biologically plausible artificial neural
networks (NNs) for unsupervised learning. Our approach is based on optimizing
principled objective functions containing a term that matches the pairwise
similarity of outputs to the similarity of inputs, hence the name -
similarity-based. Gradient-based online optimization of such similarity-based
objective functions can be implemented by NNs with biologically plausible local
learning rules. Similarity-based cost functions and associated NNs solve
unsupervised learning tasks such as linear dimensionality reduction, sparse
and/or nonnegative feature extraction, blind nonnegative source separation,
clustering and manifold learning.Comment: Accepted for publication in IEEE Signal Processing Magazin
Biologically Inspired Feedforward Supervised Learning for Deep Self-Organizing Map Networks
In this study, we propose a novel deep neural network and its supervised
learning method that uses a feedforward supervisory signal. The method is
inspired by the human visual system and performs human-like association-based
learning without any backward error propagation. The feedforward supervisory
signal that produces the correct result is preceded by the target signal and
associates its confirmed label with the classification result of the target
signal. It effectively uses a large amount of information from the feedforward
signal, and forms a continuous and rich learning representation. The method is
validated using visual recognition tasks on the MNIST handwritten dataset.Comment: Presented at MLINI-2016 workshop, 2016 (arXiv:1701.01437
Structured and Deep Similarity Matching via Structured and Deep Hebbian Networks
Synaptic plasticity is widely accepted to be the mechanism behind learning in
the brain's neural networks. A central question is how synapses, with access to
only local information about the network, can still organize collectively and
perform circuit-wide learning in an efficient manner. In single-layered and
all-to-all connected neural networks, local plasticity has been shown to
implement gradient-based learning on a class of cost functions that contain a
term that aligns the similarity of outputs to the similarity of inputs. Whether
such cost functions exist for networks with other architectures is not known.
In this paper, we introduce structured and deep similarity matching cost
functions, and show how they can be optimized in a gradient-based manner by
neural networks with local learning rules. These networks extend F\"oldiak's
Hebbian/Anti-Hebbian network to deep architectures and structured feedforward,
lateral and feedback connections. Credit assignment problem is solved elegantly
by a factorization of the dual learning objective to synapse specific local
objectives. Simulations show that our networks learn meaningful features.Comment: Accepted for publication in NeurIPS 2019; Minor typos fixe
- …