20,596 research outputs found
From neural PCA to deep unsupervised learning
A network supporting deep unsupervised learning is presented. The network is
an autoencoder with lateral shortcut connections from the encoder to decoder at
each level of the hierarchy. The lateral shortcut connections allow the higher
levels of the hierarchy to focus on abstract invariant features. While standard
autoencoders are analogous to latent variable models with a single layer of
stochastic variables, the proposed network is analogous to hierarchical latent
variables models. Learning combines denoising autoencoder and denoising sources
separation frameworks. Each layer of the network contributes to the cost
function a term which measures the distance of the representations produced by
the encoder and the decoder. Since training signals originate from all levels
of the network, all layers can learn efficiently even in deep networks. The
speedup offered by cost terms from higher levels of the hierarchy and the
ability to learn invariant features are demonstrated in experiments.Comment: A revised version of an article that has been accepted for
publication in Advances in Independent Component Analysis and Learning
Machines (2015), edited by Ella Bingham, Samuel Kaski, Jorma Laaksonen and
Jouko Lampine
Biologically plausible deep learning -- but how far can we go with shallow networks?
Training deep neural networks with the error backpropagation algorithm is
considered implausible from a biological perspective. Numerous recent
publications suggest elaborate models for biologically plausible variants of
deep learning, typically defining success as reaching around 98% test accuracy
on the MNIST data set. Here, we investigate how far we can go on digit (MNIST)
and object (CIFAR10) classification with biologically plausible, local learning
rules in a network with one hidden layer and a single readout layer. The hidden
layer weights are either fixed (random or random Gabor filters) or trained with
unsupervised methods (PCA, ICA or Sparse Coding) that can be implemented by
local learning rules. The readout layer is trained with a supervised, local
learning rule. We first implement these models with rate neurons. This
comparison reveals, first, that unsupervised learning does not lead to better
performance than fixed random projections or Gabor filters for large hidden
layers. Second, networks with localized receptive fields perform significantly
better than networks with all-to-all connectivity and can reach backpropagation
performance on MNIST. We then implement two of the networks - fixed, localized,
random & random Gabor filters in the hidden layer - with spiking leaky
integrate-and-fire neurons and spike timing dependent plasticity to train the
readout layer. These spiking models achieve > 98.2% test accuracy on MNIST,
which is close to the performance of rate networks with one hidden layer
trained with backpropagation. The performance of our shallow network models is
comparable to most current biologically plausible models of deep learning.
Furthermore, our results with a shallow spiking network provide an important
reference and suggest the use of datasets other than MNIST for testing the
performance of future models of biologically plausible deep learning.Comment: 14 pages, 4 figure
Unsupervised feature-learning for galaxy SEDs with denoising autoencoders
With the increasing number of deep multi-wavelength galaxy surveys, the
spectral energy distribution (SED) of galaxies has become an invaluable tool
for studying the formation of their structures and their evolution. In this
context, standard analysis relies on simple spectro-photometric selection
criteria based on a few SED colors. If this fully supervised classification
already yielded clear achievements, it is not optimal to extract relevant
information from the data. In this article, we propose to employ very recent
advances in machine learning, and more precisely in feature learning, to derive
a data-driven diagram. We show that the proposed approach based on denoising
autoencoders recovers the bi-modality in the galaxy population in an
unsupervised manner, without using any prior knowledge on galaxy SED
classification. This technique has been compared to principal component
analysis (PCA) and to standard color/color representations. In addition,
preliminary results illustrate that this enables the capturing of extra
physically meaningful information, such as redshift dependence, galaxy mass
evolution and variation over the specific star formation rate. PCA also results
in an unsupervised representation with physical properties, such as mass and
sSFR, although this representation separates out. less other characteristics
(bimodality, redshift evolution) than denoising autoencoders.Comment: 11 pages and 15 figures. To be published in A&
Domain Adaptive Neural Networks for Object Recognition
We propose a simple neural network model to deal with the domain adaptation
problem in object recognition. Our model incorporates the Maximum Mean
Discrepancy (MMD) measure as a regularization in the supervised learning to
reduce the distribution mismatch between the source and target domains in the
latent space. From experiments, we demonstrate that the MMD regularization is
an effective tool to provide good domain adaptation models on both SURF
features and raw image pixels of a particular image data set. We also show that
our proposed model, preceded by the denoising auto-encoder pretraining,
achieves better performance than recent benchmark models on the same data sets.
This work represents the first study of MMD measure in the context of neural
networks
- …