3 research outputs found
Denoising autoencoder with modulated lateral connections learns invariant representations of natural images
Suitable lateral connections between encoder and decoder are shown to allow
higher layers of a denoising autoencoder (dAE) to focus on invariant
representations. In regular autoencoders, detailed information needs to be
carried through the highest layers but lateral connections from encoder to
decoder relieve this pressure. It is shown that abstract invariant features can
be translated to detailed reconstructions when invariant features are allowed
to modulate the strength of the lateral connection. Three dAE structures with
modulated and additive lateral connections, and without lateral connections
were compared in experiments using real-world images. The experiments verify
that adding modulated lateral connections to the model 1) improves the accuracy
of the probability model for inputs, as measured by denoising performance; 2)
results in representations whose degree of invariance grows faster towards the
higher layers; and 3) supports the formation of diverse invariant poolings.Comment: Presentation at ICLR 2015 worksho
Theta-RBM: Unfactored Gated Restricted Boltzmann Machine for Rotation-Invariant Representations
Learning invariant representations is a critical task in computer vision. In
this paper, we propose the Theta-Restricted Boltzmann Machine ({\theta}-RBM in
short), which builds upon the original RBM formulation and injects the notion
of rotation-invariance during the learning procedure. In contrast to previous
approaches, we do not transform the training set with all possible rotations.
Instead, we rotate the gradient filters when they are computed during the
Contrastive Divergence algorithm. We formulate our model as an unfactored gated
Boltzmann machine, where another input layer is used to modulate the input
visible layer to drive the optimisation procedure. Among our contributions is a
mathematical proof that demonstrates that {\theta}-RBM is able to learn
rotation-invariant features according to a recently proposed invariance
measure. Our method reaches an invariance score of ~90% on mnist-rot dataset,
which is the highest result compared with the baseline methods and the current
state of the art in transformation-invariant feature learning in RBM. Using an
SVM classifier, we also showed that our network learns discriminative features
as well, obtaining ~10% of testing error.Comment: 9 pages, 2 figures, 3 table
Semi-Supervised Learning with Ladder Networks
We combine supervised learning with unsupervised learning in deep neural
networks. The proposed model is trained to simultaneously minimize the sum of
supervised and unsupervised cost functions by backpropagation, avoiding the
need for layer-wise pre-training. Our work builds on the Ladder network
proposed by Valpola (2015), which we extend by combining the model with
supervision. We show that the resulting model reaches state-of-the-art
performance in semi-supervised MNIST and CIFAR-10 classification, in addition
to permutation-invariant MNIST classification with all labels.Comment: Revised denoising function, updated results, fixed typo