61,637 research outputs found
CSNNs: Unsupervised, Backpropagation-free Convolutional Neural Networks for Representation Learning
This work combines Convolutional Neural Networks (CNNs), clustering via
Self-Organizing Maps (SOMs) and Hebbian Learning to propose the building blocks
of Convolutional Self-Organizing Neural Networks (CSNNs), which learn
representations in an unsupervised and Backpropagation-free manner. Our
approach replaces the learning of traditional convolutional layers from CNNs
with the competitive learning procedure of SOMs and simultaneously learns local
masks between those layers with separate Hebbian-like learning rules to
overcome the problem of disentangling factors of variation when filters are
learned through clustering. We investigate the learned representation by
designing two simple models with our building blocks, achieving comparable
performance to many methods which use Backpropagation, while we reach
comparable performance on Cifar10 and give baseline performances on Cifar100,
Tiny ImageNet and a small subset of ImageNet for Backpropagation-free methods.Comment: 18 pages,18 figures, Author's extended version of the paper. Final
version presented at 18th IEEE International Conference on Machine Learning
and Applications (ICMLA). Boca Raton, Florida / USA. 201
A critical analysis of self-supervision, or what we can learn from a single image
We look critically at popular self-supervision techniques for learning deep
convolutional neural networks without manual labels. We show that three
different and representative methods, BiGAN, RotNet and DeepCluster, can learn
the first few layers of a convolutional network from a single image as well as
using millions of images and manual labels, provided that strong data
augmentation is used. However, for deeper layers the gap with manual
supervision cannot be closed even if millions of unlabelled images are used for
training. We conclude that: (1) the weights of the early layers of deep
networks contain limited information about the statistics of natural images,
that (2) such low-level statistics can be learned through self-supervision just
as well as through strong supervision, and that (3) the low-level statistics
can be captured via synthetic transformations instead of using a large image
dataset.Comment: Accepted paper at the International Conference on Learning
Representations (ICLR) 202
- …