311 research outputs found
CIFAR-10: KNN-based Ensemble of Classifiers
In this paper, we study the performance of different classifiers on the
CIFAR-10 dataset, and build an ensemble of classifiers to reach a better
performance. We show that, on CIFAR-10, K-Nearest Neighbors (KNN) and
Convolutional Neural Network (CNN), on some classes, are mutually exclusive,
thus yield in higher accuracy when combined. We reduce KNN overfitting using
Principal Component Analysis (PCA), and ensemble it with a CNN to increase its
accuracy. Our approach improves our best CNN model from 93.33% to 94.03%
Learning Combinations of Activation Functions
In the last decade, an active area of research has been devoted to design
novel activation functions that are able to help deep neural networks to
converge, obtaining better performance. The training procedure of these
architectures usually involves optimization of the weights of their layers
only, while non-linearities are generally pre-specified and their (possible)
parameters are usually considered as hyper-parameters to be tuned manually. In
this paper, we introduce two approaches to automatically learn different
combinations of base activation functions (such as the identity function, ReLU,
and tanh) during the training phase. We present a thorough comparison of our
novel approaches with well-known architectures (such as LeNet-5, AlexNet, and
ResNet-56) on three standard datasets (Fashion-MNIST, CIFAR-10, and
ILSVRC-2012), showing substantial improvements in the overall performance, such
as an increase in the top-1 accuracy for AlexNet on ILSVRC-2012 of 3.01
percentage points.Comment: 6 pages, 3 figures. Published as a conference paper at ICPR 2018.
Code:
https://bitbucket.org/francux/learning_combinations_of_activation_function
Light Multi-segment Activation for Model Compression
Model compression has become necessary when applying neural networks (NN)
into many real application tasks that can accept slightly-reduced model
accuracy with strict tolerance to model complexity. Recently, Knowledge
Distillation, which distills the knowledge from well-trained and highly complex
teacher model into a compact student model, has been widely used for model
compression. However, under the strict requirement on the resource cost, it is
quite challenging to achieve comparable performance with the teacher model,
essentially due to the drastically-reduced expressiveness ability of the
compact student model. Inspired by the nature of the expressiveness ability in
Neural Networks, we propose to use multi-segment activation, which can
significantly improve the expressiveness ability with very little cost, in the
compact student model. Specifically, we propose a highly efficient
multi-segment activation, called Light Multi-segment Activation (LMA), which
can rapidly produce multiple linear regions with very few parameters by
leveraging the statistical information. With using LMA, the compact student
model is capable of achieving much better performance effectively and
efficiently, than the ReLU-equipped one with same model scale. Furthermore, the
proposed method is compatible with other model compression techniques, such as
quantization, which means they can be used jointly for better compression
performance. Experiments on state-of-the-art NN architectures over the
real-world tasks demonstrate the effectiveness and extensibility of the LMA
- …