3 research outputs found
Studying the Plasticity in Deep Convolutional Neural Networks using Random Pruning
Recently there has been a lot of work on pruning filters from deep
convolutional neural networks (CNNs) with the intention of reducing
computations.The key idea is to rank the filters based on a certain criterion
(say, l1-norm) and retain only the top ranked filters. Once the low scoring
filters are pruned away the remainder of the network is fine tuned and is shown
to give performance comparable to the original unpruned network. In this work,
we report experiments which suggest that the comparable performance of the
pruned network is not due to the specific criterion chosen but due to the
inherent plasticity of deep neural networks which allows them to recover from
the loss of pruned filters once the rest of the filters are fine-tuned.
Specifically we show counter-intuitive results wherein by randomly pruning
25-50% filters from deep CNNs we are able to obtain the same performance as
obtained by using state-of-the-art pruning methods. We empirically validate our
claims by doing an exhaustive evaluation with VGG-16 and ResNet-50. We also
evaluate a real world scenario where a CNN trained on all 1000 ImageNet classes
needs to be tested on only a small set of classes at test time (say, only
animals). We create a new benchmark dataset from ImageNet to evaluate such
class specific pruning and show that even here a random pruning strategy gives
close to state-of-the-art performance. Unlike existing approaches which mainly
focus on the task of image classification, in this work we also report results
on object detection and image segmentation. We show that using a simple random
pruning strategy we can achieve significant speed up in object detection (74%
improvement in fps) while retaining the same accuracy as that of the original
Faster RCNN model. Similarly we show that the performance of a pruned
Segmentation Network (SegNet) is actually very similar to that of the original
unpruned SegNet.Comment: To appear in the Journal of Machine Vision and Applications,
Springer. This work is an extended version of our previous work
arXiv:1801.10447, "Recovering from Random Pruning: On the Plasticity of Deep
Convolutional Neural Networks", accepted at WACV 201
Shapley Value as Principled Metric for Structured Network Pruning
Structured pruning is a well-known technique to reduce the storage size and
inference cost of neural networks. The usual pruning pipeline consists of
ranking the network internal filters and activations with respect to their
contributions to the network performance, removing the units with the lowest
contribution, and fine-tuning the network to reduce the harm induced by
pruning. Recent results showed that random pruning performs on par with other
metrics, given enough fine-tuning resources. In this work, we show that this is
not true on a low-data regime when fine-tuning is either not possible or not
effective. In this case, reducing the harm caused by pruning becomes crucial to
retain the performance of the network. First, we analyze the problem of
estimating the contribution of hidden units with tools suggested by cooperative
game theory and propose Shapley values as a principled ranking metric for this
task. We compare with several alternatives proposed in the literature and
discuss how Shapley values are theoretically preferable. Finally, we compare
all ranking metrics on the challenging scenario of low-data pruning, where we
demonstrate how Shapley values outperform other heuristics
Pruning Deep Convolutional Neural Networks Architectures with Evolution Strategy
Currently, Deep Convolutional Neural Networks (DCNNs) are used to solve all
kinds of problems in the field of machine learning and artificial intelligence
due to their learning and adaptation capabilities. However, most successful
DCNN models have a high computational complexity making them difficult to
deploy on mobile or embedded platforms. This problem has prompted many
researchers to develop algorithms and approaches to help reduce the
computational complexity of such models. One of them is called filter pruning,
where convolution filters are eliminated to reduce the number of parameters
and, consequently, the computational complexity of the given model. In the
present work, we propose a novel algorithm to perform filter pruning by using
Multi-Objective Evolution Strategy (ES) algorithm, called DeepPruningES. Our
approach avoids the need for using any knowledge during the pruning procedure
and helps decision-makers by returning three pruned CNN models with different
trade-offs between performance and computational complexity. We show that
DeepPruningES can significantly reduce a model's computational complexity by
testing it on three DCNN architectures: Convolutional Neural Networks (CNNs),
Residual Neural Networks (ResNets), and Densely Connected Neural Networks
(DenseNets).Comment: Accepted at Information Science