2 research outputs found
Target Aware Network Adaptation for Efficient Representation Learning
This paper presents an automatic network adaptation method that finds a
ConvNet structure well-suited to a given target task, e.g., image
classification, for efficiency as well as accuracy in transfer learning. We
call the concept target-aware transfer learning. Given only small-scale labeled
data, and starting from an ImageNet pre-trained network, we exploit a scheme of
removing its potential redundancy for the target task through iterative
operations of filter-wise pruning and network optimization. The basic
motivation is that compact networks are on one hand more efficient and should
also be more tolerant, being less complex, against the risk of overfitting
which would hinder the generalization of learned representations in the context
of transfer learning. Further, unlike existing methods involving network
simplification, we also let the scheme identify redundant portions across the
entire network, which automatically results in a network structure adapted to
the task at hand. We achieve this with a few novel ideas: (i) cumulative sum of
activation statistics for each layer, and (ii) a priority evaluation of pruning
across multiple layers. Experimental results by the method on five datasets
(Flower102, CUB200-2011, Dog120, MIT67, and Stanford40) show favorable
accuracies over the related state-of-the-art techniques while enhancing the
computational and storage efficiency of the transferred model.Comment: Accepted by the ECCV'18 Workshops (2nd International Workshop on
Compact and Efficient Feature Representation and Learning in Computer Vision
Towards Efficient Model Compression via Learned Global Ranking
Pruning convolutional filters has demonstrated its effectiveness in
compressing ConvNets. Prior art in filter pruning requires users to specify a
target model complexity (e.g., model size or FLOP count) for the resulting
architecture. However, determining a target model complexity can be difficult
for optimizing various embodied AI applications such as autonomous robots,
drones, and user-facing applications. First, both the accuracy and the speed of
ConvNets can affect the performance of the application. Second, the performance
of the application can be hard to assess without evaluating ConvNets during
inference. As a consequence, finding a sweet-spot between the accuracy and
speed via filter pruning, which needs to be done in a trial-and-error fashion,
can be time-consuming. This work takes a first step toward making this process
more efficient by altering the goal of model compression to producing a set of
ConvNets with various accuracy and latency trade-offs instead of producing one
ConvNet targeting some pre-defined latency constraint. To this end, we propose
to learn a global ranking of the filters across different layers of the
ConvNet, which is used to obtain a set of ConvNet architectures that have
different accuracy/latency trade-offs by pruning the bottom-ranked filters. Our
proposed algorithm, LeGR, is shown to be 2x to 3x faster than prior work while
having comparable or better performance when targeting seven pruned ResNet-56
with different accuracy/FLOPs profiles on the CIFAR-100 dataset. Additionally,
we have evaluated LeGR on ImageNet and Bird-200 with ResNet-50 and MobileNetV2
to demonstrate its effectiveness. Code available at
https://github.com/cmu-enyac/LeGR.Comment: CVPR 2020 Ora