4 research outputs found
A Layer Decomposition-Recomposition Framework for Neuron Pruning towards Accurate Lightweight Networks
Neuron pruning is an efficient method to compress the network into a slimmer
one for reducing the computational cost and storage overhead. Most of
state-of-the-art results are obtained in a layer-by-layer optimization mode. It
discards the unimportant input neurons and uses the survived ones to
reconstruct the output neurons approaching to the original ones in a
layer-by-layer manner. However, an unnoticed problem arises that the
information loss is accumulated as layer increases since the survived neurons
still do not encode the entire information as before. A better alternative is
to propagate the entire useful information to reconstruct the pruned layer
instead of directly discarding the less important neurons. To this end, we
propose a novel Layer Decomposition-Recomposition Framework (LDRF) for neuron
pruning, by which each layer's output information is recovered in an embedding
space and then propagated to reconstruct the following pruned layers with
useful information preserved. We mainly conduct our experiments on ILSVRC-12
benchmark with VGG-16 and ResNet-50. What should be emphasized is that our
results before end-to-end fine-tuning are significantly superior owing to the
information-preserving property of our proposed framework.With end-to-end
fine-tuning, we achieve state-of-the-art results of 5.13x and 3x speed-up with
only 0.5% and 0.65% top-5 accuracy drop respectively, which outperform the
existing neuron pruning methods.Comment: accepted by AAAI19 as ora
Attention Diversification for Domain Generalization
Convolutional neural networks (CNNs) have demonstrated gratifying results at
learning discriminative features. However, when applied to unseen domains,
state-of-the-art models are usually prone to errors due to domain shift. After
investigating this issue from the perspective of shortcut learning, we find the
devils lie in the fact that models trained on different domains merely bias to
different domain-specific features yet overlook diverse task-related features.
Under this guidance, a novel Attention Diversification framework is proposed,
in which Intra-Model and Inter-Model Attention Diversification Regularization
are collaborated to reassign appropriate attention to diverse task-related
features. Briefly, Intra-Model Attention Diversification Regularization is
equipped on the high-level feature maps to achieve in-channel discrimination
and cross-channel diversification via forcing different channels to pay their
most salient attention to different spatial locations. Besides, Inter-Model
Attention Diversification Regularization is proposed to further provide
task-related attention diversification and domain-related attention
suppression, which is a paradigm of "simulate, divide and assemble": simulate
domain shift via exploiting multiple domain-specific models, divide attention
maps into task-related and domain-related groups, and assemble them within each
group respectively to execute regularization. Extensive experiments and
analyses are conducted on various benchmarks to demonstrate that our method
achieves state-of-the-art performance over other competing methods. Code is
available at https://github.com/hikvision-research/DomainGeneralization.Comment: ECCV 2022. Code available at
https://github.com/hikvision-research/DomainGeneralizatio