39,017 research outputs found
Domain-adaptive deep network compression
Deep Neural Networks trained on large datasets can be easily transferred to
new domains with far fewer labeled examples by a process called fine-tuning.
This has the advantage that representations learned in the large source domain
can be exploited on smaller target domains. However, networks designed to be
optimal for the source task are often prohibitively large for the target task.
In this work we address the compression of networks after domain transfer.
We focus on compression algorithms based on low-rank matrix decomposition.
Existing methods base compression solely on learned network weights and ignore
the statistics of network activations. We show that domain transfer leads to
large shifts in network activations and that it is desirable to take this into
account when compressing. We demonstrate that considering activation statistics
when compressing weights leads to a rank-constrained regression problem with a
closed-form solution. Because our method takes into account the target domain,
it can more optimally remove the redundancy in the weights. Experiments show
that our Domain Adaptive Low Rank (DALR) method significantly outperforms
existing low-rank compression techniques. With our approach, the fc6 layer of
VGG19 can be compressed more than 4x more than using truncated SVD alone --
with only a minor or no loss in accuracy. When applied to domain-transferred
networks it allows for compression down to only 5-20% of the original number of
parameters with only a minor drop in performance.Comment: Accepted at ICCV 201
DeepCABAC: A Universal Compression Algorithm for Deep Neural Networks
The field of video compression has developed some of the most sophisticated
and efficient compression algorithms known in the literature, enabling very
high compressibility for little loss of information. Whilst some of these
techniques are domain specific, many of their underlying principles are
universal in that they can be adapted and applied for compressing different
types of data. In this work we present DeepCABAC, a compression algorithm for
deep neural networks that is based on one of the state-of-the-art video coding
techniques. Concretely, it applies a Context-based Adaptive Binary Arithmetic
Coder (CABAC) to the network's parameters, which was originally designed for
the H.264/AVC video coding standard and became the state-of-the-art for
lossless compression. Moreover, DeepCABAC employs a novel quantization scheme
that minimizes the rate-distortion function while simultaneously taking the
impact of quantization onto the accuracy of the network into account.
Experimental results show that DeepCABAC consistently attains higher
compression rates than previously proposed coding techniques for neural network
compression. For instance, it is able to compress the VGG16 ImageNet model by
x63.6 with no loss of accuracy, thus being able to represent the entire network
with merely 8.7MB. The source code for encoding and decoding can be found at
https://github.com/fraunhoferhhi/DeepCABAC
- …