36,283 research outputs found
Critical Data Compression
A new approach to data compression is developed and applied to multimedia
content. This method separates messages into components suitable for both
lossless coding and 'lossy' or statistical coding techniques, compressing
complex objects by separately encoding signals and noise. This is demonstrated
by compressing the most significant bits of data exactly, since they are
typically redundant and compressible, and either fitting a maximally likely
noise function to the residual bits or compressing them using lossy methods.
Upon decompression, the significant bits are decoded and added to a noise
function, whether sampled from a noise model or decompressed from a lossy code.
This results in compressed data similar to the original. For many test images,
a two-part image code using JPEG2000 for lossy coding and PAQ8l for lossless
coding produces less mean-squared error than an equal length of JPEG2000.
Computer-generated images typically compress better using this method than
through direct lossy coding, as do many black and white photographs and most
color photographs at sufficiently high quality levels. Examples applying the
method to audio and video coding are also demonstrated. Since two-part codes
are efficient for both periodic and chaotic data, concatenations of roughly
similar objects may be encoded efficiently, which leads to improved inference.
Applications to artificial intelligence are demonstrated, showing that signals
using an economical lossless code have a critical level of redundancy which
leads to better description-based inference than signals which encode either
insufficient data or too much detail.Comment: 99 pages, 31 figure
Compression of Deep Neural Networks on the Fly
Thanks to their state-of-the-art performance, deep neural networks are
increasingly used for object recognition. To achieve these results, they use
millions of parameters to be trained. However, when targeting embedded
applications the size of these models becomes problematic. As a consequence,
their usage on smartphones or other resource limited devices is prohibited. In
this paper we introduce a novel compression method for deep neural networks
that is performed during the learning phase. It consists in adding an extra
regularization term to the cost function of fully-connected layers. We combine
this method with Product Quantization (PQ) of the trained weights for higher
savings in storage consumption. We evaluate our method on two data sets (MNIST
and CIFAR10), on which we achieve significantly larger compression rates than
state-of-the-art methods
A Method for Compressing Parameters in Bayesian Models with Application to Logistic Sequence Prediction Models
Bayesian classification and regression with high order interactions is
largely infeasible because Markov chain Monte Carlo (MCMC) would need to be
applied with a great many parameters, whose number increases rapidly with the
order. In this paper we show how to make it feasible by effectively reducing
the number of parameters, exploiting the fact that many interactions have the
same values for all training cases. Our method uses a single ``compressed''
parameter to represent the sum of all parameters associated with a set of
patterns that have the same value for all training cases. Using symmetric
stable distributions as the priors of the original parameters, we can easily
find the priors of these compressed parameters. We therefore need to deal only
with a much smaller number of compressed parameters when training the model
with MCMC. The number of compressed parameters may have converged before
considering the highest possible order. After training the model, we can split
these compressed parameters into the original ones as needed to make
predictions for test cases. We show in detail how to compress parameters for
logistic sequence prediction models. Experiments on both simulated and real
data demonstrate that a huge number of parameters can indeed be reduced by our
compression method.Comment: 29 page
Human Motion Capture Data Tailored Transform Coding
Human motion capture (mocap) is a widely used technique for digitalizing
human movements. With growing usage, compressing mocap data has received
increasing attention, since compact data size enables efficient storage and
transmission. Our analysis shows that mocap data have some unique
characteristics that distinguish themselves from images and videos. Therefore,
directly borrowing image or video compression techniques, such as discrete
cosine transform, does not work well. In this paper, we propose a novel
mocap-tailored transform coding algorithm that takes advantage of these
features. Our algorithm segments the input mocap sequences into clips, which
are represented in 2D matrices. Then it computes a set of data-dependent
orthogonal bases to transform the matrices to frequency domain, in which the
transform coefficients have significantly less dependency. Finally, the
compression is obtained by entropy coding of the quantized coefficients and the
bases. Our method has low computational cost and can be easily extended to
compress mocap databases. It also requires neither training nor complicated
parameter setting. Experimental results demonstrate that the proposed scheme
significantly outperforms state-of-the-art algorithms in terms of compression
performance and speed
Domain-adaptive deep network compression
Deep Neural Networks trained on large datasets can be easily transferred to
new domains with far fewer labeled examples by a process called fine-tuning.
This has the advantage that representations learned in the large source domain
can be exploited on smaller target domains. However, networks designed to be
optimal for the source task are often prohibitively large for the target task.
In this work we address the compression of networks after domain transfer.
We focus on compression algorithms based on low-rank matrix decomposition.
Existing methods base compression solely on learned network weights and ignore
the statistics of network activations. We show that domain transfer leads to
large shifts in network activations and that it is desirable to take this into
account when compressing. We demonstrate that considering activation statistics
when compressing weights leads to a rank-constrained regression problem with a
closed-form solution. Because our method takes into account the target domain,
it can more optimally remove the redundancy in the weights. Experiments show
that our Domain Adaptive Low Rank (DALR) method significantly outperforms
existing low-rank compression techniques. With our approach, the fc6 layer of
VGG19 can be compressed more than 4x more than using truncated SVD alone --
with only a minor or no loss in accuracy. When applied to domain-transferred
networks it allows for compression down to only 5-20% of the original number of
parameters with only a minor drop in performance.Comment: Accepted at ICCV 201
Restricted Recurrent Neural Networks
Recurrent Neural Network (RNN) and its variations such as Long Short-Term
Memory (LSTM) and Gated Recurrent Unit (GRU), have become standard building
blocks for learning online data of sequential nature in many research areas,
including natural language processing and speech data analysis. In this paper,
we present a new methodology to significantly reduce the number of parameters
in RNNs while maintaining performance that is comparable or even better than
classical RNNs. The new proposal, referred to as Restricted Recurrent Neural
Network (RRNN), restricts the weight matrices corresponding to the input data
and hidden states at each time step to share a large proportion of parameters.
The new architecture can be regarded as a compression of its classical
counterpart, but it does not require pre-training or sophisticated parameter
fine-tuning, both of which are major issues in most existing compression
techniques. Experiments on natural language modeling show that compared with
its classical counterpart, the restricted recurrent architecture generally
produces comparable results at about 50\% compression rate. In particular, the
Restricted LSTM can outperform classical RNN with even less number of
parameters
- …