28 research outputs found
Bayesian Compression for Deep Learning
Compression and computational efficiency in deep learning have become a
problem of great significance. In this work, we argue that the most principled
and effective way to attack this problem is by adopting a Bayesian point of
view, where through sparsity inducing priors we prune large parts of the
network. We introduce two novelties in this paper: 1) we use hierarchical
priors to prune nodes instead of individual weights, and 2) we use the
posterior uncertainties to determine the optimal fixed point precision to
encode the weights. Both factors significantly contribute to achieving the
state of the art in terms of compression rates, while still staying competitive
with methods designed to optimize for speed or energy efficiency.Comment: Published as a conference paper at NIPS 201
Lossless compression with state space models using bits back coding
We generalize the 'bits back with ANS' method to time-series models with a
latent Markov structure. This family of models includes hidden Markov models
(HMMs), linear Gaussian state space models (LGSSMs) and many more. We provide
experimental evidence that our method is effective for small scale models, and
discuss its applicability to larger scale settings such as video compression
Compressing Sets and Multisets of Sequences
This is the accepted manuscript for a paper published in IEEE Transactions on Information Theory, Vol. 61, No. 3, March 2015, doi: 10.1109/TIT.2015.2392093. © 2015 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.This paper describes lossless compression algorithms
for multisets of sequences, taking advantage of the
multiset’s unordered structure. Multisets are a generalization of
sets, where members are allowed to occur multiple times. A multiset
can be encoded naïvely by simply storing its elements in some
sequential order, but then information is wasted on the ordering.
We propose a technique that transforms the multiset into an
order-invariant tree representation, and derive an arithmetic
code that optimally compresses the tree. Our method achieves
compression even if the sequences in the multiset are individually
incompressible (such as cryptographic hash sums). The algorithm
is demonstrated practically by compressing collections of SHA-1
hash sums, and multisets of arbitrary, individually encodable
objects.This work was supported in part by the Engineering
and Physical Sciences Research Council under Grant EP/I036575 and in
part by a Google Research Award. This paper was presented at the 2014 Data
Compression Conferenc
Universal Deep Image Compression via Content-Adaptive Optimization with Adapters
Deep image compression performs better than conventional codecs, such as
JPEG, on natural images. However, deep image compression is learning-based and
encounters a problem: the compression performance deteriorates significantly
for out-of-domain images. In this study, we highlight this problem and address
a novel task: universal deep image compression. This task aims to compress
images belonging to arbitrary domains, such as natural images, line drawings,
and comics. To address this problem, we propose a content-adaptive optimization
framework; this framework uses a pre-trained compression model and adapts the
model to a target image during compression. Adapters are inserted into the
decoder of the model. For each input image, our framework optimizes the latent
representation extracted by the encoder and the adapter parameters in terms of
rate-distortion. The adapter parameters are additionally transmitted per image.
For the experiments, a benchmark dataset containing uncompressed images of four
domains (natural images, line drawings, comics, and vector arts) is constructed
and the proposed universal deep compression is evaluated. Finally, the proposed
model is compared with non-adaptive and existing adaptive compression models.
The comparison reveals that the proposed model outperforms these. The code and
dataset are publicly available at https://github.com/kktsubota/universal-dic.Comment: Accepted at the IEEE/CVF Winter Conference on Applications of
Computer Vision (WACV) 202
Practical lossless compression with latent variables using bits back coding
Deep latent variable models have seen recent success in many data domains. Lossless compression is an application of these models which, despite having the potential to be highly useful, has yet to be implemented in a practical manner. We present 'Bits Back with ANS' (BB-ANS), a scheme to perform lossless compression with latent variable models at a near optimal rate. We demonstrate this scheme by using it to compress the MNIST dataset with a variational auto-encoder model (VAE), achieving compression rates superior to standard methods with only a simple VAE. Given that the scheme is highly amenable to parallelization, we conclude that with a sufficiently high quality generative model this scheme could be used to achieve substantial improvements in compression rate with acceptable running time. We make our implementation available open source at https://github.com/bits-back/bits-back
Practical Lossless Compression with Latent Variables using Bits Back Coding
Deep latent variable models have seen recent success in many data domains.
Lossless compression is an application of these models which, despite having
the potential to be highly useful, has yet to be implemented in a practical
manner. We present `Bits Back with ANS' (BB-ANS), a scheme to perform lossless
compression with latent variable models at a near optimal rate. We demonstrate
this scheme by using it to compress the MNIST dataset with a variational
auto-encoder model (VAE), achieving compression rates superior to standard
methods with only a simple VAE. Given that the scheme is highly amenable to
parallelization, we conclude that with a sufficiently high quality generative
model this scheme could be used to achieve substantial improvements in
compression rate with acceptable running time. We make our implementation
available open source at https://github.com/bits-back/bits-back
Generalization Gap in Amortized Inference
The ability of likelihood-based probabilistic models to generalize to unseen data is central to many machine learning applications such as lossless compression. In this work, we study the generalization of a popular class of probabilistic model - the Variational Auto-Encoder (VAE). We discuss the two generalization gaps that affect VAEs and show that overfitting is usually dominated by amortized inference. Based on this observation, we propose a new training objective that improves the generalization of amortized inference. We demonstrate how our method can improve performance in the context of image modeling and lossless compression