Advances in Compression using Probabilistic Models

The increasing demand for data transmission and storage necessitate the use of efficient compression methods. Compression algorithms work by mapping data to a more compact representation from which the original data can be recovered. To operate efficiently, they need to capture the characteristics of the data distribution, which can be difficult, especially for high-dimensional data. One emerging solution lies in applying probabilistic machine learning to capture the data distribution in an unsupervised manner. Once a probabilistic model for the data is defined, variational inference can be used to infer its parameters from data. Variational inference is closely related to the optimal compression size, as stated by Hinton's bits-back argument: the evidence lower bound, the objective optimized by variational inference, corresponds to a lower bound on the optimal compression size of the average datapoint. However, current compression methods rely on variational inference merely as a heuristic, and they do not approach its postulated efficiency. In this thesis, we present principled and practical algorithms that get closer to this limit. After discussing our approach, we demonstrate its efficacy in image compression and model compression. First, we focus on image compression, where we use a variational autoencoder to learn a mapping between the images and their unobserved, latent representations. We propose a stochastic coding scheme to encode the latent representation, from which the original image can be approximately reconstructed. Next, we look at the compression of deep learning models. We use variational inference to approximate the posterior distribution of the weights in a neural network, and apply our stochastic coding scheme to encode a weight configuration. Finally, we investigate a connection between variational inference and our compression algorithm. We show that a technique we used for compression can improve variational inference by generating samples from a highly flexible posterior approximation, without significantly increasing the computational costs

Nonresponse in the 1996 income survey (supplement to the microcensus)

Einkommensstudie: ergänzender freiwilliger Fragebogen für ein Viertel des Mikrozensus vom April 1996, 18117 Haushalte, 16% Verweigerer (2988). Die eine Beteiligung verweigernden Haushalte sind nicht zufallsverteilt. Vielmehr weisen einige im Rahmen des Mikrozensus erhobenen Merkmale eine Verbindung mit Antwortverweigerung auf. Vor allem das Qualifikationsniveau des Haushaltsvorstands und regionale Faktoren sind für das Antwortverhalten aussagekräftig. Ältere Haushalte auf dem Land mit niedrigem Einkommen und niedrigem Qualifikationsniveau weisen eine höhere Beteiligung an der Befragung auf, während in Budapest, vor allem in den Gruppen mit hohem Einkommen und hoher Qualifikation, die Quote der Antwortverweigerer hoch ist. (ICEÜbers)"Income survey: supplementary voluntary questionnaire to the randomly selected one quarter of the April 1996 microcensus, 18.117 households, 16% (2.988) refusals. The characteristics of the households which refused to answer the income survey (they are not randomly distributed) were carefully studied The most important results and measures which were done to reduce the bias are presented. A substantial number of census variables for households were found to be associated with nonresponse. The characteristics most strongly associated with Income Survey response rate were the qualification level of head of household and the type of region. Higher response rate was found in the countryside among the older households with low income and low qualification and high refusal rate in Budapest, mainly in high income groups with high qualification level, as well." (author's abstract

Guarantee Regions for Local Explanations

Interpretability methods that utilise local surrogate models (e.g. LIME) are very good at describing the behaviour of the predictive model at a point of interest, but they are not guaranteed to extrapolate to the local region surrounding the point. However, overfitting to the local curvature of the predictive model and malicious tampering can significantly limit extrapolation. We propose an anchor-based algorithm for identifying regions in which local explanations are guaranteed to be correct by explicitly describing those intervals along which the input features can be trusted. Our method produces an interpretable feature-aligned box where the prediction of the local surrogate model is guaranteed to match the predictive model. We demonstrate that our algorithm can be used to find explanations with larger guarantee regions that better cover the data manifold compared to existing baselines. We also show how our method can identify misleading local explanations with significantly poorer guarantee regions

Sampling the Variational Posterior with Local Refinement.

Variational inference is an optimization-based method for approximating the posterior distribution of the parameters in Bayesian probabilistic models. A key challenge of variational inference is to approximate the posterior with a distribution that is computationally tractable yet sufficiently expressive. We propose a novel method for generating samples from a highly flexible variational approximation. The method starts with a coarse initial approximation and generates samples by refining it in selected, local regions. This allows the samples to capture dependencies and multi-modality in the posterior, even when these are absent from the initial approximation. We demonstrate theoretically that our method always improves the quality of the approximation (as measured by the evidence lower bound). In experiments, our method consistently outperforms recent variational inference methods in terms of log-likelihood and ELBO across three example tasks: the Eight-Schools example (an inference task in a hierarchical model), training a ResNet-20 (Bayesian inference in a large neural network), and the Mushroom task (posterior sampling in a contextual bandit problem)

Sampling the Variational Posterior with Local Refinement

Variational inference is an optimization-based method for approximating the posterior distribution of the parameters in Bayesian probabilistic models. A key challenge of variational inference is to approximate the posterior with a distribution that is computationally tractable yet sufficiently expressive. We propose a novel method for generating samples from a highly flexible variational approximation. The method starts with a coarse initial approximation and generates samples by refining it in selected, local regions. This allows the samples to capture dependencies and multi-modality in the posterior, even when these are absent from the initial approximation. We demonstrate theoretically that our method always improves the quality of the approximation (as measured by the evidence lower bound). In experiments, our method consistently outperforms recent variational inference methods in terms of log-likelihood and ELBO across three example tasks: the Eight-Schools example (an inference task in a hierarchical model), training a ResNet-20 (Bayesian inference in a large neural network), and the Mushroom task (posterior sampling in a contextual bandit problem)