Compressing Probability Distributions

We show how to store good approximations of probability distributions in small space

A Method for Compressing Parameters in Bayesian Models with Application to Logistic Sequence Prediction Models

Bayesian classification and regression with high order interactions is largely infeasible because Markov chain Monte Carlo (MCMC) would need to be applied with a great many parameters, whose number increases rapidly with the order. In this paper we show how to make it feasible by effectively reducing the number of parameters, exploiting the fact that many interactions have the same values for all training cases. Our method uses a single ``compressed'' parameter to represent the sum of all parameters associated with a set of patterns that have the same value for all training cases. Using symmetric stable distributions as the priors of the original parameters, we can easily find the priors of these compressed parameters. We therefore need to deal only with a much smaller number of compressed parameters when training the model with MCMC. The number of compressed parameters may have converged before considering the highest possible order. After training the model, we can split these compressed parameters into the original ones as needed to make predictions for test cases. We show in detail how to compress parameters for logistic sequence prediction models. Experiments on both simulated and real data demonstrate that a huge number of parameters can indeed be reduced by our compression method.Comment: 29 page

Cryptography from Information Loss

© Marshall Ball, Elette Boyle, Akshay Degwekar, Apoorvaa Deshpande, Alon Rosen, Vinod. Reductions between problems, the mainstay of theoretical computer science, efficiently map an instance of one problem to an instance of another in such a way that solving the latter allows solving the former.1 The subject of this work is “lossy” reductions, where the reduction loses some information about the input instance. We show that such reductions, when they exist, have interesting and powerful consequences for lifting hardness into “useful” hardness, namely cryptography. Our first, conceptual, contribution is a definition of lossy reductions in the language of mutual information. Roughly speaking, our definition says that a reduction C is t-lossy if, for any distribution X over its inputs, the mutual information I(X; C(X)) ≤ t. Our treatment generalizes a variety of seemingly related but distinct notions such as worst-case to average-case reductions, randomized encodings (Ishai and Kushilevitz, FOCS 2000), homomorphic computations (Gentry, STOC 2009), and instance compression (Harnik and Naor, FOCS 2006). We then proceed to show several consequences of lossy reductions: 1. We say that a language L has an f-reduction to a language L0 for a Boolean function f if there is a (randomized) polynomial-time algorithm C that takes an m-tuple of strings X = (x1, . . ., xm), with each xi ∈ {0, 1}n, and outputs a string z such that with high probability, L0(z) = f(L(x1), L(x2), . . ., L(xm)) Suppose a language L has an f-reduction C to L0 that is t-lossy. Our first result is that one-way functions exist if L is worst-case hard and one of the following conditions holds: f is the OR function, t ≤ m/100, and L0 is the same as L f is the Majority function, and t ≤ m/100 f is the OR function, t ≤ O(m log n), and the reduction has no error This improves on the implications that follow from combining (Drucker, FOCS 2012) with (Ostrovsky and Wigderson, ISTCS 1993) that result in auxiliary-input one-way functions. 2. Our second result is about the stronger notion of t-compressing f-reductions – reductions that only output t bits. We show that if there is an average-case hard language L that has a t-compressing Majority reduction to some language for t = m/100, then there exist collision-resistant hash functions. This improves on the result of (Harnik and Naor, STOC 2006), whose starting point is a cryptographic primitive (namely, one-way functions) rather than average-case hardness, and whose assumption is a compressing OR-reduction of SAT (which is now known to be false unless the polynomial hierarchy collapses). Along the way, we define a non-standard one-sided notion of average-case hardness, which is the notion of hardness used in the second result above, that may be of independent interest

The work value of information

We present quantitative relations between work and information that are valid both for finite sized and internally correlated systems as well in the thermodynamical limit. We suggest work extraction should be viewed as a game where the amount of work an agent can extract depends on how well it can guess the micro-state of the system. In general it depends both on the agent's knowledge and risk-tolerance, because the agent can bet on facts that are not certain and thereby risk failure of the work extraction. We derive strikingly simple expressions for the extractable work in the extreme cases of effectively zero- and arbitrary risk tolerance respectively, thereby enveloping all cases. Our derivation makes a connection between heat engines and the smooth entropy approach. The latter has recently extended Shannon theory to encompass finite sized and internally correlated bit strings, and our analysis points the way to an analogous extension of statistical mechanics.Comment: 5 pages, 4 figure

Intrinsic viscosity of a suspension of weakly Brownian ellipsoids in shear

We analyze the angular dynamics of triaxial ellipsoids in a shear flow subject to weak thermal noise. By numerically integrating an overdamped angular Langevin equation, we find the steady angular probability distribution for a range of triaxial particle shapes. From this distribution we compute the intrinsic viscosity of a dilute suspension of triaxial particles. We determine how the viscosity depends on particle shape in the limit of weak thermal noise. While the deterministic angular dynamics depends very sensitively on particle shape, we find that the shape dependence of the intrinsic viscosity is weaker, in general, and that suspensions of rod-like particles are the most sensitive to breaking of axisymmetry. The intrinsic viscosity of a dilute suspension of triaxial particles is smaller than that of a suspension of axisymmetric particles with the same volume, and the same ratio of major to minor axis lengths.Comment: 14 pages, 6 figures, 1 table, revised versio

Large Alphabets and Incompressibility

We briefly survey some concepts related to empirical entropy -- normal numbers, de Bruijn sequences and Markov processes -- and investigate how well it approximates Kolmogorov complexity. Our results suggest $\ell$th-order empirical entropy stops being a reasonable complexity metric for almost all strings of length $m$ over alphabets of size $n$ about when $n^\ell$ surpasses $m$