9 research outputs found

    A Divergence Formula for Randomness and Dimension

    Full text link
    If SS is an infinite sequence over a finite alphabet Σ\Sigma and β\beta is a probability measure on Σ\Sigma, then the {\it dimension} of S S with respect to β\beta, written dimβ(S)\dim^\beta(S), is a constructive version of Billingsley dimension that coincides with the (constructive Hausdorff) dimension dim(S)\dim(S) when β\beta is the uniform probability measure. This paper shows that dimβ(S)\dim^\beta(S) and its dual \Dim^\beta(S), the {\it strong dimension} of SS with respect to β\beta, can be used in conjunction with randomness to measure the similarity of two probability measures α\alpha and β\beta on Σ\Sigma. Specifically, we prove that the {\it divergence formula} \dim^\beta(R) = \Dim^\beta(R) =\frac{\CH(\alpha)}{\CH(\alpha) + \D(\alpha || \beta)} holds whenever α\alpha and β\beta are computable, positive probability measures on Σ\Sigma and RΣR \in \Sigma^\infty is random with respect to α\alpha. In this formula, \CH(\alpha) is the Shannon entropy of α\alpha, and \D(\alpha||\beta) is the Kullback-Leibler divergence between α\alpha and β\beta. We also show that the above formula holds for all sequences RR that are α\alpha-normal (in the sense of Borel) when dimβ(R)\dim^\beta(R) and \Dim^\beta(R) are replaced by the more effective finite-state dimensions \dimfs^\beta(R) and \Dimfs^\beta(R). In the course of proving this, we also prove finite-state compression characterizations of \dimfs^\beta(S) and \Dimfs^\beta(S).Comment: 18 page

    A Divergence Formula for Randomness and Dimension (Short Version)

    Full text link
    If SS is an infinite sequence over a finite alphabet Σ\Sigma and β\beta is a probability measure on Σ\Sigma, then the {\it dimension} of S S with respect to β\beta, written dimβ(S)\dim^\beta(S), is a constructive version of Billingsley dimension that coincides with the (constructive Hausdorff) dimension dim(S)\dim(S) when β\beta is the uniform probability measure. This paper shows that dimβ(S)\dim^\beta(S) and its dual \Dim^\beta(S), the {\it strong dimension} of SS with respect to β\beta, can be used in conjunction with randomness to measure the similarity of two probability measures α\alpha and β\beta on Σ\Sigma. Specifically, we prove that the {\it divergence formula} \dim^\beta(R) = \Dim^\beta(R) =\CH(\alpha) / (\CH(\alpha) + \D(\alpha || \beta)) holds whenever α\alpha and β\beta are computable, positive probability measures on Σ\Sigma and RΣR \in \Sigma^\infty is random with respect to α\alpha. In this formula, \CH(\alpha) is the Shannon entropy of α\alpha, and \D(\alpha||\beta) is the Kullback-Leibler divergence between α\alpha and β\beta

    Asymptotic divergences and strong dichotomy

    Get PDF
    The Schnorr-Stimm dichotomy theorem [31] concerns finite-state gamblers that bet on infinite sequences of symbols taken from a finite alphabet S. The theorem asserts that, for any such sequence S, the following two things are true. (1) If S is not normal in the sense of Borel (meaning that every two strings of equal length appear with equal asymptotic frequency in S), then there is a finite-state gambler that wins money at an infinitely-often exponential rate betting on S. (2) If S is normal, then any finite-state gambler betting on S loses money at an exponential rate betting on S. In this paper we use the Kullback-Leibler divergence to formulate the lower asymptotic divergence div(S||a) of a probability measure a on S from a sequence S over S and the upper asymptotic divergence Div(S||a) of a from S in such a way that a sequence S is a-normal (meaning that every string w has asymptotic frequency a(w) in S) if and only if Div(S||a) = 0. We also use the Kullback-Leibler divergence to quantify the total risk RiskG(w) that a finite-state gambler G takes when betting along a prefix w of S. Our main theorem is a strong dichotomy theorem that uses the above notions to quantify the exponential rates of winning and losing on the two sides of the Schnorr-Stimm dichotomy theorem (with the latter routinely extended from normality to a-normality). Modulo asymptotic caveats in the paper, our strong dichotomy theorem says that the following two things hold for prefixes w of S. (10) The infinitely-often exponential rate of winning in 1 is 2Div(S||a)|w| . (20) The exponential rate of loss in 2 is 2-RiskG(w) . We also use (10) to show that 1 - Div(S||a)/c, where c = log(1/mina¿S a(a)), is an upper bound on the finite-state a-dimension of S and prove the dual fact that 1 - div(S||a)/c is an upper bound on the finite-state strong a-dimension of S

    Mutual dimension, data processing inequalities, and randomness

    Get PDF
    This dissertation makes progress in the area of constructive dimension, an effectivization of classical Hausdorff dimension. Using constructive dimension, one may assign a non-zero number to the dimension of individual sequences and individual points in Euclidean space. The primary objective of this dissertation is to develop a framework for mutual dimension, i.e., the density of algorithmic mutual information between two infinite objects, that has similar properties as those of classical Shannon mutual information. Chapter 1 presents a brief history of the development of constructive dimension along with its relationships to algorithmic information theory, algorithmic randomness, and classical Hausdorff dimension. Some applications of this field are discussed and an overview of each subsequent chapter is provided. Chapter 2 defines and analyzes the mutual algorithmic information between two points x and y at a given precision r. In fact, we describe two plausible definitions for this quantity, I_r(x:y) and J_r(x:y), and show that they are closely related. In order to do this, we prove and make use of a generalization of Levin\u27s coding theorem. Chapter 3 defines the lower and upper mutual dimensions between two points in Euclidean space and presents results on its basic properties. A large portion of this chapter is dedicated to studying data processing inequalities for points in Euclidean space. Generally speaking, a data processing inequality says that the amount of information between two objects cannot be significantly increased when one of the objects is processed by a particular type of transformation. We show that it is possible to derive several kinds of data processing inequalities for points in Euclidean space depending on the continuity properties of the computable transformation that is used. Chapter 4 focuses on extending mutual dimension to sequences over an arbitrary alphabet. First, we prove that the mutual dimension between two sequences is equal to the mutual dimension between the sequences\u27 real representations. Using this result, we show that the lower and upper mutual dimensions between sequences have nice properties. We also provide an analysis of data processing inequalities for sequences where transformations are represented by Turing functionals whose use and yield are bounded by computable functions. Chapter 5 relates mutual dimension to the study of algorithmic randomness. Specifically, we show that a particular class of coupled random sequences, i.e., sequences generated by independent tosses of coins whose biases may or may not be correlated, can be characterized by classical Shannon mutual information. We also prove that any two sequences that are independently random with respect to computable probability measures have zero mutual dimension and that the converse of this statement is not true. We conclude this chapter with some initial investigations on Billingsley mutual dimension, i.e., mutual dimension with respect to probability measures, and prove the existence of a mutual divergence formula

    A Divergence Formula for Randomness and Dimension

    Get PDF
    If S is an infinite sequence over a finite alphabet Σ and β is a probability measure on Σ, then the dimension of S with respect to β, written dim β (S), is a constructive version of Billingsley dimension that coincides with the (constructive Hausdorff) dimension dim(S) when β is the uniform probability measure. This paper shows that dim β (S) and its dual Dim β (S), the strong dimension of S with respect to β, can be used in conjunction with randomness to measure the similarity of two probability measures α and β on Σ. Specifically, we prove that the divergence formula dim β (R) = Dim β (R)
    corecore