Let \mu be a computable ergodic shift-invariant measure over the Cantor
space. Providing a constructive proof of Shannon-McMillan-Breiman theorem,
V'yugin proved that if a sequence x is Martin-L\"of random w.r.t. \mu then the
strong effective dimension Dim(x) of x equals the entropy of \mu. Whether its
effective dimension dim(x) also equals the entropy was left as an problem
question. In this paper we settle this problem, providing a positive answer. A
key step in the proof consists in extending recent results on Birkhoff's
ergodic theorem for Martin-L\"of random sequences

Hoyrup, Mathieu

English

arXiv

Dagstuhl Research Online Publication Server

The dimension of ergodic random sequences

International audienceLet m be a computable ergodic shift-invariant measure over the set of infinite binary sequences. Providing a constructive proof of Shannon-McMillan-Breiman theorem, V'yugin proved that if x is a Martin-Löf random binary sequence w.r.t. m then its strong effective dimension Dim(x) equals the entropy of m. Whether its effective dimension dim(x) also equals the entropy was left as an open problem. In this paper we settle this problem, providing a positive answer. A key step in the proof consists in extending recent results on Birkhoff's ergodic theorem for Martin-Löf random sequences. At the same time, we present extensions of some previous results. As pointed out by a referee the main result can also be derived from results by Hochman [Upcrossing inequalities for stationary sequences and applications. The Annals of Probability, 37(6):2135--2149, 2009], using rather different considerations

INRIA a CCSD electronic archive server

HAL Id: inria-00606457https://hal.inria.fr/inria-00606457v4Submitted on 21 Jul 2011HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.The dimension of ergodic random sequencesMathieu HoyrupTo cite this version:Mathieu Hoyrup. The dimension of ergodic random sequences. STACS’12 (29th Symposium onTheoretical Aspects of Computer Science), Feb 2012, Paris, France. pp.567-576. ￿inria-00606457v4￿The dimension of ergodic random sequencesMathieu HoyrupJuly 21, 2011AbstractLet µ be a computable ergodic shift-invariant measure over {0, 1}N . Providinga constructive proof of Shannon-McMillan-Breiman theorem, V’yugin proved that ifx ∈ {0, 1}N is Martin-Löf random w.r.t. µ then the strong effective dimension Dim(x)of x equals the entropy of µ. Whether its effective dimension dim(x) also equals theentropy was left as an problem question. In this paper we settle this problem, provid-ing a positive answer. A key step in the proof consists in extending recent results onBirkhoff’s ergodic theorem for Martin-Löf random sequences.Keywords: Shannon-McMillan-Breiman theorem; Martin-Löf random sequence; effectiveHausdorff dimension; compression rate; entropy.1 IntroductionThe effective dimension and strong effective dimension of an infinite binary sequence x aredefined asdim(x) = lim infnK(x↾n)nDim(x) = lim supnK(x↾n)n,where K(w) is the Kolmogorov complexity of w.They can be characterized as effective versions of Hausdorff and packing dimensionsrespectively, or by divergence of s-gales (see [Lut00, May02, AHLM07] for the originalresults and [Lut05] for a survey).Let p ∈ [0, 1] be a computable real number and µp the Bernoulli measure over Cantorspace given by µp[w] = p|w|1(1−p)|w|0. It is well-known that if an infinite binary sequence xis Martin-Löf random w.r.t. µp then dim(x) = Dim(x) = h(µp), where h(µp) is the entropyof µp defined byh(µp) = −p log(p)− (1− p) log(1− p). (1)1This result is not difficult to prove and reduces to the strong law of large numbers forMartin-Löf random sequences, as on the one hand1K(x↾n) = − log µp[x↾n] +O(1)for µp-random sequences by Levin-Schnorr theorem, and on the other hand−1nlog µp[x↾n] = −|x↾n|1nlog(p)−|x↾n|0nlog(1− p)which converge to h(µp) for µp-random sequences, by the Strong Law of Large Numbersfor Martin-Löf random sequences.This result highlights the relationship between Shannon’s information theory, Kol-mogorov algorithmic information theory and effective randomness.Ergodic theory provides a natural extension of information theory in which manyresults can be transferred, with more involved proofs, from the case of independent iden-tically distributed random variables to the ergodic case, where independence is only re-quired asymptotically, in the average (see Section 2 for a precise definition).First, the strong law of large numbers extends to Birkhoff’s ergodic theorem. Second,the coincidence between local information and entropy extends through the Shannon-McMillan-Breiman theorem. Whether Martin-Löf randomness fits with these theoremshas been an open problem for a while. The first results were proved by V’yugin [V’y98],based on non-classical, constructive proofs of the theorems. He proved, in particular:Theorem 1.1 (Effective Birkhoff ergodic theorem I). Let µ be a computable shift-invariantergodic measure over {0, 1}N and f ∈ L1(µ) be computable. For every Martin-Löf µ-randomsequence x,limn→∞1nn−1∑k=0f ◦ T k(x) =∫f dµ.The entropy of an ergodic measure is defined ash(µ) = limn→∞−1n∑|w|=nµ[w] logµ[w]. (2)Observe that (1) and (2) are consistent as they give the same quantity when µ is a Bernoullimeasure.Theorem 1.2 (Effective Shannon-McMillan-Breiman theorem I). Let µ be a computable shift-invariant ergodic measure over {0, 1}N. For every Martin-Löf µ-random sequence x,lim supn→∞K(x↾n)n= lim supn→∞−1nlog µ[x↾n] = h(µ).1K is the prefix version of Kolmogorov complexity2The question whether lim inf K↾nncoincides with h(µ) for every Martin-Löf µ-randomwas left open by V’yugin. An alternative proof of Theorem 1.2 approximating ergodicmeasures by Markovian measures was later developed by Nakamura [Nak05], but alsoleft the question open. In this paper we provide a positive answer to this question.A classical proof of the Shannon-McMillan-Breiman theorem uses Birkhoff’s ergodictheorem, applied to some particular functions. The problem in making it effective is thatthese functions are not computable in general. Recent works have been achieved to pushthe effective ergodic theorem to the largest possible class of functions. Here we extend itenough to get the full effective Shannon-McMillan-Breiman theorem.In Section 2 we recall basic notions of computability, randomness and ergodic theory.In Section 3 we develop effective versions of Birkhoff’s ergodic theorem. In Section 4 wepresent our main result.2 Background and notationsWe work on the Cantor space {0, 1}N of infinite binary sequences. A finite word w ∈{0, 1}∗ determines the cylinder [w] ⊆ {0, 1}N of infinite sequences starting with w. Ifx ∈ {0, 1}N and n ∈ N, x↾n is the prefix of x of length n, and is also denoted x0x1 . . . xn−1.The cylinders form a base of the product topology.Effective topology. An open set U ⊆ {0, 1}N is effective if it is a recursively enumerableunion of cylinders. A closed set is effective it its complement is an effective open set.A function f : {0, 1}N → R is computable if there is a Turing machine that on oracle xand input n computes a rational number q such that |q − f(x)| < 2−n. Equivalently, f iscomputable if for every rational numbers a < b, f−1(a, b) is effectively open, uniformlyin a, b. A function f : {0, 1}N → [0,+∞] is lower (resp. upper) semi-computable if thereis a Turing machine that on oracle x and input n computes a rational number qn suchthat f(x) = supn qn (resp. f(x) = infn qn). Equivalently, f is lower (resp. upper) semi-computable if for every rational number a, f−1(a,+∞] (resp. f−1[0, a)) is effectively open,uniformly in a.Kolmogorov complexity and Martin-Löf randomness. For w ∈ {0, 1}∗, K(w) is theprefix version of Kolmogorov complexity, defined by Levin and Chaitin independently.It is defined as the length of a shortest input of a universal Turing machine with prefix-freedomain computing w on that input.A probability measure µ over {0, 1}N is determined by its value on cylinders µ[w], forw ∈ {0, 1}∗. µ is computable if µ[w] is a computable real number, uniformly in w. Givena computable probability measure µ, a sequence x ∈ {0, 1}N is Martin-Löf µ-random,denoted x ∈ MLµ, if there is c such that for all n,K(x↾n) ≥ − log µ[x↾n]− c.3Martin-Löf’s original definition [ML66] was expressed in terms of tests rather than com-plexity, but the one given here, due to Levin and Chaitin [Cha75] independentely, wasproved to be equivalent by Levin [Lev73] and Schnorr [Sch73].The functiontµ(x) = supn{− log µ[x↾n]−K(x↾n)}is lower semi-computable and∫2tµ dµ ≤ 1. Moreover, it was proved in [Gác80] that 2tµis maximal in the sense that for every integrable lower semi-computable function f :{0, 1}N → [0,+∞], there exists cf such that f ≤ cf2tµ. We call such an f a µ-test. Ittests Martin-Löf randomness in the sense that x ∈ MLµ iff f(x) < ∞ for each µ-test f ifftµ(x) < ∞. More can be found on this subject in [Nie09, DH10].Ergodic theory. We recall some basic notions of ergodic theory, more details can befound in [Smo71, Pet83]. We denote by T : {0, 1}N → {0, 1}N the shift map defined byT (x0x1 . . .) = x1x2 . . .. A measure µ over {0, 1}N is shift-invariant if for all Borel sets A,µ(T−1A) = µ(A), equivalently if µ[0w] + µ[1w] = µ[w] for all w ∈ {0, 1}∗. µ is ergodic iffor all Borel sets A such that T−1A = A up to a null sets, µ(A) = 0 or 1. Equivalently, µ isergodic if for all u, v ∈ {0, 1}∗,limn→∞1nn−1∑k=0µ([u] ∩ T−k[v]) = µ[u] · µ[v].3 Effective ergodic theoremsThe following theorem, taken from [BDMS10], extends a result of Kučera from the uni-form measure to any ergodic shift-invariant measure:Theorem 3.1 (Effective Poincaré recurrence theorem). Let µ be a computable ergodic shift-invariant measure and C ⊆ {0, 1}N an effective closed set such that µ(C) > 0. Every Martin-Löfµ-random sequence has a tail in C, i.e. for every x ∈ MLµ there exists k such that Tk(x) ∈ C.In [BDH+10] and [FGMN10] independently this result was used to prove that not onlythe orbit of x eventually falls into C, but it does so with frequency µ(C).Theorem 3.2 (Effective Birkhoff ergodic theorem II). Let µ be a computable ergodic shift-invariant measure and C ⊆ {0, 1}N an effective closed set such that µ(C) > 0. For every Martin-Löf µ-random sequence x,limn→∞1n|{k < n : T k(x) ∈ C}| = µ(C).We first generalize the result from sets to functions:Theorem 3.3 (Effective Birkhoff ergodic theorem III). Let µ be a computable ergodic shift-invariant measure. Assume f : {0, 1}N → [0,+∞] is:4• either lower semi-computable,• or upper semi-computable and bounded by a µ-test.For each x ∈ MLµ,limn→∞n−1∑k=0f ◦ T k(x) =∫f dµ.Proof. Let us introduce the notation Afn(x) =1n(f(x) + . . .+ f ◦ T n−1(x)).If f is lower semi-computable, then there is a sequence of uniformly computable non-negative functions fn ր f . Applying Theorem 1.1 to fn and x ∈ MLµ gives lim infk Afk(x) ≥lim infk Afnk (x) =∫fn dµ. By the monotone convergence theorem,∫fn dµ ր∫f dµ, solim infk Afk(x) ≥∫f dµ. If∫f dµ = ∞ we are done. Otherwise, let q >∫f dµ be a rationalnumber. The set CK := {x : ∀k ≥ K,Afk(x) ≤ q} is effectively closed and by the classicalergodic theorem, there exists K such that µ(CK) > 0. Theorem 3.1 tells us that if x ∈ MLµthen there is n such that T n(x) ∈ CK . As a result, lim supAfk(x) = lim supAfk(Tn(x)) ≤ q.As this is true of every q >∫f dµ, we get the result.Now, if f is upper semi-computable and f ≤ t where t is a µ-test, then for x ∈ MLµ,applying the preceding result to t and t− f ,Afn(x) = Atn(x)− (At−fn (x)) →∫t dµ−∫(t− f) dµ =∫f dµ.We then extend this result further:Corollary 3.1 (Effective Birkhoff ergodic theorem IV). Let f : {0, 1}N → [0,+∞] be ∆02 onMLµ, i.e. there is a sequence fn of uniformly computable functions such that f(x) = limn fn(x)for each x ∈ MLµ. Assume that f is dominated by a µ-test. For every x ∈ MLµ,limn→∞n−1∑k=0f ◦ T k(x) =∫f dµ.Proof. Let gN = infn≥N fn and hN = min(t, supn≥N fn). On MLµ, gN ր f and hN ց f .By the monotone and dominated convergence theorem, the convergences hold in L1(µ).Applying Theorem 3.2 to gN and hN gives the result. More precisely, for every x ∈ MLµand every N ,lim infnAfn(x) ≥ lim infnAgNn (x) =∫gN dµlim supnAfn(x) ≤ lim supnAhNn (x) =∫hN dµ,so∫f dµ = supN∫gN dµ ≤ lim infnAfn(x) ≤ lim supnAfn(x) ≤ infN∫hN dµ =∫f dµ.54 The effective Shannon-McMillan-Breiman theoremWe now present our main result.Theorem 4.1 (Effective Shannon-McMillan-Breiman theorem II). Let µ be a computable shift-invariant probability measure. For each x ∈ MLµ,limn→∞K(x↾n)n= limn→∞−1nlog µ[x↾n] = h(µ).A proof of the classical result, stating the result for a.e. x, can be found in [Smo71,Pet83]. It makes use of martingale convergence theorems and ergodic theorems. The maindifficulty in adapting the proof is to make sure that the effective versions of the ergodictheorem can be applied. The rest of this section is devoted to the proof of Theorem 4.1.An easy calculation shows that− logµ[x↾n] =n−1∑k=0fn−1−k ◦ Tk(x) (3)wherefk(x) := − log µ[x0|x1 . . . xk] = − logµ[x0 . . . xk]µ[x1 . . . xk]for k ≥ 1,f0(x) := − log µ[x0].Lemma 4.1. fk(x) converge for each x ∈ MLµ.Proof. Define the computable martingaled(ǫ) = 2d(x0) =1µ[x0]d(x0 . . . xk) =µ[x1 . . . xk]µ[x0 . . . xk]for k ≥ 1.By the effective Doob’s convergence theorem (see Theorem 7.1.3 on page 270 in [DH10]),for each x ∈ MLµ, d(x0 . . . xk) converges, and so does fk(x) = log d(x0 . . . xk).Let f(x) be the limit. We write−1nlogµ[x↾n] =1nn−1∑k=0(fn−1−k ◦ Tk(x)− f ◦ T k(x)) +1nn−1∑k=0f ◦ T k(x)and prove that the first term tends to 0 while the second term converges to∫f dµ = h(µ).We will use the following lemma (Corollary 2.2 on page 261 in [Pet83], Lemma 4.26 onpage 26 in [Smo71]).6Lemma 4.2. f ∗ := supk fk ∈ L1.As fk → f a.e. and the convergence is dominated by f∗ ∈ L1, fk → f in L1.Proposition 4.1. For each x ∈ MLµ,limn1nn−1∑k=0f ◦ T k(x) =∫f dµ = hµ(P ). (4)Proof. That∫f dµ = h(µ) is a classical result and follows from h(µ) = limk∫fk dµ and theL1-convergence of fk to f .f ∗ is lower semi-computable and by Lemma 4.2 it is a µ-test. By construction, f is ∆02on MLµ and it is dominated by f∗ so it satisfies the conditions of Corollary 3.1, from whichthe result follows directly.Proposition 4.2. For each x ∈ MLµ,limn→∞1nn−1∑k=0fn−1−k ◦ Tk(x)− f ◦ T k(x) = 0. (5)Proof. LetgN = supk≥N|fk − f | and g̃N = supk,j≥N|fk − fj |.For x ∈ MLµ,|fk(x)− f(x)| = limj|fk(x)− fj(x)|= lim supj|fk(x)− fj(x)|≤ supj≥N|fk(x)− fj(x)|,so gN(x) ≤ g̃N(x). As fk → f a.e., g̃N → 0 a.e. As g̃N ≤ 2f∗ ∈ L1, g̃N → 0 in L1 by thedominated convergence theorem. On MLµ,∣∣∣∣∣1nn−1∑k=0fn−1−k ◦ Tk − f ◦ T k∣∣∣∣∣≤1nn−1∑k=0|fn−1−k ◦ Tk − f ◦ T k|=1nn−1−N∑k=0|fn−1−k ◦ Tk − f ◦ T k|+1nn−1∑k=n−N|fn−1−k ◦ Tk − f ◦ T k|≤1nn−1−N∑k=0gN ◦ Tk +1nn−1∑k=n−N(f ∗ + f) ◦ T k≤1nn−1−N∑k=0g̃N ◦ Tk +1nn−1∑k=0(f ∗ + f) ◦ T k −1nn−N−1∑k=0(f ∗ + f) ◦ T k.7Fix N and let n → ∞. As g̃N ∈ L1 is lower semi-computable, the first term converges to∫g̃N dµ by the Effective Ergodic Theorem 3.3. As f∗+f is ∆02 on MLµ and is dominated bythe µ-test 2f ∗, the second and the third terms converge to∫(f ∗ + f) dµ by Corollary 3.1so their limits cancel each other.As∫g̃N dµ → 0, we have proved equality (5).Putting equalities (3), (4) and (5) together gives, for x ∈ MLµ,limn−1nlog µ[x↾n] = limn1nn−1∑k=0fn−1−k ◦ Tk(x) = h(µ).References[AHLM07] Krishna B. Athreya, John M. Hitchcock, Jack H. Lutz, and Elvira Mayor-domo. Effective strong dimension in algorithmic information and compu-tational complexity. SIAM J. Comput., 37(3):671–705, 2007. 1[BDH+10] Laurent Bienvenu, Adam R. Day, Mathieu Hoyrup, Ilya Mezhirov, andAlexander Shen. A constructive version of Birkhoff’s ergodic theorem forMartin-Löf random points. Submitted. ArXiv 1007.5249, 2010. 4[BDMS10] Laurent Bienvenu, Adam Day, Ilya Mezhirov, and Alexander Shen. Ergodic-type characterizations of algorithmic randomness. In Computability in Europe(CIE 2010), volume 6158 of Lecture Notes in Computer Science, pages 49–58.Springer, 2010. 4[Cha75] Gregory J. Chaitin. A theory of program size formally identical to informationtheory. J. ACM, 22(3):329–340, 1975. 4[DH10] Rod Downey and Denis Hirschfeldt. Algorithmic Randomness and Complexity.Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2010. 4, 6[FGMN10] Johanna N.Y. Franklin, Noam Greenberg, Joseph S. Miller, and Keng MengNg. Martin-Löf random points satisfy Birkhoff’s ergodic theorem for effec-tively closed sets. To appear in the Proceedings of the American MathematicalSociety, 2010. 4[Gác80] Péter Gács. Exact expressions for some randomness tests. Z. Math. Log. Grdl.M., 26:385–394, 1980. 4[Lev73] Leonid A. Levin. On the notion of a random sequence. Soviet MathematicsDoklady, 14:1413–1416, 1973. 4[Lut00] Jack H. Lutz. Dimension in complexity classes. In IEEE Conference on Compu-tational Complexity, pages 158–169, 2000. 18[Lut05] Jack H. Lutz. Effective fractal dimensions. Mathematical Logic Quarterly,51(1):62–72, 2005. 1[May02] Elvira Mayordomo. A Kolmogorov complexity characterization of construc-tive Hausdorff dimension. Inf. Process. Lett., 84(1):1–3, 2002. 1[ML66] Per Martin-Löf. The definition of random sequences. Information and Control,9(6):602–619, 1966. 4[Nak05] Masahiro Nakamura. Ergodic theorems for algorithmically random se-quences. Proceedings of the Symposium on Information Theory and Its Applica-tions, 2005. 3[Nie09] A. Nies. Computability and randomness. Oxford logic guides. Oxford Univer-sity Press, 2009. 4[Pet83] Karl Petersen. Ergodic Theory. Cambridge Univ. Press, 1983. 4, 6[Sch73] Claus-Peter Schnorr. Process complexity and effective random tests. J. Com-put. Syst. Sci., 7(4):376–388, 1973. 4[Smo71] Meir Smorodinsky. Ergodic Theory, Entropy, volume 214 of Lecture Notes inMathematics. Springer-Verlag, Berlin Heibelberg New York, 1971. 4, 6[V’y98] Vladimir V. V’yugin. Ergodic theorems for individual random sequences.Theoretical Computer Science, 207(4):343–361, 1998. 29

https://hal.inria.fr/inria-00606457/document

The dimension of ergodic random sequences

Abstract

Similar works

Full text

Available Versions

Dagstuhl Research Online Publication Server

INRIA a CCSD electronic archive server