Let A be finite set equipped with a probability distribution P, and let M be a “mass” function on A. A characterization is given for the most efficient way in which A n can be covered using spheres of a fixed radius. A covering is a subset C n of A n with the property that most of the elements of A n are within some fixed distance from at least one element of C n , and “most of the elements” means a set whose probability is exponentially close to one (with respect to the product distribution P n ). An efficient covering is one with small mass M n (C n ). With different choices for the geometry on A, this characterization gives various corollaries as special cases, including Marton’s error-exponents theorem in lossy data compression, Hoeffding’s optimal hypothesis testing exponents, and a new sharp converse to some measure concentration inequalities on discrete spaces

C McDiarmid

I Csiszár

I Kontoyiannis

K Marton

LH Harper

M Talagrand

R Ahlswede

RE Blahut

W Hoeffding

English

arXiv

Ioannis Kontoyiannis

Ali Devin Sezer

Crossref

A Remark on Unified Error Exponents: Hypothesis Testing, Data Compression and Measure Concentration

Kontoyiannis, Ioannis

Sezer, Ali Devin

Apollo (Cambridge)

arXiv:math/0210055v1  [math.PR]  3 Oct 2002A Remark on Unified Error Exponents:Hypothesis Testing, Data Compression & Measure ConcentrationI. Kontoyiannis A.D. SezerNovember 3, 2018AbstractLet A be finite set equipped with a probability distribution P , and let M be a “mass” functionon A. A characterization is given for the most efficient way in which An can be covered usingspheres of a fixed radius. A covering is a subset Cn of An with the property that most of theelements of An are within some fixed distance from at least one element of Cn, and “most ofthe elements” means a set whose probability is exponentially close to one (with respect to theproduct distribution Pn). An efficient covering is one with small mass Mn(Cn). With differentchoices for the geometry on A, this characterization gives various corollaries as special cases,including Marton’s error-exponents theorem in lossy data compression, Hoeffding’s optimal hy-pothesis testing exponents, and a new sharp converse to some measure concentration inequalitieson discrete spaces.1Ioannis Kontoyiannis is with the Division of Applied Mathematics and the Department of Computer Sci-ence, Brown University, Box F, 182 George St., Providence, RI 02912, USA. Email: yiannis@dam.brown.eduWeb: www.dam.brown.edu/people/yiannis/2Ali Devin Sezer is with the Division of Applied Mathematics, Brown University, Box F, 182 George St., Provi-dence, RI 02912, USA. Email: ali sezer@brown.edu3I. Kontoyiannis was supported in part by NSF grants #0073378-CCR and DMS-9615444, and by USDA-IFAFSgrant #00-52100-9615.1 IntroductionLet A be a finite set and P a probability distribution on A. Suppose that the distance (or “dis-tortion”) ρ(x, y) between any two points x, y ∈ A is measured by a given nonnegative functionρ : A×A → [0,∞), and for strings xn1 = (x1, x2, . . . , xn) and yn1 = (y1, y2, . . . , yn) in An letρn(xn1 , yn1 ) be the corresponding coordinate-wise distance (or single-letter distortion measure) onAn ×An:ρn(xn1 , yn1 ) =1nn∑i=1ρ(xi, yi).Since A is a finite set, the function ρ is bounded above byDmax△= maxx,y∈Aρ(x, y) = maxxn1,yn1∈Anρn(xn1 , yn1 ).Without loss of generality we assume throughout that P (a) > 0 for all a ∈ A, and thatfor each a ∈ A there exists a b ∈ A with ρ(a, b) = 0 (otherwise we may consider ρ′(x, y) =[ρ(x, y)−minz∈A ρ(x, z)] instead of ρ(x, y)).Given a D ≥ 0, we want to cover “most” of An using balls B(yn1 ,D), whereB(yn1 ,D) = {xn1 ∈ An : ρn(xn1 , yn1 ) ≤ D}is the closed ball of radius D centered at yn1 ∈ An. To be precise, given a set Cn ⊂ An, we write[Cn]D for the D-blowup of Cn,[Cn]D△=⋃yn1∈CnB(yn1 ,D).A D-covering of An is a sequence of subsets Cn of An, n ≥ 1, such that the Pn-probability of thepart of An which is not covered by Cn within distance D has exponentially small probability,Pr{“error”}△= 1− Pn([Cn]D) ≈ 2−nE , (1)for some E > 0. We are interested in “efficient” coverings of An, that is, given a “mass function”M : A → (0,∞), we want to find D-coverings {Cn} that satisfy (1) and also have small massMn(Cn)△=∑yn1∈CnMn(yn1 ) =∑yn1∈Cnn∏i=1M(yi).Clearly there is a trade-off between finding coverings {Cn} with small mass, and coverings with agood (i.e., large) error-exponent E as in (1). Typically, the better the error-exponent, the largerthe Cn, and the bigger their mass would tend to be.Motivated, in part, by the following example and by the applications illustrated in the examplesof the following section, in our main result we give a precise characterization of this trade-off.2Example: Measure Concentration on the Binary Cube.Consider the n-dimensional binary cube An = {0, 1}n. We measure distance on An by theproportion of mismatches between two binary strings xn1 and yn1 , i.e., we take ρn(xn1 , yn1 ) to be theHamming distance,ρn(xn1 , yn1 ) =1nn∑i=1I{xi 6=yi}, xn1 , yn1 ∈ An, (2)which also coincides with the normalized graph distance when An is equipped with the nearest-neighbor graph structure. For simplicity, in this example we consider natural logarithms andexponentials.A well-known measure concentration inequality [10, Prop. 2.1.1][9, Thm. 3.5] gives a preciselower bound on the sphere-covering error probability of an arbitrary Cn: For any D ≥ 0, anyproduct distribution Pn on An, and any Cn ⊂ An,Pr{“error”} = 1− Pn([Cn]D) ≤e−nD2/2Pn(Cn).Therefore, if {Cn} is any D-covering consisting of sets with Pn(Cn) ≈ e−nr for some r > 0, thenthe union of the balls B(yn1 ,D) centered at the points yn1 ∈ Cn covers all of An except for a set ofprobability no greater than≈ e−n(D22−r). (3)It is then natural to ask, what is the best achievable error exponent among all D-coverings {Cn}with probability no greater that ≈ e−nr? In other words, we are asking for small sets with thelargest possible “boundary,” sets Cn with “volume” Pn(Cn) no greater than e−nr but whose D-blowups [Cn]D cover as much of An as possible. As pointed in [6], this question can be thought ofas the opposite of the usual isoperimetric problem.Taking M = P in the general setting described above, we obtain the answer to this question asa corollary to our general result in the following section; see Corollary 3.2 ResultsGiven any D ≥ 0 and any R ∈ R, let E(R,D) denote the best achievable error-exponent among allD-coverings with mass asymptotically bounded by 2nR. Letting C(R) denote the collection of allsequences of subsets Cn of An with lim supn1n logMn(Cn) ≤ R, define,E(R,D)△= sup{Cn}∈C(R)lim infn→∞−1nlog[1− Pn([Cn]D)],where ‘log’ denotes the logarithm taken to base 2.3A weaker version of this problem was recently considered in [6], where it was shown that theprobability of error can only decrease to zero if R is greater than R(D;P,M),R(D;P,M)△= inf(X,Y ): X∼P, Eρ(X,Y )≤D{H(PX,Y ‖P × PY ) + E[logM(Y )]}, (4)where the infimum is taken over all jointly distributed random variables (X,Y ) such that X hasdistribution P and Eρ(X,Y ) ≤ D, and PX,Y denotes the joint distribution of X,Y , PY denotesthe marginal distribution of Y , and H(µ‖ν) denotes the relative entropy between two probabilitymeasures µ and ν on the same finite set S,H(µ‖ν)△=∑s∈Sµ(s) logµ(s)ν(s).Therefore, the error-exponent E(R,D) can only be nontrivial (i.e., nonzero) for R > R(D;P,M).Also note that any Cn ⊂ An has1nlogMn(Cn) ≤1nlogMn(An) = logM(A).Hence, from now on we restrict attention to the range of interesting values forR between R(D;P,M)and Rmax△= logM(A).Theorem. For all D ∈ [0,Dmax) and all R(D;P,M) < R < Rmax, the best achievable exponent ofthe error probability, among all D-coverings {Cn} with mass asymptotically bounded by 2nR, isE(R,D) = E∗(R,D)△= infQ :R(D;Q,M)>RH(Q‖P ),where R(D;P,M) is defined in (4) and H(Q‖P ) denotes the relative entropy (or Kullback-Leiblerdivergence) between two distributions P and Q.Remarks.1. A slightly different error-exponent. Alternatively, we can define a version of the optimalerror-exponent by considering only D-coverings {Cn} with mass bounded by 2nR for all n:E′(R,D)△= lim infn→∞−1nlog{minCn :Mn(Cn)≤2nR[1− Pn([Cn]D)]}.From the theorem it easily follows that E′(R,D) is also equal to E∗(R,D) at all points R whereE∗(R,D) is continuous and, since it is nondecreasing in R, E∗(R,D) is indeed continuous at allexcept countably many values of R. But in general it may fail to be continuous everywhere, asillustrated in the discussions by Marton [7] and Ahlswede [1] for the special case of lossy datacompression (which corresponds to taking M(x) ≡ 1; see Example 2 below).42. Proof. The proof of the theorem is a modification of Marton’s [7] original argument for thecase of error-exponents in lossy data compression. The optimal sets {Cn} achieving E∗(R,D) arerandomly generated, and they are universal in that their construction only depends on R, D, andM . Therefore, they achieve the optimal error-exponent simultaneously for all distributions P .Example 1: Hypothesis Testing.Let P0 and P1 be two probability distributions A with all positive probabilities. Suppose thatthe null hypothesis that a sample Xn1 = (X1,X2, . . . ,Xn) of n independent observations comesfrom P0 is to be tested against the simple alternative that Xn1 comes from P1. Any test betweenthese two hypotheses is simply a decision region Cn ⊂ An: If Xn1 ∈ Cn we declare that Xn1 ∼ Pn1 ,otherwise we declare Xn1 ∼ Pn0 . The set Cn is called the critical region, and the type-I and type-IIprobabilities of error associated with the test are, respectively,αn = Pn0 (Cn) and βn = Pn1 (Ccn).Clearly we wish to have αn and βn both decrease to zero as fast as possible. In particular, we mayask how quickly βn can decay to zero if we require that αn decays exponentially at some rate r > 0,i.e., αn ≈ 2−nr. In statistical terminology, we are asking for the fastest rate of decay of the type-IIerror probability among all tests with significance level αn ≤ 2−nr.Formally, we want to identify the best exponent of the error probability βn = 1−Pn1 (Cn) amongall Cn with Pn0 (Cn) ≤ 2−nr. Taking P = P1, M = P0, R = −r, and allowing no distortion, thisquestion reduces exactly to the our earlier sphere-covering problem. [To be precise, allowing nodistortion means we take D = 0 with ρ(x, y) being Hamming distortion as in (2).] Accordingly,R(D;P,M) = R(0;P1, P0) turns out to be equal to −H(P1‖P0), and from the theorem we immedi-ately obtain the following classical result of Hoeffding. Also see [2, Thms. 9, 10] and [3, Ex.12, p.43]for versions of this result in the information theory literature.Corollary 1. (Hypothesis Testing) [5] Let {Cn} be an arbitrary sequence of tests with associatederror probabilities αn and βn as above. Among all tests withlim supn→∞1nlogαn ≤ −rfor some r ∈ (0,H(P1‖P0)), the fastest achievable asymptotic rate of decay of βn islimn→∞−1nlog βn = infQ :H(Q‖P0)<rH(Q‖P1).As mentioned earlier, the optimal decision regions Cn in the Corollary are randomly generated.Therefore, although they do achieve asymptotically optimal performance, they are not optimal forfinite n in the Neyman-Pearson sense.5Example 2: Lossy Data Compression.Suppose data Xn1 = (X1,X2, . . . ,Xn) is generated by a stationary, memoryless source, i.e., Xn1are i.i.d. (independent and identically distributed) random variables, with distribution P on thefinite alphabet A. The objective of lossy data compression is to find efficient representations yn1 ∈ Anfor all source strings xn1 ∈ An. In particular, suppose that the maximum amount of distortionρn(xn1 , yn1 ) that we are willing to tolerate between any source string xn1 and its representation yn1is some D ≥ 0, where {ρn} is a family of single-letter distortion measures as in (1). Then theproblem is to find an efficient codebook Cn ⊂ An such that for most of the source strings xn1 thereis a yn1 ∈ Cn with ρn(xn1 , yn1 ) ≤ D.Here, an efficient codebook Cn is one that leads to good compression, i.e., one whose size is assmall as possible. And, on the other hand, we also want to make sure that the probability that asource string cannot be represented by any element of Cn with distortion D or less, is small. TakingM to be counting measure (M(x) = 1 for all x ∈ A), the mass Mn(Cn) of the codebook becomesits size |Cn|, and the problem of finding a good codebook reduces to the earlier sphere-coveringquestion. Accordingly, the rate-function R(D;P ;M) reduces to Shannon’s rate-distortion functionR(D;P ), and the theorem yields Marton’s error-exponents result.Corollary 2. (Lossy Data Compression) [7] Let D ≥ 0 be a given distortion level, and R(D;P ) <R < log |A|. Among all sequences of codebooks {Cn} with asymptotic rate no greater thanR bits/symbol,lim supn→∞1nlog |Cn| ≤ R,the fastest achievable asymptotic rate of decay of the probability of error islimn→∞−1nlog[1− Pn([Cn]D)]= infQ :R(D;Q)>RH(Q‖P ).Example 3: Measure Concentration on the Binary Cube.Consider again the setting of the example described in the introduction. There we asked forthe best achievable error exponent among all D-coverings {Cn} with probability no greater that≈ e−nr. Taking M = P in the theorem, we obtain the answer to this question in the followingCorollary. Let He(P‖Q) denote the relative entropy expressed in nats rather than bits, He(P‖Q) =(loge 2)H(P‖Q), and similarly write Re(D;P,M) = (loge 2)R(D;P,M).Corollary 3. (Converse Measure Concentration) Let D ≥ 0 and 0 < r < −Re(D;P,P ). Amongall D-coverings {Cn} withlim supn→∞1nloge Pn(Cn) ≤ −r,the fastest achievable asymptotic rate of decay of the probability of error islimn→∞−1nloge[1− Pn([Cn]D)]= E∗(r,D),6whereE∗(r,D)△= infQ :Re(D;Q,P )>−rHe(Q‖P ).Although the exponent E∗(r,D) above is not as explicit as (D22 − r) in (3), it is easy to evaluatenumerically and it contains much more useful information. For example, Figure 1 shows the graphof E∗(r,D) as a function of r, for D = 0.3, P being the Bernoulli(0.4 ) distribution, and r runningover the range r ∈ (0.6109, 0.6393) where E∗(r,D) is nontrivial (i.e., finite and nonzero). In thiscase, (3) is only useful when (D22 − r) is positive, i.e., for r ∈ (0, 0.045): There (3) says that,whenever Pn(Cn) ≈ e−nr for some r ∈ (0, 0.045), the probability of error decays exponentially fast.But in that range, and in fact for all r up to ≈ 0.61, we have E∗(r,D) = ∞ so there are sets Cnwith Pn(Cn) ≈ e−nr and probability of error decaying super-exponentially fast. Moreover, in therange r ∈ (0.6109, 0.6393) where E∗(r,D) is nontrivial, we can choose Cn with Pn(Cn) ≈ e−nr andPr{“error”} ≈ e−nE∗(r,D).0.59 0.6 0.61 0.62 0.63 0.64 0.6500.020.040.060.080.10.12 (infinity)Figure 1: Graph of the error-exponent function E∗(r,D) in Corollary 3 as a function of r, forD = 0.3 and P (1) = 0.4. Note that E∗(r,D) is infinite for all r ∈ (0, 0.6109), and that it is zero forr > 0.6393.Finally we remark that the “extremal” sets in the classical isoperimetric problem, namely,those Cn that achieve equality in (3), are very different from the extremal sets in Corollary 3.7The former are well-known to be Hamming balls Bn centered at 0n = (0, 0, . . . , 0) ∈ An, Bn ={xn1 : ρn(xn1 , 0n) ≤ δ} (see [4][8, p. 174][10, Sec. 2.3]), while the latter are collections of strings yn1randomly selected from a collection of suitable strings.Extensions.1. Different alphabets. Although we assumed from the start that ρ(x, y) is a distortion measureon A × A, it is straightforward to generalize the main result as well as the subsequent discussionabove to the case when ρ(x, y) is a distortion measure between the “source” alphabet A and adifferent (“reproduction”) alphabet Â, as long as it is still the case that for each a ∈ A there existsa b ∈ Â with ρ(a, b) = 0. The necessary modifications to the statements and proofs follow exactlyas in the case of Marton’s result; see [3, Sec. 2.4].2. Strong converse. As mentioned earlier, the theorem is stated only for values of R aboveR(D;P,M) since we trivially have E(R,D) = 0 for R < R(D;P,M); see [6, Thm. 1]. In that rangeit is also possible to prove a “strong converse” showing that, not only E(R,D) = 0, but in fact theprobability of error goes to one exponentially fast with a certain rate.3 ProofFirst we prove the converse part of the theorem, asserting that E(R,D) ≤ E∗(R,D).Note that the rate-function R(D;P,M) defined in (4) is jointly uniformly continuous in D ≥0 and P ; this can be easily seen to be the case by arguing along the lines of the proof of [3,Lemma 2.2.2] for the rate-distortion function R(D;P ). Now let {Cn} be an arbitrary D-coveringwith {Cn} ∈ C(R). Take any Q on A such that R(D;Q,M) > R (if no such Q exists then theclaim is trivially true), and let δ > 0 be such that R(D;Q,M) > R + δ. Since {Cn} ∈ C(R), wehave logMn(Cn) < n(R+ δ/2), eventually, and by the continuity of R(D;Q,M) in D we can findan η > 0 small enough so thatlogMn(Cn) < n(R+ δ/2) < nR(D + η;Q,M), eventually.Therefore, by the “weak converse” in [6, Thm. 1], we must also haveEQn[minyn1∈Cnρn(Xn1 , yn1 )]> D + η, eventually, (5)where Xn1 denote n i.i.d. random variables with distribution Qn. WritingZn△= minyn1∈Cnρn(Xn1 , yn1 ),the bound in equation (5) implies thatD + η < E[Zn] ≤ DQn(Zn ≤ D) +Dmax Qn(Zn > D)8i.e.,Qn(Zn > D) >ηDmax −D.From Stein’s lemma [3, Cor. 1.1.2] we also know that, for any P and any ǫ > 0,limn→∞1nlog[minBn⊂An :Qn(Bn)>ǫPn(Bn)]= −D(Q‖P ).Taking ǫ = η/(Dmax −D) > 0 and applying this to the eventsBn△= {Zn > D} = [Cn]cD,yieldslim infn→∞1nlog[1− Pn([Cn]D)]≥ −D(Q‖P ),and since this holds for all Q with R(D;Q,M) > R, we obtainlim supn→∞−1nlog[1− Pn([Cn]D)]≤ E∗(R,D).Finally, since {Cn} ∈ C(R) was arbitrary, this establishes that E(R,D) ≤ E∗(R,D), as required.To prove the direct part of the theorem, asserting the existence of a D-covering {Cn} ∈ C(R)such thatlim infn→∞−1nlog[1− Pn([Cn]D)]≥ E∗(R,D),we follow the same outline as in the proof of the direct part of [3, Thm. 2.4.5].Using the joint uniform continuity of R(D;P,M) in D ≥ 0 and P , the proof of the type-coveringlemma [3, Lemma 2.4.1] can be generalized to the corresponding statement with R(D;P,M) in placeof R(D;P ). The main new observation here is that, since all the elements yn1 of the covering setB are drawn from the set T n[Y ∗] of Y∗-typical strings, where (X∗, Y ∗) achieve the infimum in thedefinition (4) of R(D;P,M), their mass Mn(yn1 ) satisfies1nlogMn(yn1 ) ≤ E[logM(Y∗)] + δn[∑ylogM(y)],where the sequence δn → 0 as n → ∞.Finally, following the same steps as in the proof of the direct part of [3, Thm. 2.4.5] andreplacing R(D;P ) by R(D;P,M), we obtain the existence of a D-covering {Cn} ∈ C(R) with errorexponent no worse than E∗(R,D) − δ, where δ > 0 is an arbitrary constant. This proves thatE(R,D) ≥ E∗(R,D), and completes the proof. ✷AcknowledgmentsWe wish to thank Amir Dembo and Neri Merhav for asking us (independently) whether the resultsof [6] could be extended to the case of error-exponents.9References[1] R. Ahlswede. Extremal properties of rate-distortion functions. IEEE Trans. Inform. Theory,36(1):166–171, 1990.[2] R.E. Blahut. Hypothesis testing and information theory. IEEE Trans. Inform. Theory,20(4):405–417, 1974.[3] I. Csiszár and J. Körner. Information Theory: Coding Theorems for Discrete MemorylessSystems. Academic Press, New York, 1981.[4] L.H. Harper. Optimal numberings and isoperimetric problems on graphs. J. CombinatorialTheory, 1:385–393, 1966.[5] W. Hoeffding. Asymptotically optimal tests for multinomial distributions. Ann. Math. Statist.,36:369–408, 1965.[6] I. Kontoyiannis. Sphere-covering, measure concentration, and source coding. IEEE Trans.Inform. Theory, 47:1544–1552, May 2001.[7] K. Marton. Error exponent for source coding with a fidelity criterion. IEEE Trans. Inform.Theory, 20:197–199, 1974.[8] C. McDiarmid. On the method of bounded differences. In Surveys in combinatorics (Norwich,1989), pages 148–188. London Math. Soc. Lecture Note Ser., 141, Cambridge Univ. Press,Cambridge, 1989.[9] C. McDiarmid. Concentration. In Probabilistic methods for algorithmic discrete mathematics,pages 195–248. Algorithms Combin., 16, Springer, Berlin, 1998.[10] M. Talagrand. Concentration of measure and isoperimetric inequalities in product spaces. Inst.Hautes Études Sci. Publ. Math., No. 81:73–205, 1995.10

A Remark on Unified Error Exponents: Hypothesis Testing, Data Compression and Measure Concentration

Abstract

Similar works

Full text

Available Versions

Crossref

Apollo (Cambridge)