Tom Verhoeff

English

Research Papers in Economics

The Laws of Large Numbers Compared
Tom Verhoeff
July 1993
1 Introduction
Probability Theory includes various theorems known as Laws of Large Numbers;
for instance, see [Fel68, Hea71, Ros89]. Usually two major categories are distin-
guished: Weak Laws versus Strong Laws. Within these categories there are numer-
ous subtle variants of differing generality. Also the Central Limit Theorems are
often brought up in this context.
Many introductory probability texts treat this topic superﬁcially, and more than
once their vague formulations are misleading or plainly wrong. In this note, we
consider a special case to clarify the relationship between the Weak and Strong
Laws. The reason for doing so is that I have not been able to ﬁnd a concise formal
exposition all in one place. The material presented here is certainly not new and
was gleaned from many sources.
In the following sections, X1; X2;:::is a sequence of independent and identi-
cally distributed random variables with ﬁnite expectation . We deﬁne the associ-
ated sequence N Xi of partial sample means by
N Xn D
1
n
n X
iD1
Xi :
The Laws of Large Numbers make statements about the convergence of N Xn to .
Both laws relate bounds on sample size, accuracy of approximation, and degree of
conﬁdence. The Weak Laws deal with limits of probabilities involving N Xn.T h e
Strong Laws deal with probabilities involving limits of N Xn. Especially the math-
ematical underpinning of the Strong Laws requires a careful approach ([Hea71,
Ch. 5] is an accessible presentation).
2 The Weak Law of Large Numbers
Let’s not beat about the bush. Here is what the Weak Law says about convergence
of N Xn to .
2.1 Theorem (Weak Law of Large Numbers)W e h a v e
8 ">0 lim
n!1
Pr
 
j N Xn − j"

D1: (1)
1This is often abbreviated to
N Xn
P
!  as n !1
or in words: N Xn converges in probability to  as n !1 .
On account of the deﬁnition of limit and the fact that probabilities are at most 1,
Equation (1) can be rewritten as
8">08>09N>0 8nN Pr
 
j N Xn − j"

1−: (2)
The proof of the Weak Law is easy when the Xi’s have a ﬁnite variance. It is
most often based on Chebyshev’s Inequality.
2.2 Theorem (Chebyshev’s Inequality)L e t X be a random variable with ﬁnite
mean  and ﬁnite variance  2.T h e nw eh a v e
Pr.jX − ja / 
 2
a 2
for all a > 0.
A slightly different way of putting it is this: For all a > 0, we have
Pr.jX − ja / 
1
a 2 :
Thus, the probability that X deviates from its expected value by at least k standard
deviations is at most 1=k2. Chebyshev’s Inequality is sharp when no further as-
sumptions are made about X’s distribution, but for practical applications it is often
too sloppy. For example, the probability that X remains within 3 of  is at least 8
9,
no matter what distribution X has. However, when X is known to have a normal
distribution, this probability in fact exceeds 0:9986.
We now prove the Weak Law when the variance is ﬁnite. Let  2 be the variance
of each Xi. In that case, we have E N Xn D  and Var N Xn D  2=n.L e t ">0.
Substituting X;;;a:D N Xn;;=
p
n;"in Chebyshev’s Inequality then yields
Pr
 
j N Xn − j"


2
n " 2 : (3)
Hence, for >0 and for all n  maxf1;2="2g we have
Pr
 
j N Xn − j <"

> 1−
which completes the proof.
2The Central Limit Theorem
Note that  D 0 is uninteresting because in that case we have Pr.Xn D / D 1
(on account of Chebyshev’s Inequality and continuity of probability for monotonic
sequences of events).
In the case of ﬁnite non-zero variance, the Central Limit Theorem provides a
much stronger result.
2.3 Theorem (Central Limit Theorem)I f t h e X i ’shaveﬁnite non-zero variance  2,
then for all a  b,
lim
n!1Pr

a 
N Xn − 
=
p
n
 b

D 8.b/−8.a/ (4)
where 8 is the standard normal distribution deﬁned by
8.z/ D
1
p
2
Z z
−1
e−1
2x2
dx :
Convergence in (4) is uniform in a and b.
The Central Limit Theorem can be interpreted as stating that for large n, the ran-
dom variable N Xn approximately has a normal distribution with mean  and stan-
dard deviation =
p
n.
We now prove that the Central Limit Theorem implies the Weak Law of Large
Numbers when 0 <<1 . First observe that substituting a;b :D− c =;c= in
the Central Limit Theorem yields
lim
n!1
Pr

j N Xn − j
c
p
n

D 8
c


−8

−
c


: (5)
Let ">0a n d>0. Take c > 0 such that 8.−c=/  =3 (this is possible since
8.z/ ! 0a sz!− 1 )a n dt a k eNsuch that c=
p
N  " and the limit in (5) is
approached closer than =3f o ra l lnN.W ed e r i v ef o rnN(with hints placed
between braces):
Pr
 
j N Xn − j"

f monotonicity of Pr, using c=
p
n  c=
p
N  ", on account of deﬁni-
tion of N g
Pr
 
j N Xn − jc =
p
n

f deﬁnition of N g
8.c=/ − 8.−c=/ − =3
Df 8.z/C8.−z/ D 1 g
1−28.−c=/ − =3
f deﬁnition of c g
1 − 
3This concludes the proof.
If convergence to the standard normal distribution is assumed to be ‘good’
(much better than ), then we can take bound N such that
8
 "

p
N

 1 −

2
: (6)
Compare this to the bound N   2="2 on account of Chebyshev’s Inequality.
As an example, consider the case where we want to be 95% certain that the sam-
ple mean falls within 1
4 of ;t h a ti s ,D0 : 05 and " D =4. Chebyshev’s
Inequality yields N  16=0:05 D 320 and the standard normal approximation
yields
p
N=4  1:96 or N  61:47. Thus, if the standard normal approximation is
‘good’ then our need is already fulﬁlled by the mean of 62 samples, instead of the
320 required by Chebyshev’s Inequality.
I would like to emphasize the following points concerning the Central Limit The-
orem.
 There exist estimates of how closely the standard normal distribution ap-
proximates the distribution of the sample mean. Consult [Fel71, Hea71] for
the Berry–Ess´ een bound.
 If the Xi’s themselves have a normal distribution, then so does the sample
mean and the ‘approximation’ in the Central Limit Theorem is in fact exact.
 The more general versions of the Weak Law are not derivable from (more
general versions of) the Central Limit Theorem.
3 The Strong Law of Large Numbers
Let’s start again with the theorem.
3.1 Theorem (Strong Law of Large Numbers)W e h a v e
Pr

lim
n!1
N Xn D 

D 1 : (7)
This is often abbreviated to
N Xn
a:s:
!  as n !1
or in words: N Xn converges almost surely to  as n !1 .
One of the problems with such a law is the assignment of probabilities to state-
ments involving inﬁnitely many random variables. For that purpose, one needs a
careful introduction of notions like sample space, probability measure,a n drandom
variable. See for instance [Tuc67, Hea71, Chu74a, LR79].
Using some Probability Theory, the Strong Law can be rewritten into a form
with probabilities involving ﬁnitely many random variables only. Werewrite Equa-
tion (7) in a chain of equivalences:
4Pr

lim
n!1
N Xn D 

D 1
,f deﬁnition of limit g
Pr
 
8">09N>0 8nN j N Xn −j"

D1 . 8 /
,f Note 1 below g
8">0Pr
 
9N>0 8nN j N Xn − j"

D1 . 9 /
,f Note 2 below g
8">08>09N>0 Pr
 
8nN j N Xn − j"

1−. 10/
,f Note 3 below g
8">08>09N>0 8r0Pr
 
8NnNCr j N Xn − j"

1−. 11/
Comparing Equations (2) and (10) we immediately infer the Weak Law from the
Strong Law, which explains their names.
In order to supply the notes to above derivation, let .;
F; P/be an appropriate
probability space for the random variables Xi, and deﬁne events A", BN,a n dC r
for ">0, N > 0, and r  0b y
A " Df ! 2  j9 N> 08 n  NjN X n.!/ − j" g
B N Df ! 2  j8 n  NjN X n.!/ − j" g
C r Df ! 2  j8 N n  NC rjN X n.!/ − j" g:
These events satisfy the following monotonicity properties:
A"  A"0 for "  "0
BN  BNC1
Cr  CrC1 :
Therefore, on account of the continuity of probability measure P for monotonic
chains of events, we have
P.
T1
mD1 A1=m/ D lim
m!1
P.A1=m/ (12)
P.
S1
ND1 BN/ D lim
N!1
P.BN/ (13)
P.
T1
rD0 Cr/ D lim
r!1
P.Cr/: (14)
Note 1. We derive
Pr
 
8">09N>0 8nN j N Xn −j"

D1
,f deﬁnitions of Pr and A" g
P.
T
">0 A"/ D 1
,f monotonicity of A",u s i n g1 =m!0a sm!1g
P.
T 1
m D 1A 1 = m/D1
,f (12) g
5lim
m!1 P.A1=m/ D 1
,f property of limits, using that P.A1=m/ is descending and at most 1 g
8m>0 P.A1=m/ D 1
,f see ﬁrst two steps, also using monotonicity of P g
8">0Pr
 
9N>0 8nN j N Xn − j"

D1
Note 2. We derive for ">0
Pr
 
9N>0 8nN j N Xn − j"

D1
,f deﬁnitions of Pr and BN, and set theory g
P.
S1
ND1 BN/ D 1
,f (13) g
lim
N!1
P.BN/ D 1
,f deﬁnition of limit, using P.Bk/  1 g
8>09N>0 8kN P.Bk/  1−
,f monotonicity of P,u s i n gB kB Nfor k  N g
8>09N>0 P.BN/  1−
,f deﬁnitions of Pr and BN g
8>09N>0 Pr
 
8nN j N Xn − j"

1−
Note 3. We derive for ">0, >0, and N > 0
Pr
 
8nN j N Xn − j"

1−
,f deﬁnitions of Pr and Cr, and set theory g
P.
T
r0 Cr/  1 − 
,f (14) g
lim
r!1
P.Cr/  1 − 
,f property of limits, using that P.Cr/ is descending g
8r0 P.Cr/  1 − 
,f deﬁnitions of Pr and Cr g
8r0 Pr
 
8NnNCr j N Xn − j"

1−
Quotes from the literature
I have not been able to ﬁnd a reference that explicitly presents the preceding chain
of equivalent expressions for the Strong Law ([Chu74a, Ch. 4] comes close). Many
authors take one of these expression as deﬁnition. Below are some typical quotes
to illustrate the state of affairs. Note that each of these quotes contains a partly
verbal expression, which in some cases is even ambiguous as to the order of the
quantiﬁers.
The paraphrasing of the Strong Law in [Ros89, p. 351] resembles Equation (8),
though it is also possible to read it as Equation (9):
6“In particular, [the Strong Law] shows that, with probability 1, for any
positive value ",

 
 
n X
iD1
Xi
n
− 

 
 
will be greater than " only a ﬁnite number of times.”
Equation (10) can be recognized in [Hea71, p. 226]:
“Indeed for arbitrarily small ">0, >0, and large N D N.";/,...
the [deﬁnition] of Xn
a:s:
! X ... can be restated ... as
P
 
1 \
nDN
f! jjX n.!/ − X.!/j <" g
!
>1−
...”
Equation (11) resembles the deﬁnition in [Fel68, p. 259]:
“We say that the sequence Xk obeys the strong law of large numbers
if to every pair >0, >0, there corresponds an N such that there
is probability 1− or better that for every r > 0a l lrC1 inequalities
jSn − mnj
n
< ; nDN;NC1 ;:::;NCr
will be satisﬁed.”
Proofs of the Strong Law
Again the case with ﬁnite variance is easier than the general case. Most of the
proofs that I have encountered for the Strong Law assuming ﬁnite variance are
based on Kolmogorov’s Inequality, which is a generalization of Chebyshev’s In-
equality. Even in that case there are still some technical hurdles (I will not go into
these). Consequently, the proofs do not give rise to an explicit bound N in (11)
in terms of " and . An exception is [Eis69], where the Hajek–R´ enyi Inequality is
used, which is a generalization of Kolmogorov’s Inequality.
There is, however, a nice overview article, namely [HR80], that speciﬁcally
looks at bounds N in terms of " and . It shows, among other things, that
Pr
 
9nN j N Xn − j"


2
N" 2 : (15)
Compare this result to (3): it is the same upper bound but for a much larger event.
This creates the impression that the Weak Law is not that much weaker than the
Strong Law.
74 Concluding Remarks
We have looked at one special case to clarify the relationship between the Weak
and the Strong Law of Large Numbers. The case was special in that we have
assumed Xi to be a sequence of independent and identically distributed random
variables with ﬁnite expectation and that we have considered the convergence of
partial sample means to the common expectation. Historically, it was preceded
by more special cases, for instance, with the Xi restricted to Bernoulli variables.
Nowadays these laws are treated in much more generality.
Mymaininterest wasfocused onEquations (8) through (11), which areequivalent—
but subtly different—formulations of the Strong Law of Large Numbers. Further-
more, I have looked at “constructive” bounds related to the rate of convergence.
Let me conclude with some sobering quotes (also to be found in [Chu74b,
p. 233]). Feller writes in [Fel68, p. 152]:
“[Theweak lawoflarge numbers] isofverylimited interest andshould
be replaced by the more precise and more useful strong law of large
numbers.”
In [Wae71, p. 98], van der Waerden writes:
“[The strong law of large numbers] scarcely plays a role in mathemat-
ical statistics.”
Acknowledgment I would like to thank Fred Steutel for discussing the issues
in this paper and for pointing out reference [HR80].
References
[Chu74a] Kai Lai Chung. A Course in Probability Theory. Academic Press, sec-
ond edition, 1974.
[Chu74b] Kai Lai Chung. Elementary Probability Theory and Stochastic
Processes. Springer, 1974.
[Eis69] Marin Eisen. Introduction to Mathematical Probability Theory.
Prentice-Hall, 1969.
[Fel68] William Feller. An Introduction to Probability Theory and Its Applica-
tions, volume I. Wiley, third edition, 1968.
[Fel71] William Feller. An Introduction to Probability Theory and Its Applica-
tions, volume II. Wiley, second edition, 1971.
[Hea71] C. R. Heathcote. Probability: Elements of the Mathematical Theory.
Unwin, 1971.
8[HR80] A. H. Hoekstra and J. T. Runnenburg. Inequalities compared. Statistica
Neerlandica, 34(2):67–82, 1980.
[LR79] R. G. Laha and V. K. Rohatgi. Probability Theory. Wiley, 1979.
[Ros89] Sheldon Ross. A First Course in Probability. Macmillan, third edition,
1989.
[Tuc67] Howard G. Tucker. A Graduate Course in Probability. Academic Press,
1967.
[Wae71] B. L.Waerden, van der. Mathematische Statistik. Springer, third edition,
1971.
9

The Laws of Large Numbers Compared

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics