Predrag Jelenkovic

English

Research Papers in Economics

Independent random variables
E6711: Lectures 3
Prof. Predrag Jelenkovi´ c
1 Last two lectures
￿ probability spaces
￿ probability measure
￿ random variables and stochastic processes
￿ distribution functions
￿ independence
￿ conditional probability
￿ memoriless property of geometric and exponential distributions
￿ expectation
￿ conditional expectation (double expectation)
￿ mean-square estimation
1Let
f
X
j
;
j
￿
1
g be a sequence of independent random variables
and
N
n
=
n
X
j
=
1
X
j
be a partial sum of the ﬁrst
n of these r.v.s. In many applications un-
derstanding the statistical behavior of these sums is very important.
Thus, a big part of probability theory studies the characteristics of
N
n.
In this lecture we review some of the well-known theorems of
probability theory:
￿ Markov and Chebyshev’s inequalities
￿ Laws of Large Numbers
￿ Central Limit Theorem
2 Inequalities
Proposition 2.1 (Markov’s inequality) If
X is a nonnegative ran-
dom variable, then for any
a
>
0
P
[
X
￿
a
]
￿
E
X
a
:
Proof: For
a
>
0, let us deﬁne an indicator function
1
[
X
￿
a
]
=
￿
1
i
f
X
￿
a
0
o
t
h
e
r
w
i
s
e
:
Then,
1
[
X
￿
a
]
￿
X
a
;
2thus, by taking the expected value on both sides in the preceding
inequality we obtain
E
1
[
X
￿
a
]
=
P
[
X
￿
a
]
￿
E
X
a
:
3
Corollary 2.1 (Chebyshev’s inequality) If
X is a random variable
with ﬁnite mean
￿ and variance
￿
2
=
E
(
X
￿
￿
)
2, then for any
￿
>
0
P
[
j
X
￿
￿
j
￿
￿
]
￿
￿
2
￿
2
:
Proof: Let
Y
=
(
X
￿
￿
)
2, then
P
[
j
X
￿
￿
j
￿
￿
]
=
P
[
(
X
￿
￿
)
2
￿
￿
2
]
=
P
[
Y
￿
￿
2
]
￿
E
Y
￿
2
=
￿
2
￿
2
;
where the last inequality follows from Markov’s inequality.
3
Corollary 2.2 (Chernoff’s bound) Let
M
(
t
)
d
e
f
=
E
e
t
X
<
1forsome
t
>
0, then
P
[
X
￿
y
]
￿
e
￿
t
y
M
(
t
)
:
Proof: Let
Y
=
e
t
X and
a
=
e
t
y, then
P
[
X
￿
y
]
=
P
[
t
X
￿
t
y
]
=
P
[
e
t
X
￿
e
t
y
]
=
P
[
Y
￿
a
]
￿
E
Y
a
=
e
￿
t
y
M
(
t
)
;
3note that the last inequality follows from Markov’s inequality.
3
3 Laws of Large Numbers: ergodic theorems
Ergodic theory studies the conditions under which the sample path
average
Y
d
e
f
=
X
1
+
￿
￿
￿
+
X
n
n
(3.1)
converges to the mean
￿
=
E
X
1 as
n
!
1.
Theorem 3.1 (Weak Law of Large Numbers) Let
X
1
;
X
2
;
:
:
:,b ea
sequence of independent random variables with ﬁnite mean
￿ and
variance
￿
2. Then, for any
￿
>
0
P
￿
￿
￿
￿
￿
X
1
+
X
2
+
￿
￿
￿
+
X
n
n
￿
￿
￿
￿
￿
￿
￿
￿
￿
!
0
a
s
n
!
1
:
Proof: Recall the deﬁnition of
Y from equation (3.1), then
E
Y
=
E
￿
X
1
+
X
2
+
￿
￿
￿
+
X
n
n
￿
=
￿
and
V
a
r
(
Y
)
=
V
a
r
￿
X
1
+
X
2
+
￿
￿
￿
+
X
n
n
￿
=
V
a
r
(
X
1
)
+
￿
￿
￿
+
V
a
r
(
X
n
)
n
2
=
￿
2
n
Thus, by Chebyshev’s inequality
P
[
j
Y
￿
￿
j
￿
￿
]
￿
￿
2
n
￿
2
!
0
a
s
n
!
1
;
4we conclude the proof of the theorem.
3
Now we know that
P
￿
￿
￿
￿
￿
X
1
+
X
2
+
￿
￿
￿
+
X
n
n
￿
￿
￿
￿
￿
￿
￿
converges to zero, however it is not clear how fast? This problem is
investigated by the theory of Large Deviations.
Recall
M
(
t
)
=
E
e
t
X
1 and deﬁne the rate function
l
(
a
)
d
e
f
=
￿
l
o
g
￿
i
n
f
t
￿
0
e
￿
t
a
M
(
t
)
￿
=
s
u
p
t
￿
0
(
t
a
￿
l
o
g
M
(
t
)
)
:
Then
Theorem 3.2 For every
a
>
E
X
1 and
n
￿
1
P
￿
X
1
+
X
2
+
￿
￿
￿
+
X
n
n
￿
a
￿
￿
e
￿
n
l
(
a
)
:
Proof: For any
t
>
0
P
￿
X
1
+
X
2
+
￿
￿
￿
+
X
n
n
￿
a
￿
=
P
h
e
t
(
X
1
+
X
2
+
￿
￿
￿
+
X
n
)
￿
e
t
a
n
i
(Chernoff’s inequality)
￿
e
￿
t
a
n
E
e
t
(
X
1
+
X
2
+
￿
￿
￿
+
X
n
)
=
￿
e
￿
t
a
E
e
t
X
1
￿
n
:
Thus,
P
￿
X
1
+
X
2
+
￿
￿
￿
+
X
n
n
￿
a
￿
￿
i
n
f
t
￿
0
￿
e
￿
t
a
E
e
t
X
1
￿
n
=
e
￿
n
l
(
a
)
:
3
5The preceding two theorems estimate the probabilities that a sam-
ple path mean is close to the (ensemble) mean. The following the-
orem goes one step further in showing that for almost every ﬁxed
omega the sample path average converges to the mean (in the ordi-
nary deterministic sense).
Theorem 3.3 (Strong Law of Large Numbers) Let
X
1
;
X
2
;
:
:
:,b e
a sequence of independent random variables with ﬁnite mean
￿ and
K
d
e
f
=
E
X
4
1
<
1. Then, for almost every
! (or with probability 1)
X
1
+
X
2
+
￿
￿
￿
+
X
n
n
!
￿
a
s
n
!
1
:
Remark: For this theorem to hold it is enough to assume that the
mean
￿
=
E
X
1 exists (i.e., it could be even inﬁnite). However, in
order to present a simpler proof, we impose a stronger assumption
E
X
4
1
<
1.
Proof: To begin, assume that
￿
=
E
X
j
=
0; then
E
N
4
n
=
E
[
(
X
1
+
￿
￿
￿
+
X
n
)
(
X
1
+
￿
￿
￿
+
X
n
)
(
X
1
+
￿
￿
￿
+
X
n
)
(
X
1
+
￿
￿
￿
+
X
n
)
]
:
Now, expanding the right-hand side of the equation above will result
in terms of the form (
i
6
=
j
6
=
k)
E
X
4
i
E
[
X
3
i
X
j
]
=
E
X
3
i
E
X
j
=
0 by independence
E
X
2
i
X
2
j
E
[
X
2
i
X
j
X
k
]
=
E
X
2
i
E
X
j
E
X
k
=
0 by independence
E
[
X
i
X
j
X
k
X
l
]
=
E
X
i
E
X
j
E
X
k
E
X
l
=
0 by independence
:
6Next, there are
n terms of the form
E
X
4
i and for each
i
6
=
j there are
￿
4
2
￿
=
6 terms in the expansion that are equal to
E
X
2
i
X
2
j. Hence,
E
N
4
n
=
n
E
X
4
1
+
6
￿
n
2
￿
(
E
X
2
1
)
2
=
n
K
+
3
n
(
n
￿
1
)
(
E
X
2
1
)
2
: (3.2)
Also,
K
<
1 implies
E
X
2
1
<
1, since
0
￿
V
a
r
(
X
2
1
)
=
E
X
4
1
￿
(
E
X
2
1
)
2
)
(
E
X
2
1
)
2
￿
K (3.3)
Now, by replacing (3.3) in (3.2), we obtain
E
N
4
n
n
4
￿
K
n
3
+
3
K
n
2
￿
4
K
n
2
:
Thus,
E
1
X
n
=
1
N
4
n
n
4
=
1
X
n
=
1
E
N
4
n
n
4
￿
1
X
n
=
1
4
K
n
2
<
1
:
Therefore, with probability 1
1
X
n
=
1
N
4
n
n
4
<
1
;
which implies that, with probability 1
l
i
m
n
!
1
N
4
n
n
4
=
0
;
or equivalently
P
￿
l
i
m
n
!
1
N
n
n
=
0
￿
=
1
:
This concludes the proof of the case
￿
=
0.I f
￿
6
=
0, then deﬁne
X
0
j
=
X
j
￿
E
X
j and use the same proof.
3
74 Central Limit Theorem
Central Limit Theorem, Similarly to the Large Deviation Theorem,
measures the deviation of the sample mean from the expected value
￿.
Theorem 4.1 (Cental Limit Theorem (CLT)) Let
X
j
;
j
￿
1 be a
sequence of i.i.d. r.v.s with mean
￿ and variance
￿
2
<
1. Then, the
distribution of
Z
n
d
e
f
=
X
1
+
￿
￿
￿
+
X
n
￿
n
￿
￿
p
n
tends to standard normal distribution as
n
!
1, i.e., for any real
number
a
P
￿
X
1
+
￿
￿
￿
X
n
￿
n
￿
￿
p
n
￿
a
￿
!
1
p
2
￿
Z
a
￿
1
e
￿
x
2
=
2
d
x
a
s
n
!
1
:
First, we state the following key lemma that will be used in the
proof of CLT.
Lemma 4.1 Let
Z
1
;
Z
2
;
:
:
:, be a sequenceof r.v.shavingdistribution
functions
F
Z
n andmomentgeneratingfunctions
M
Z
n
(
t
)
=
E
e
t
Z
n
;
n
￿
1; And let
Z be a random variable having distribution
F
Z and mo-
ment generating function
M
Z
(
t
).I t
M
Z
n
(
t
)
!
M
Z
(
t
) as
n
!
1,
for all
t, then
F
Z
n
(
x
)
!
F
Z
(
x
)
a
s
x
!
1
:
Proof: Omitted.
3
8Proof of CLT: Assume that
￿
=
0 and
￿
2
=
1. Then, moment
generating function (m.g.f.) of
X
j
=
p
n is equal to
E
h
e
t
X
j
=
p
n
i
=
M
(
t
=
p
n
) where
M
(
t
)
=
E
e
t
X
j
:
Thus, the m.g.f. of
P
n
j
=
1
X
j
=
p
n is equal to
￿
M
￿
t
p
n
￿
￿
n
:
Now, if
L
(
t
)
d
e
f
=
l
o
g
M
(
t
), then
l
o
g
￿
M
￿
t
p
n
￿
￿
n
=
n
L
￿
t
p
n
￿
=
L
(
t
=
p
n
)
n
￿
1
:
Thus
l
i
m
n
!
1
L
(
t
=
p
n
)
n
￿
1
=
l
i
m
n
!
1
￿
L
0
(
t
=
p
n
)
n
￿
3
=
2
t
￿
2
n
￿
2 (by L’Hospital’s rule)
=
l
i
m
n
!
1
￿
L
0
(
t
=
p
n
)
t
￿
2
n
￿
1
=
2
=
l
i
m
n
!
1
￿
L
0
0
(
t
=
p
n
)
n
￿
3
=
2
t
2
￿
2
n
￿
3
=
2 (by L’Hospital’s rule)
=
l
i
m
n
!
1
L
0
0
(
t
=
p
n
)
t
2
2
Next, note that
L
(
0
)
=
0
L
0
(
0
)
=
M
0
(
0
)
M
(
0
)
=
￿
=
0
L
0
0
(
0
)
=
M
(
0
)
M
0
0
(
0
)
￿
(
M
0
(
0
)
)
2
(
M
(
0
)
)
2
=
E
X
2
=
1
:
Hence, for any ﬁnite
t
l
i
m
n
!
1
L
0
0
(
t
=
p
n
)
=
1
9and, therefore
l
i
m
n
!
1
n
L
(
t
=
p
n
)
=
t
2
2
;
or, equivalently
l
i
m
n
!
1
￿
M
￿
t
p
n
￿
￿
n
=
e
t
2
=
2
:
On the other hand, if
Z is a standard normal r.v., then
E
e
t
N
=
e
t
2
=
2
;
which, by Lemma 4.1, concludes the proof of the theorem for
￿
=
0
and
￿
2
=
1.
For the general case
￿
6
=
0 and
￿
2
6
=
1, we can introduce new
variables
X
￿
j
d
e
f
=
X
j
￿
￿
￿
;
clearly
E
X
￿
j
=
0
a
n
d
V
a
r
(
X
￿
j
)
=
1
;
and, therefore, we can use the already proved case.
3
10

Independent Random Variables

http://www.dklevine.com/archive/strong-law-proof.pdf

Independent Random Variables

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics