Robinson (1951) showed that the learning process of Discrete Fictitious Play converges from any initial condition to the set of Nash equilibria in two-player zero-sum games. In several earlier works, Brown (1949, 1951) makes some heuristic arguments for a similar convergence result for the case of Continuous Fictitious Play (CFP). The standard reference for a formal proof is Harris (1998); his argument requires several technical lemmas, and moreover, involves the advanced machinery of Lyapunov functions. In this note we present a simple alternative proof. In particular, we show that Brown''s convergence result follows easily from a result obtained by Monderer et al. (1997).mathematical economics;

Driesen Bram

English

Research Papers in Economics

Bram Driesen 
 
Continuous fictitious play in zero-
sum games 
 
RM/09/049 
 
 Continuous Fictitious Play in Zero-Sum
Games
Bram Driesen∗
This version, October 2009
Abstract
Robinson (1951) showed that the learning process of Discrete Ficti-
tious Play converges to Nash equilibrium in two-player zero-sum games for
any initial condition. In several earlier works, Brown (1949, 1951) makes
some heuristic arguments for a similar convergence result for the case of
Continuous Fictitious Play (CFP). The standard reference for a formal
proof is Harris (1998); his argument requires several technical lemmas,
and moreover, involves the advanced machinery of Lyapunov functions.
In this note we present a simple alternative proof. In particular, we show
that Brown’s convergence result follows easily from a result obtained by
Monderer et al. (1997).
JEL-Classiﬁcation: C72, D83
Keywords: Fictitious play; Zero-sum games.
1 Introduction
Fictitious Play is a procedure in which at each instance, players of a game my-
opically play their best replies against the opponents’ past play. The origins of
Fictitious Play lie in a series of papers by Brown (1949, 1951) and Robinson
(1951). While today, the algorithm is usually interpreted as a myopic learning
process, the initial objective of these studies was to develop an easy method to
ﬁnd the value of zero-sum games. In his 1949 article, Brown already argued
heuristically that the continuous version of the procedure – Continuous Ficti-
tious Play (CFP) – must converge at a linear speed to the set of Nash equilibria,
and thus, the value of the game. This result is again mentioned – without proof
– in Brown (1951). Brown’s to a large extent heuristic approach towards CFP
may be explained by the fact that he was primarily interested in a discrete
algorithm, and thus, only considered convergence of CFP in zero-sum games
∗Department of Quantitative Economics, University of Maastricht, P.O. Box 616, 6200 MD
Maastricht, The Netherlands. Telephone: +31-43-3883835. Telefax: +31-43-3884874. Email
adress: B.Driesen@maastrichtuniversity.nl
1as support for his conjecture that Discrete Fictitious Play (DFP) in zero-sum
games converges to equilibrium.
The proof of this conjecture was obtained by Robinson (1951) in the early
ﬁfties. However, it was not until Hofbauer (1994) and Harris (1998) that the
result for CFP was rigorously proven. Today, Harris’ proof has become the
standard reference. Essentially, he shows that the sum of players’ instantaneous
improvement steps in a zero-sum game, is a Lyapunov function. While this
argument appears to be quite simple at ﬁrst glance, it requires several rather
technical lemmas1. Moreover, it makes use of non-trivial results on dynamical
systems. Diﬀerent versions of Harris’ proof – often in a simpliﬁed form – have
appeared in the literature. See a.o. Krishna and Sj¨ ostr¨ om (1997), Shamma and
Arslan (2004), and Hofbauer and Sorin (2006).
In this note we provide a simple alternative argument. We show that the
convergence of CFP in two-player zero-sum games follows trivially from a re-
sult obtained by Monderer et al. (1997) that says that for CFP in any game,
each player’s instantaneous expected payoﬀ will in the long run coincide with
the average payoﬀ that player has realized so far. Using the same approach,
convergence of DFP is established up to the condition of infrequent switching
(Fudenberg and Levine, 1994; Monderer et al. 1997). That is, if for DFP in
a zero-sum game players play each pure strategy proﬁle for increasingly long
periods of time, then the DFP converges to equilibrium. Simulations lead us
to conjecture that infrequent switching is the engine behind Robinson’s result2.
Whether DFP in zero-sum games satisﬁes the condition of infrequent switching
in general, remains an open question.
2 Preliminaries
2.1 Zero-sum Games
A zero-sum game is represented as an m × n matrix A. There are two players
– 1 and 2. Player 1 [2] has a pure strategy set I [J] of cardinality m [n]. If
player 1 plays strategy i and 2 plays j, then player 1 receives a payoﬀ of aij
and 2 of −aij. Let ∆m and ∆n be the sets of probability distributions over sets
of respectively m and n alternatives; ∆m and ∆n are players’ respective sets of
mixed strategies over I and J. A pure strategy i ∈ I is also denoted as the unit
vector ei ∈ ∆m; we write ˜ I := {ei | i ∈ I}. Analogously, we denote player 2’s
j-th pure strategy also as the j-th unit vector fj ∈ ∆n, and ˜ J := {fj | j ∈ J}.
Let β1 : ∆n → ∆m be player 1’s best-reply correspondence. That is,
β1(ˆ q) := {p ∈ ∆m | pAˆ q ≥ p′Aˆ q for all p′ ∈ ∆m}.
Player 2’s best-reply correspondence, β2 : ∆m → ∆n is deﬁned as
β2(ˆ p) := {q ∈ ∆n | ˆ pAq ≤ ˆ pAq′ for all q′ ∈ ∆n}.
1See also Hofbauer and Sorin (2006).
2Note that convergence of CFP implies convergence of DFP for a class of games that
includes zero-sum games. See Harris (1998) and Hofbauer and Sorin (2006).
2Deﬁne β : ∆m × ∆n → ∆m × ∆n as β(p,q) := (β1(q),β2(p)). The set of Nash
equilibria of a game A is deﬁned as
NE(A) := {(p,q) ∈ ∆m × ∆n | (p,q) ∈ β(p,q)}.
That is, NE(A) is the set of ﬁxed points of the best-reply correspondence β.
Deﬁne the function W : ∆m × ∆n → R as
W(ˆ p, ˆ q) := H(ˆ q) − L(ˆ p),
where H(ˆ q) := maxp∈∆m pAˆ q and L(ˆ p) := minq∈∆n ˆ pAq.
Theorem 2.1 W(p,q) ≥ 0, with equality if and only if (p,q) ∈ NE(A).
This is a classic result that follows from von Neumann’s (1928) maximin
theorem. For an exact proof, see a.o. Myerson (1991).
2.2 Continuous Fictitious Play
A Continuous Fictitious Play (CFP) in A is a pair (x(t),y(t)) of Lebesgue
measurable functions x : [0,∞) → ˜ I and y : [0,∞) → ˜ J, such that for almost
all t ≥ 1, we have
(x(t),y(t)) ∈ β
￿
1
t
Z t
0
x(τ)dτ,
1
t
Z t
0
y(τ)dτ
￿
.
Players’ beliefs (p(t),q(t)) are deﬁned as functions
p(t) :=
1
t
Z t
0
x(τ)dτ, and q(t) :=
1
t
Z t
0
y(τ)dτ
on the domain [1,∞). Furthermore, we deﬁne the set Ω of limit points of
(p(t),q(t)) as
Ω :=
\
T≥1
cl({(p(t),q(t)) | t ≥ T}),
where cl denotes the topological closure. We say (p,q) converges to Nash equilib-
rium if Ω ⊆ NE(A,B). Expected and average payoﬀs are deﬁned for t ∈ [1,∞)
as
E1(t) := x(t)Aq(t), and E2(t) := −p(t)Ay(t),
and
P1(t) :=
1
t
Z t
0
x(τ)Ay(τ)dτ, and P2(t) := −P1(t).
Monderer et al. (1997) introduce the concept of belief aﬃrmation; a CFP is
said to be belief aﬃrming if
lim
t→∞
(Ei(t) − Pi(t)) = 0
for i = 1,2. In a belief aﬃrming process players’ expected payoﬀs and average
realized payoﬀs coincide in the long run.
33 CFP Convergence in Zero-Sum Games
The aim of this note is to prove the following.
Theorem 3.1 CFP in zero-sum games converges to Nash equilibrium.
The proof is divided into three simple lemmas. The ﬁrst is due to Monderer
et al. (1997). For completeness, we include the proof.
Lemma 3.2 (Monderer et al. 1997) Every CFP is belief aﬃrming.
Proof. Each function fi : [1,∞) → R, i ∈ I, deﬁned as
fi(t) :=
Z t
0
eiAy(τ)dτ,
is continuous on [1,∞). Since tE1(t) = max{fi(t) | i ∈ I} for all t ≥ 1, also
tE1(t) is continuous on [1,∞). The function tP1(t) is also continuous on [1,∞);
it follows that tE1(t) − tP1(t) is continuous on [1,∞).
Suppose (x( ),y( )) is continuous at t. Since x( ) and y( ) are by deﬁnition
pure strategies, there is an ε > 0 such that x(τ) and y(τ) are constant for all
τ ∈ (t − ε,t + ε). This implies that the derivatives of tE1(t) and tP1(t) at t
are both equal to x(t)Ay(t). It follows that tE1(t) − tP1(t) is constant on each
interval on which (x( ),y( )) is continuous. Continuity of tE1(t) − tP1(t) on
[1,∞) then implies
tE1(t) − tP1(t) = K,
for all t ∈ [1,∞), where K := E1(1) − P1(1). Thus,
lim
t→∞
(E1(t) − P1(t)) = lim
t→∞
1
t
(tE1(t) − tP1(t)) = lim
t→∞
K
t
= 0. (1)
A similar argument applies to player 2.
Lemma 3.3 limt→∞ W(p(t),q(t)) = 0.
Proof. It follows from Lemma 3.2 that CFP is belief aﬃrming. This implies
0 = lim
t→∞(E1(t) − P1(t)) + lim
t→∞(E2(t) − P2(t))
= lim
t→∞
(E1(t) − P1(t) + E2(t) − P2(t))
= lim
t→∞
(E1(t) + E2(t))
= lim
t→∞
W(p(t),q(t)),
4where the third equality follows from P1(t) = −P2(t) for all t, and the fourth
from W(p(t),q(t)) = E1(t) + E2(t) for all t.
Proof of Theorem 3.1 Take some (p∗,q∗) ∈ Ω and a sequence of times
(tk)∞
k=1 such that limk→∞(p(tk),q(tk)) = (p∗,q∗). Note that the existence of
such a sequence is implied by (p∗,q∗) ∈ Ω. Since W is continuous in (p,q) we
have
lim
k→∞
W(p(tk),q(tk)) = W(p∗,q∗).
By Lemma 3.3 it follows that W(p∗,q∗) = 0, which by Theorem 2.1 implies
(p∗,q∗) ∈ NE(A). Hence, Ω ⊆ NE(A).
Remark 3.5 From Equation (1) it follows that (E1(t) − P1(t)) converges
at rate 1
t; this also holds for player 2. It follows that (p(t),q(t)) converges to
equilibrium at a rate of 1
t, conﬁrming Harris’ (1998) result.
Remark 3.6 Our proof of Theorem 3.1 seems to be closer to Brown’s
heuristic argument than the one usually presented in the literature. For in-
stance, Brown (1951, pp. 375) says:3 “In the system of diﬀerential equations
the convergence rests on the fact that tV (t) and tV (t) maintain a constant dif-
ference between them.”
4 DFP in Zero-Sum Games
A Discrete Fictitious Play (DFP) in A is a sequence (x(t),y(t)) in ˜ I × ˜ J, t ∈ N,
with (x(0),y(0)) ∈ ˜ I × ˜ J and for all t ∈ N \ {0},
(x(t),y(t)) ∈ β
 
1
t
t−1 X
τ=0
(x(τ),y(τ))
!
.
The sequence of beliefs (p(t),q(t)) : N \ {0} → ∆m × ∆n, is given by
1
t
t−1 X
τ=0
(x(τ),y(τ)).
Like before, we deﬁne E1(t) := x(t)Aq(t), E2(t) := −p(t)Ay(t),
P1(t) :=
1
t
t−1 X
τ=0
x(τ)Ay(τ), and P2(t) := −P1(t).
The following result was proven by Robinson (1951).
3Here, V (t) ≡ E1(t) and V (t) ≡ −E2(t).
5Theorem 4.1 (Robinson 1951) DFP in zero-sum games converges to Nash
equilibrium.
A DFP satisﬁes the condition of infrequent switching4 if
lim
t→∞
1
t
t X
τ=1
Mi(τ) = 0,
for i = 1,2, where
M1(t) :=
(
1 if x(t)  = x(t − 1)
0 otherwise.
and M2(t) :=
(
1 if y(t)  = y(t − 1)
0 otherwise.
for all t ∈ N \ {0}. Monderer et al. (1997), and Fudenberg and Levine (1994)
established the following result.
Lemma 4.2 Every DFP satisfying infrequent switching is belief aﬃrming.
Hence, if the DFP process satisﬁes infrequent switching, then its convergence
in zero-sum games follows along the lines of the previous section. Whether DFP
in zero-sum games satisﬁes infrequent switching remains an open question.
References
[1] Brown, G. W. (1949). Some notes on computation of games solutions.
RAND report P–78, The RAND Corporation, Santa Monica, California.
[2] Brown, G. W. (1951). Iterative solution of games by ﬁctitious play. In T. C.
koopmans (Ed.), Activity Analysis of Production and Allocation, Chap-
ter 24, pp. 374–376. Wiley, New York.
[3] Fudenberg, D. and D. K. Levine (1995). Consistency and cautious ﬁctitious
play. Journal of Economic Dynamics and Control 19(4), 1065–1089.
[4] Harris, C. (1998). On the rate of convergence of continuous-time ﬁctitious
play. Games and Economic Behavior 22(2), 238–259.
[5] Hofbauer, J. and S. Sorin (2006). Best-response dynamics for continuous
zero-sum games. Discrete and Dynamical Systems–Series B 6(1), 215–224.
[6] Krishna, V. and T. Sj¨ ostr¨ om (1997). In S. Hart and A. Mas-Colell (Eds.),
Cooperation: Game-theoretic Approaches, Volume 55 of NATO-ASI Series
F, pp. 257–273. Springer, Berlin.
[7] Monderer, D., D. Samet, and A. Sela (1997). Belief aﬃrming in learning
processes. Journal of Economic Theory 73(2), 438–452.
4Monderer et al. (1997) call DFP’s that satisfy infrequent switching “smooth” Fictitious
Plays.
6[8] Myerson, R. B. (1991). Game theory: Analysis of conﬂict. Harvard Uni-
versity Press, Cambridge, Massachusetts.
[9] von Neumann, J. (1928). Zur theorie der gesellschaftsspiele. Mathematische
Annalen 100(1), 295–320.
[10] Robinson, J. (1951). An interative method of solving a game. Annals of
Mathematics 54(2), 296–301.
[11] Shamma, J. S. and G. Arslan (2004). Uniﬁed convergence proofs of
continuous-time ﬁctitious play. IEEE Transactions on Automatic Con-
trol 49(7), 1137–1142.
7

An interative method of solving a game.

Belief aﬃrming in learning processes.

Best-response dynamics for continuous zero-sum games.

Consistency and cautious ﬁctitious play.

Iterative solution of games by ﬁctitious play.

On the rate of convergence of continuous-time ﬁctitious play.

Sj¨ ostr¨ om

Some notes on computation of games solutions. RAND report P–78, The RAND Corporation,

Uniﬁed convergence proofs of continuous-time ﬁctitious play.

Zur theorie der gesellschaftsspiele.

Continuous fictitious play in zero-sum games

http://edocs.ub.unimaas.nl/loader/file.asp?id=1454

Continuous fictitious play in zero-sum games

Abstract

Similar works

Full text

Available Versions

Research Papers in Economics