Abstract Asymptotic properties of optimal strategies for two-armed bandit processes with geometrically distributed survival times are derived. These results provide asymptotic boundary conditions and further extend structure properties of optimal strategies for bandit processes with delayed responses

Xikui Wang

CiteSeerX

Statistics & Probability Letters 60 (2002) 211–217
Asymptotic properties of bandit processes with
geometric responses
Xikui Wang∗
Department of Statistics, University of Manitoba, Winnipeg, Man., Canada R3T 2N2
Received October 2001; received in revised form July 2002
Abstract
Asymptotic properties of optimal strategies for two-armed bandit processes with geometrically distributed
survival times are derived. These results provide asymptotic boundary conditions and further extend structure
properties of optimal strategies for bandit processes with delayed responses.
c© 2002 Elsevier Science B.V. All rights reserved.
MSC: 62L05; 62L15
Keywords: Bandit processes; Clinical trial; Dynamic programming; Geometric survival time; Optimal strategy
1. Introduction
Bandit processes with geometrically distributed survival times are studied in Eick (1988) and
Wang (2000). Bandit models have been proposed as alternative adaptive designs of clinical trials
when the traditional randomized designs become ethically infeasible in desperate medical situations
(Pullman and Wang, 2001). The monograph by Berry and Fristedt (1985) is an excellent introduction
to the subject of bandit problems.
Assume two treatments x and y for a common disease. Observations of patients’ survival times
after treatments may be censored. Under treatment y, patients’ survival times Y have a known
expected value k ¿ 1. Patients’ survival times X on the unknown treatment x are conditionally
independent and geometrically distributed with an unknown probability of success ∈ (0; 1). At times
0; 1; 2; : : : ; patients are recruited into the trial sequentially and treated one at a time. Our objective
is to sequentially allocate treatments to patients so as to maximize the total expected discounted
survival times for all patients.
∗ Corresponding author. Fax: +1-204-474-7621.
E-mail address: xikui wang@umanitoba.ca (X. Wang).
0167-7152/02/$ - see front matter c© 2002 Elsevier Science B.V. All rights reserved.
PII: S0167-7152(02)00319-X
212 X. Wang / Statistics & Probability Letters 60 (2002) 211–217
Under some regularity conditions, Eick (1988) characterizes the optimal strategy by break-even
values of parameters. Unfortunately these values are formidable computationally, even in very simple
situations. Wang (2000) initiates a study of structural properties of these break-even values, which
could potentially lead to eFcient computations and simulations.
The purpose of this paper is to explore further structural properties of these break-even values.
Main results in Eick (1988) and Wang (2000) are summaried in Section 2. New results and proofs
are presented in Section 3.
2. The model and background
Assume that the unknown probability of success ∈ (0; 1) follows a prior distribution 
. SuFcient
statistics consist of s and f, which are, respectively, the observed numbers of successes and failures
under the unknown treatment x at the current time point. The posterior distribution for  is then of
the form (s; f)
, with (0; 0)
=
. Under the currently updated prior (s; f)
, the conditional expected
survival time under the unknown treatment is denoted as E(X |(s; f)
).
A strategy  is a sequence of rules specifying a treatment to be allocated to the patient at
time 0; 1; 2; 3; : : : ; given current information. If Zi denotes the ith patient’s survival time under a
strategy , then the worth of the strategy  is deJned as the expected total discounted survival
time
W () = E
( ∞∑
i=1
iZi
)
;
where D = (1; 2; : : :) is a discount sequence satisfying i¿ 0 and
∑∞
i=1 i ¡∞. The goal is to
Jnd an optimal strategy ∗ such that W (∗) = V =max W (). A treatment is optimal for a given
patient if it is allocated by an optimal strategy.
Due to the loss-of-memory property of the geometric distribution, the bandit process with de-
layed responses becomes a discrete time Markov decision process. The state is characterized by
((s; f)
; r; D), where r is the number of patients previously treated with the unknown treatment
x who are still alive. Eick (1988) calls r the size of the information bank. The action space
is {1; 2}.
At each state ((s; f)
; r; D), let V (x)((s; f)
; r; D) (V (y)((s; f)
; r; D)) be the worth of the strategy
that allocates initially the unknown treatment x (respectively, the known treatment y) and then
follows an optimal strategy. The dynamic programming equation becomes
V ((s; f)
; r; D) = max{V (x)((s; f)
; r; D); V (y)((s; f)
; r; D)}:
Moreover,
((s; f)
; r; D) = V (x)((s; f)
; r; D)− V (y)((s; f)
; r; D)
is the advantage of the unknown treatment x over the known treatment y and characterizes the
initially optimal selection of treatment: treatment x (treatment y) is optimal at state ((s; f)
; r; D) if
and only if ((s; f)
; r; D)¿ (6)0. When ((s; f)
; r; D) = 0, both treatments x and y are optimal
and there is no unique optimal selection.
X. Wang / Statistics & Probability Letters 60 (2002) 211–217 213
We assume the following conditions which are also assumed in Eick (1988):
Condition A. i¿
∑∞
j=i+1 j for i = 1; 2; : : : ;
Condition B. 
 is not concentrated at a single point, and 
{(0; 1)}= 0.
Under Condition A, Eick (1988) shows that for given 
 and r, ((s; f)
; r; D) is nondecreasing
in s and nonincreasing in f and k. Furthermore, ((s; f)
; r; D) is strictly monotone if there is a
strict inequality in Condition A for i=1 and Condition B is also true. This implies that the optimal
initial selection of treatment is characterized by the equation ((s; f)
; r; D) = 0.
Unfortunately this equation is formidable to solve in general. To gain insights on structural prop-
erties of the solutions to this equation, Wang (2000) demonstrates that for all f; k; r and 
, if
((s∗; f)
; r; D) = 0 and Condition A is true, then 06 s∗6 s∗1 where E(X |(s∗1 ; f)
) = k. Moreover,
if there is a strict inequality in Condition A for i = 1 and Condition B is also true, then s∗ is a
nondecreasing function of both f and k. Assume that D = (1; ; 2; : : :) is geometric and denote
Dn = (1; ; 2; : : : ; n−1; 0; 0; : : :). For given f and k, let s∗n be such that ((s∗n ; f)
; 0; Dn) = 0. Then
under Condition A,
06 · · ·6 s∗n6 · · ·6 s∗26 s∗1
and the limit s∗ = limn→∞ s∗n exists and satisJes ((s∗; f)
; 0; D) = 0. Moreover, if E(X |(s; f)
)
is nonincreasing in f and strictly increasing in s, then s∗¡s∗1 . If Condition B is also true and
q= P(X = 1|(0; f)
) = 1, then s∗¿ 0.
It is conjectured that similar results hold in general when r ¿ 0. These general structural properties
have been demonstrated through simulations but theoretical proofs have yet to be found. In this note,
we prove some limiting properties for the sequence s∗n . These properties provide asymptotic boundary
conditions for s∗n .
3. Asymptotic boundary structures for break-even values of s
To explicitly express the dependency of s∗n on r and f, write s∗n as s∗n(r; f) where ((s∗n ; f)

; r; Dn) = 0. We show that for a given f, s∗n(r; f) approaches s∗1(0; f) as the size r of the in-
formation bank goes to inJnity. On the other hand for given r, s∗n(r; f) approaches inJnity as the
number of observed deaths f on the unknown treatment goes to inJnity. Throughout this section, we
further assume
Condition C. limf→∞ E(X |(s; f)
) = 0 and limf→∞ p = 0 for any given s and 
, where p is
the probability of success under the unknown treatment x at the state ((s; f)
; r; D). Moreover,
E(X |(s; f)
) is nonincreasing in f and strictly increasing in s, and is continuous in s.
It is worth pointing out that these assumptions are intuitive and nonrestrictive. They are true, for
example, when 
 is a beta distribution because (s; f)
 is again a beta distribution. In what follows,
denote q= 1− p, D = (1; ; 2; : : :) and Dn = (1; ; 2; : : : ; n−1; 0; 0; : : :).
Lemma 1. Given s; 
 and r, the known arm is always optimal when f goes to in6nity. That is,
limf→∞ V ((s; f)
; r; Dn) = k
∑n−1
m=0 
m for any n.
214 X. Wang / Statistics & Probability Letters 60 (2002) 211–217
Proof. We prove by induction. The result is clearly true when n = 1 because ((s; f)
; r; D1) =
E(X |(s; f)
)− k. Suppose that the result is true when n= m. Then for n= m+ 1,
((s; f)
; r; Dm+1) =E(X |(s; f)
)− k
+ 
r+1∑
i=0
(
r + 1
i
)
V ((s+ j; f + r + 1− j)
; j; Dm)pjqr+1−j
− 
r∑
i=0
(
r
i
)
V ((s+ j; f + r − j)
; j; Dm)pjqr−j:
Since V is bounded, based on Condition C and the hypothesized result at n= m,
lim
f→∞
((s; f)
; r; Dn+1) =−k +  lim
f→∞
V ((s; f + r + 1)
; 0; Dm)
−  lim
f→∞
V ((s; f + r)
; 0; Dm) =−k ¡ 0:
Lemma 2.
r+1∑
i=0
(
r + 1
i
)
F(i) =
r∑
i=0
(
r
i
)
[F(i) + F(i + 1)]
for any function F .
Proof. It is well known that(
r + 1
i
)
=
(
r
i
)
+
(
r
i − 1
)
:
So
r+1∑
i=0
(
r + 1
i
)
F(i) =F(r + 1) + F(0) +
r∑
i=1
(
r
i
)
F(i) +
r∑
i=1
(
r
i − 1
)
F(i)
=
r∑
i=0
(
r
i
)
F(i) +
r+1∑
i=1
(
r
i − 1
)
F(i)
=
r∑
i=0
(
r
i
)
[F(i) + F(i + 1)]:
Our Jrst asymptotic boundary property has been observed without proof in Eick (1988).
X. Wang / Statistics & Probability Letters 60 (2002) 211–217 215
Theorem 1. limr→∞ [((s; f)
; r; Dn)− ((s; f)
; r; D1)] = 0 for any n.
Proof. From Lemma 2,
((s; f)
; r; Dn)− ((s; f)
; r; D1)
= 
r+1∑
i=0
(
r + 1
i
)
V ((s+ i; f + r + 1− i)
; i; Dn−1)piqr+1−i
− 
r∑
i=0
(
r
i
)
V ((s+ i; f + r − i)
; i; Dn−1)piqr−i
= 
r∑
i=0
(
r
i
)
[V ((s+ i; f + r + 1− i)
; i; Dn−1)q
+V ((s+ i + 1; f + r − i)
; i + 1; Dn−1)p
−V ((s+ i; f + r − i)
; i; Dn−1)]piqr−i
= 
r∑
i=0
(
r
i
)
W (i; r)piqr−i:
This is the Euler sum of the triangular sequence W (i; r). Clearly, this sequence goes to 0 as
r → ∞ for any Jxed i because the known arm is always optimal for each of the three V ’s and
p + q = 1. Therefore the Euler sum converges to 0 as well. That is, limr→∞ [((s; f)
; r; Dn) −
((s; f)
; r; D1)] = 0.
Replacing s by s∗n(r; f) in Theorem 1 implies limr→∞ ((s∗n(r; f); f)
; r; D1) = 0 and therefore
limr→∞ E(X |((s∗n(r; f); f)
) = k. Because of the continuity of E(X |((s; f)
) in s, we have
E
(
X
∣∣∣( lim
r→∞ (s
∗
n(r; f); f)

))
= k = E(X |((s∗1(0; f); f)
)):
Therefore from Condition C,
Corollary 1. limr→∞ s∗i (r; f) = s∗1(0; f) for any i = 2; 3; : : : :
From Wang (2000), s∗ is a nondecreasing function of f and hence the limit limf→∞ s∗n(r; f)
exists. In fact,
Theorem 2. For any given r and Dn, limf→∞ s∗n(r; f) =∞.
216 X. Wang / Statistics & Probability Letters 60 (2002) 211–217
Proof. ((s∗n(r; f); f)
; r; Dn) = 0 implies
E(X |(s∗n(r; f); f)
) + 
r+1∑
i=0
(
r + 1
i
)
V ((s∗n(r; f) + i; f + r + 1− i)
; i; Dn−1)piqr+1−i
= k + 
r∑
i=0
(
r
i
)
[V ((s∗n(r; f) + i; f + r − i)
; i; Dn−1)piqr−i]: (∗)
If limf→∞ s∗n(r; f) = M ¡∞, then E(X |(s∗n(r; f); f)
)6E(X |(M;f)
) and limf→∞
E(X |(s∗n(r; f); f)
) = 0 from Condition C. Moreover, the same arguments in the proof of Theo-
rem 1 imply
lim
f→∞
[

r+1∑
i=0
(
r + 1
i
)
V ((s+ i; f + r + 1− i)
; i; Dn−1)piqr+1−i
− 
r∑
i=0
(
r
i
)
[V ((s+ i; f + r − i)
; i; Dn−1)piqr−i]
]
= 0:
Taking the limit as f → ∞ on both sides of equation (∗) yields a contradiction, k = 0. Hence
limf→∞ s∗n(r; f) =∞.
Finally, we establish the relationship between s∗n(r; f) and s∗n(0; f) when f → ∞. On one hand,
((s∗n(r; f); f)
; r; Dn) = 0. On the other hand,
Theorem 3. limf→∞ ((s∗n(0; f); f)
; r; Dn) = 0 for any r and Dn.
Proof. Write s∗n(0; f) = s∗n . Then ((s∗n ; f)
; 0; Dn) = 0 implies
E(X |(s∗n ; f)
)− k = V ((s∗n ; f)
; 0; Dn−1)
− V ((s∗n + 1; f)
; 1; Dn−1)p− V ((s∗n ; f + 1)
; 0; Dn−1)q:
Therefore,
((s∗n ; f)
; r; Dn) =E(X |(s∗n ; f)
)− k
+ 
r+1∑
i=0
(
r + 1
i
)
V ((s∗n + j; f + r + 1− j)
; j; Dn−1)pjqr+1−j
− 
r∑
i=0
(
r
i
)
V ((s∗n + j; f + r − j)
; j; Dn−1)pjqr−j
= V ((s∗n ; f)
; 0; Dn−1)− V ((s∗n + 1; f)
; 1; Dn−1)p
X. Wang / Statistics & Probability Letters 60 (2002) 211–217 217
− V ((s∗n ; f + 1)
; 0; Dn−1)q
+ 
r+1∑
i=0
(
r + 1
i
)
V ((s∗n + j; f + r + 1− j)
; j; Dn−1)pjqr+1−j
− 
r∑
i=0
(
r
i
)
V ((s∗n + j; f + r − j)
; j; Dn−1)pjqr−j:
Since V is bounded, under Condition C,
lim
f→∞
((s∗n ; f)
; r; Dn)
=  lim
f→∞
[V ((s∗n ; f)
; 0; Dn−1)− V ((s∗n ; f + 1)
; 0; Dn−1)
+V ((s∗n ; f + r + 1)
; 0; Dn−1)− V ((s∗n ; f + r)
; 0; Dn−1)]:
From Wang (2000), s∗n6 s∗n−1, s∗n−1 is nondecreasing in f, and there is an optimal stopping solution
when r = 0. Therefore, the known arm is always optimal for each of the four V ’s in the above
expression and limf→∞ ((s∗n(0; f); f)
; r; Dn) = 0.
Corollary 2. If we assume that (
; r; Dn) is continuous in 
 and (
; r; Dn)=0 has a unique root
for 
, then limf→∞ (s∗n(0; f); f)
= limf→∞ (s∗n(r; f); f)
 given appropriate interpretations of the
limits of distributions.
Proof. The result is clear from
lim
f→∞
((s∗n(0; f); f)
; r; Dn) = 
(
lim
f→∞
(s∗n(0; f); f)
; r; Dn
)
= 0
and ((s∗n(r; f); f)
; r; Dn) = 0:
Acknowledgements
This research is supported by Natural Sciences and Engineering Research Council (NSERC) of
Canada.
References
Berry, D.A., Fristedt, B., 1985. Bandit Problems—Sequential Allocation of Experiments. Chapman & Hall, London,
New York.
Eick, S.G., 1988. The two-armed bandit with delayed responses. Ann. Statist. 16, 254–265.
Pullman, D., Wang, X., 2001. Adaptive designs, informed consent, and the ethics of research. Control. Clin. Trials 22,
203–210.
Wang, X., 2000. A bandit process with delayed responses. Statist. Probab. Lett. 48, 303–307.


Asymptotic properties of bandit processes with geometric responses

http://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=D4B355EC46AED1F8C1929612CB86667B?doi=10.1.1.1055.7837&rep=rep1&type=pdf

Asymptotic properties of bandit processes with geometric responses

Abstract

Similar works

Full text

Available Versions

CiteSeerX