1,210 research outputs found
Trend Detection based Regret Minimization for Bandit Problems
We study a variation of the classical multi-armed bandits problem. In this
problem, the learner has to make a sequence of decisions, picking from a fixed
set of choices. In each round, she receives as feedback only the loss incurred
from the chosen action. Conventionally, this problem has been studied when
losses of the actions are drawn from an unknown distribution or when they are
adversarial. In this paper, we study this problem when the losses of the
actions also satisfy certain structural properties, and especially, do show a
trend structure. When this is true, we show that using \textit{trend
detection}, we can achieve regret of order with
respect to a switching strategy for the version of the problem where a single
action is chosen in each round and when actions
are chosen each round. This guarantee is a significant improvement over the
conventional benchmark. Our approach can, as a framework, be applied in
combination with various well-known bandit algorithms, like Exp3. For both
versions of the problem, we give regret guarantees also for the
\textit{anytime} setting, i.e. when the length of the choice-sequence is not
known in advance. Finally, we pinpoint the advantages of our method by
comparing it to some well-known other strategies
Multi-armed bandit problem with precedence relations
Consider a multi-phase project management problem where the decision maker
needs to deal with two issues: (a) how to allocate resources to projects within
each phase, and (b) when to enter the next phase, so that the total expected
reward is as large as possible. We formulate the problem as a multi-armed
bandit problem with precedence relations. In Chan, Fuh and Hu (2005), a class
of asymptotically optimal arm-pulling strategies is constructed to minimize the
shortfall from perfect information payoff. Here we further explore optimality
properties of the proposed strategies. First, we show that the efficiency
benchmark, which is given by the regret lower bound, reduces to those in Lai
and Robbins (1985), Hu and Wei (1989), and Fuh and Hu (2000). This implies that
the proposed strategy is also optimal under the settings of aforementioned
papers. Secondly, we establish the super-efficiency of proposed strategies when
the bad set is empty. Thirdly, we show that they are still optimal with
constant switching cost between arms. In addition, we prove that the Wald's
equation holds for Markov chains under Harris recurrent condition, which is an
important tool in studying the efficiency of the proposed strategies.Comment: Published at http://dx.doi.org/10.1214/074921706000001067 in the IMS
Lecture Notes Monograph Series
(http://www.imstat.org/publications/lecnotes.htm) by the Institute of
Mathematical Statistics (http://www.imstat.org
- …