159,332 research outputs found
Thompson Sampling for Bandits with Clustered Arms
We propose algorithms based on a multi-level Thompson sampling scheme, for the stochastic multi-armed bandit and its contextual variant with linear expected rewards, in the setting where arms are clustered. We show, both theoretically and empirically, how exploiting a given cluster structure can significantly improve the regret and computational cost compared to using standard Thompson sampling. In the case of the stochastic multi-armed bandit we give upper bounds on the expected cumulative regret showing how it depends on the quality of the clustering. Finally, we perform an empirical evaluation showing that our algorithms perform well compared to previously proposed algorithms for bandits with clustered arms
Tsallis-INF: An Optimal Algorithm for Stochastic and Adversarial Bandits
We derive an algorithm that achieves the optimal (within constants)
pseudo-regret in both adversarial and stochastic multi-armed bandits without
prior knowledge of the regime and time horizon. The algorithm is based on
online mirror descent (OMD) with Tsallis entropy regularization with power
and reduced-variance loss estimators. More generally, we define an
adversarial regime with a self-bounding constraint, which includes stochastic
regime, stochastically constrained adversarial regime (Wei and Luo), and
stochastic regime with adversarial corruptions (Lykouris et al.) as special
cases, and show that the algorithm achieves logarithmic regret guarantee in
this regime and all of its special cases simultaneously with the adversarial
regret guarantee.} The algorithm also achieves adversarial and stochastic
optimality in the utility-based dueling bandit setting. We provide empirical
evaluation of the algorithm demonstrating that it significantly outperforms
UCB1 and EXP3 in stochastic environments. We also provide examples of
adversarial environments, where UCB1 and Thompson Sampling exhibit almost
linear regret, whereas our algorithm suffers only logarithmic regret. To the
best of our knowledge, this is the first example demonstrating vulnerability of
Thompson Sampling in adversarial environments. Last, but not least, we present
a general stochastic analysis and a general adversarial analysis of OMD
algorithms with Tsallis entropy regularization for and explain
the reason why works best
Monte-Carlo tree search with uncertainty propagation via optimal transport
This paper introduces a novel backup strategy for Monte-Carlo Tree Search
(MCTS) designed for highly stochastic and partially observable Markov decision
processes. We adopt a probabilistic approach, modeling both value and
action-value nodes as Gaussian distributions. We introduce a novel backup
operator that computes value nodes as the Wasserstein barycenter of their
action-value children nodes; thus, propagating the uncertainty of the estimate
across the tree to the root node. We study our novel backup operator when using
a novel combination of -Wasserstein barycenter with -divergence,
by drawing a notable connection to the generalized mean backup operator. We
complement our probabilistic backup operator with two sampling strategies,
based on optimistic selection and Thompson sampling, obtaining our Wasserstein
MCTS algorithm. We provide theoretical guarantees of asymptotic convergence to
the optimal policy, and an empirical evaluation on several stochastic and
partially observable environments, where our approach outperforms well-known
related baselines
Context Attentive Bandits: Contextual Bandit with Restricted Context
We consider a novel formulation of the multi-armed bandit model, which we
call the contextual bandit with restricted context, where only a limited number
of features can be accessed by the learner at every iteration. This novel
formulation is motivated by different online problems arising in clinical
trials, recommender systems and attention modeling. Herein, we adapt the
standard multi-armed bandit algorithm known as Thompson Sampling to take
advantage of our restricted context setting, and propose two novel algorithms,
called the Thompson Sampling with Restricted Context(TSRC) and the Windows
Thompson Sampling with Restricted Context(WTSRC), for handling stationary and
nonstationary environments, respectively. Our empirical results demonstrate
advantages of the proposed approaches on several real-life datasetsComment: IJCAI 201
Bandit Models of Human Behavior: Reward Processing in Mental Disorders
Drawing an inspiration from behavioral studies of human decision making, we
propose here a general parametric framework for multi-armed bandit problem,
which extends the standard Thompson Sampling approach to incorporate reward
processing biases associated with several neurological and psychiatric
conditions, including Parkinson's and Alzheimer's diseases,
attention-deficit/hyperactivity disorder (ADHD), addiction, and chronic pain.
We demonstrate empirically that the proposed parametric approach can often
outperform the baseline Thompson Sampling on a variety of datasets. Moreover,
from the behavioral modeling perspective, our parametric framework can be
viewed as a first step towards a unifying computational model capturing reward
processing abnormalities across multiple mental conditions.Comment: Conference on Artificial General Intelligence, AGI-1
Shrinkage Estimators in Online Experiments
We develop and analyze empirical Bayes Stein-type estimators for use in the
estimation of causal effects in large-scale online experiments. While online
experiments are generally thought to be distinguished by their large sample
size, we focus on the multiplicity of treatment groups. The typical analysis
practice is to use simple differences-in-means (perhaps with covariate
adjustment) as if all treatment arms were independent. In this work we develop
consistent, small bias, shrinkage estimators for this setting. In addition to
achieving lower mean squared error these estimators retain important
frequentist properties such as coverage under most reasonable scenarios. Modern
sequential methods of experimentation and optimization such as multi-armed
bandit optimization (where treatment allocations adapt over time to prior
responses) benefit from the use of our shrinkage estimators. Exploration under
empirical Bayes focuses more efficiently on near-optimal arms, improving the
resulting decisions made under uncertainty. We demonstrate these properties by
examining seventeen large-scale experiments conducted on Facebook from April to
June 2017
- …