23,247 research outputs found
Boltzmann Exploration Done Right
Boltzmann exploration is a classic strategy for sequential decision-making
under uncertainty, and is one of the most standard tools in Reinforcement
Learning (RL). Despite its widespread use, there is virtually no theoretical
understanding about the limitations or the actual benefits of this exploration
scheme. Does it drive exploration in a meaningful way? Is it prone to
misidentifying the optimal actions or spending too much time exploring the
suboptimal ones? What is the right tuning for the learning rate? In this paper,
we address several of these questions in the classic setup of stochastic
multi-armed bandits. One of our main results is showing that the Boltzmann
exploration strategy with any monotone learning-rate sequence will induce
suboptimal behavior. As a remedy, we offer a simple non-monotone schedule that
guarantees near-optimal performance, albeit only when given prior access to key
problem parameters that are typically not available in practical situations
(like the time horizon and the suboptimality gap ). More
importantly, we propose a novel variant that uses different learning rates for
different arms, and achieves a distribution-dependent regret bound of order
and a distribution-independent bound of order
without requiring such prior knowledge. To demonstrate the
flexibility of our technique, we also propose a variant that guarantees the
same performance bounds even if the rewards are heavy-tailed
Martingales in self-similar growth-fragmentations and their connections with random planar maps
The purpose of the present work is twofold. First, we develop the theory of
general self-similar growth-fragmentation processes by focusing on martingales
which appear naturally in this setting and by recasting classical results for
branching random walks in this framework. In particular, we establish
many-to-one formulas for growth-fragmentations and define the notion of
intrinsic area of a growth-fragmentation. Second, we identify a distinguished
family of growth-fragmentations closely related to stable L\'evy processes,
which are then shown to arise as the scaling limit of the perimeter process in
Markovian explorations of certain random planar maps with large degrees (which
are, roughly speaking, the dual maps of the stable maps of Le Gall & Miermont.
As a consequence of this result, we are able to identify the law of the
intrinsic area of these distinguished growth-fragmentations. This generalizes a
geometric connection between large Boltzmann triangulations and a certain
growth-fragmentation process, which was established in arXiv:1507.02265 .Comment: 50 pages, 5 figures. Final version: to appear in Probab. Theory
Related Field
- …