322 research outputs found
Variance Reduction in Monte Carlo Counterfactual Regret Minimization (VR-MCCFR) for Extensive Form Games using Baselines
Learning strategies for imperfect information games from samples of
interaction is a challenging problem. A common method for this setting, Monte
Carlo Counterfactual Regret Minimization (MCCFR), can have slow long-term
convergence rates due to high variance. In this paper, we introduce a variance
reduction technique (VR-MCCFR) that applies to any sampling variant of MCCFR.
Using this technique, per-iteration estimated values and updates are
reformulated as a function of sampled values and state-action baselines,
similar to their use in policy gradient reinforcement learning. The new
formulation allows estimates to be bootstrapped from other estimates within the
same episode, propagating the benefits of baselines along the sampled
trajectory; the estimates remain unbiased even when bootstrapping from other
estimates. Finally, we show that given a perfect baseline, the variance of the
value estimates can be reduced to zero. Experimental evaluation shows that
VR-MCCFR brings an order of magnitude speedup, while the empirical variance
decreases by three orders of magnitude. The decreased variance allows for the
first time CFR+ to be used with sampling, increasing the speedup to two orders
of magnitude
Counterfactual Risk Minimization: Learning from Logged Bandit Feedback
We develop a learning principle and an efficient algorithm for batch learning
from logged bandit feedback. This learning setting is ubiquitous in online
systems (e.g., ad placement, web search, recommendation), where an algorithm
makes a prediction (e.g., ad ranking) for a given input (e.g., query) and
observes bandit feedback (e.g., user clicks on presented ads). We first address
the counterfactual nature of the learning problem through propensity scoring.
Next, we prove generalization error bounds that account for the variance of the
propensity-weighted empirical risk estimator. These constructive bounds give
rise to the Counterfactual Risk Minimization (CRM) principle. We show how CRM
can be used to derive a new learning method -- called Policy Optimizer for
Exponential Models (POEM) -- for learning stochastic linear rules for
structured output prediction. We present a decomposition of the POEM objective
that enables efficient stochastic gradient optimization. POEM is evaluated on
several multi-label classification problems showing substantially improved
robustness and generalization performance compared to the state-of-the-art.Comment: 10 page
Hierarchical Deep Counterfactual Regret Minimization
Imperfect Information Games (IIGs) offer robust models for scenarios where
decision-makers face uncertainty or lack complete information. Counterfactual
Regret Minimization (CFR) has been one of the most successful family of
algorithms for tackling IIGs. The integration of skill-based strategy learning
with CFR could potentially enhance learning performance for complex IIGs. For
this, a hierarchical strategy needs to be learnt, wherein low-level components
represent specific skills and the high-level component manages the transition
between skills. This hierarchical approach also enhances interpretability,
helping humans pinpoint scenarios where the agent is struggling and intervene
with targeted expertise. This paper introduces the first hierarchical version
of Deep CFR (HDCFR), an innovative method that boosts learning efficiency in
tasks involving extensively large state spaces and deep game trees. A notable
advantage of HDCFR over previous research in this field is its ability to
facilitate learning with predefined (human) expertise and foster the
acquisition of transferable skills that can be applied to similar tasks. To
achieve this, we initially construct our algorithm on a tabular setting,
encompassing hierarchical CFR updating rules and a variance-reduced Monte-Carlo
sampling extension, and offer its essential theoretical guarantees. Then, to
adapt our algorithm for large-scale applications, we employ neural networks as
function approximators and suggest deep learning objectives that coincide with
those in the tabular setting while maintaining the theoretical outcomes
- …