336 research outputs found
Online Convex Optimization for Sequential Decision Processes and Extensive-Form Games
Regret minimization is a powerful tool for solving large-scale extensive-form
games. State-of-the-art methods rely on minimizing regret locally at each
decision point. In this work we derive a new framework for regret minimization
on sequential decision problems and extensive-form games with general compact
convex sets at each decision point and general convex losses, as opposed to
prior work which has been for simplex decision points and linear losses. We
call our framework laminar regret decomposition. It generalizes the CFR
algorithm to this more general setting. Furthermore, our framework enables a
new proof of CFR even in the known setting, which is derived from a perspective
of decomposing polytope regret, thereby leading to an arguably simpler
interpretation of the algorithm. Our generalization to convex compact sets and
convex losses allows us to develop new algorithms for several problems:
regularized sequential decision making, regularized Nash equilibria in
extensive-form games, and computing approximate extensive-form perfect
equilibria. Our generalization also leads to the first regret-minimization
algorithm for computing reduced-normal-form quantal response equilibria based
on minimizing local regrets. Experiments show that our framework leads to
algorithms that scale at a rate comparable to the fastest variants of
counterfactual regret minimization for computing Nash equilibrium, and
therefore our approach leads to the first algorithm for computing quantal
response equilibria in extremely large games. Finally we show that our
framework enables a new kind of scalable opponent exploitation approach
Smoothing Method for Approximate Extensive-Form Perfect Equilibrium
Nash equilibrium is a popular solution concept for solving
imperfect-information games in practice. However, it has a major drawback: it
does not preclude suboptimal play in branches of the game tree that are not
reached in equilibrium. Equilibrium refinements can mend this issue, but have
experienced little practical adoption. This is largely due to a lack of
scalable algorithms.
Sparse iterative methods, in particular first-order methods, are known to be
among the most effective algorithms for computing Nash equilibria in
large-scale two-player zero-sum extensive-form games. In this paper, we
provide, to our knowledge, the first extension of these methods to equilibrium
refinements. We develop a smoothing approach for behavioral perturbations of
the convex polytope that encompasses the strategy spaces of players in an
extensive-form game. This enables one to compute an approximate variant of
extensive-form perfect equilibria. Experiments show that our smoothing approach
leads to solutions with dramatically stronger strategies at information sets
that are reached with low probability in approximate Nash equilibria, while
retaining the overall convergence rate associated with fast algorithms for Nash
equilibrium. This has benefits both in approximate equilibrium finding (such
approximation is necessary in practice in large games) where some probabilities
are low while possibly heading toward zero in the limit, and exact equilibrium
computation where the low probabilities are actually zero.Comment: Published at IJCAI 1
Quasi-Perfect Stackelberg Equilibrium
Equilibrium refinements are important in extensive-form (i.e., tree-form)
games, where they amend weaknesses of the Nash equilibrium concept by requiring
sequential rationality and other beneficial properties. One of the most
attractive refinement concepts is quasi-perfect equilibrium. While
quasi-perfection has been studied in extensive-form games, it is poorly
understood in Stackelberg settings---that is, settings where a leader can
commit to a strategy---which are important for modeling, for example, security
games. In this paper, we introduce the axiomatic definition of quasi-perfect
Stackelberg equilibrium. We develop a broad class of game perturbation schemes
that lead to them in the limit. Our class of perturbation schemes strictly
generalizes prior perturbation schemes introduced for the computation of
(non-Stackelberg) quasi-perfect equilibria. Based on our perturbation schemes,
we develop a branch-and-bound algorithm for computing a quasi-perfect
Stackelberg equilibrium. It leverages a perturbed variant of the linear program
for computing a Stackelberg extensive-form correlated equilibrium. Experiments
show that our algorithm can be used to find an approximate quasi-perfect
Stackelberg equilibrium in games with thousands of nodes
- …