On the Nonconvergence of Fictitious Play in Coordination Games

It is natural to conjecture that fictitious play converges in coordination games, but this is shown by counterexample to be false. Variants of fictitious play in which past actions are eventually forgotten and there are small stochastic perturbations are much better behaved: over the long run players manage to coordinate with high probability

On Similarities between Inference in Game Theory and Machine Learning

In this paper, we elucidate the equivalence between inference in game theory and machine learning. Our aim in so doing is to establish an equivalent vocabulary between the two domains so as to facilitate developments at the intersection of both fields, and as proof of the usefulness of this approach, we use recent developments in each field to make useful improvements to the other. More specifically, we consider the analogies between smooth best responses in fictitious play and Bayesian inference methods. Initially, we use these insights to develop and demonstrate an improved algorithm for learning in games based on probabilistic moderation. That is, by integrating over the distribution of opponent strategies (a Bayesian approach within machine learning) rather than taking a simple empirical average (the approach used in standard fictitious play) we derive a novel moderated fictitious play algorithm and show that it is more likely than standard fictitious play to converge to a payoff-dominant but risk-dominated Nash equilibrium in a simple coordination game. Furthermore we consider the converse case, and show how insights from game theory can be used to derive two improved mean field variational learning algorithms. We first show that the standard update rule of mean field variational learning is analogous to a Cournot adjustment within game theory. By analogy with fictitious play, we then suggest an improved update rule, and show that this results in fictitious variational play, an improved mean field variational learning algorithm that exhibits better convergence in highly or strongly connected graphical models. Second, we use a recent advance in fictitious play, namely dynamic fictitious play, to derive a derivative action variational learning algorithm, that exhibits superior convergence properties on a canonical machine learning problem (clustering a mixture distribution)

On the Nonconvergence of Fictitious Play in Coordination Games (Revised version of WP-95-001)

It is shown by example that learning rules of the fictitious play type fail to converge in certain kinds of coordination games. By contrast, learning rules in which past actions are eventually forgotten and which incorporate small stochastic perturbations are better behaved: over the long run, players manage to coordinate with probability one

Coordination in Networks Formation: Experimental Evidence on Learning and Salience

We present experiments on repeated non-cooperative network formation games, based on Bala and Goyal (2000). We treat the one-way and the two-ways flow models, each for high and low link costs. The models show both multiple equilibria and coordination problems. We conduct experiments under various conditions which control for salient labeling and learning dynamics. Contrary to previous experiments, we find that coordination on non-empty Strict Nash equilibria is not an easy task for subjects to achieve, even in the mono-directional model where the Strict Nash equilibria is a wheel. We find that salience significantly helps coordination, but only when subjects are pre-instructed to think of the wheel network as a reasonable way to play the networking game. Evidence on learning behavior provides support for subjects choosing strategies consistent with various learning rules, which include as the main ones Reinforcement and Fictitious Play.Experiments, Networks, Behavioral game theory, Salience, Learning dynamics

Experience-weighted Attraction Learning in Normal Form Games

In ‘experience-weighted attraction’ (EWA) learning, strategies have attractions that reflect initial predispositions, are updated based on payoff experience, and determine choice probabilities according to some rule (e.g., logit). A key feature is a parameter δ that weights the strength of hypothetical reinforcement of strategies that were not chosen according to the payoff they would have yielded, relative to reinforcement of chosen strategies according to received payoffs. The other key features are two discount rates, φ and ρ, which separately discount previous attractions, and an experience weight. EWA includes reinforcement learning and weighted fictitious play (belief learning) as special cases, and hybridizes their key elements. When δ= 0 and ρ= 0, cumulative choice reinforcement results. When δ= 1 and ρ=φ, levels of reinforcement of strategies are exactly the same as expected payoffs given weighted fictitious play beliefs. Using three sets of experimental data, parameter estimates of the model were calibrated on part of the data and used to predict a holdout sample. Estimates of δ are generally around .50, φ around .8 − 1, and ρ varies from 0 to φ. Reinforcement and belief-learning special cases are generally rejected in favor of EWA, though belief models do better in some constant-sum games. EWA is able to combine the best features of previous approaches, allowing attractions to begin and grow flexibly as choice reinforcement does, but reinforcing unchosen strategies substantially as belief-based models implicitly do

Self-tuning experience weighted attraction learning in games

Self-tuning experience weighted attraction (EWA) is a one-parameter theory of learning in games. It addresses a criticism that an earlier model (EWA) has too many parameters, by fixing some parameters at plausible values and replacing others with functions of experience so that they no longer need to be estimated. Consequently, it is econometrically simpler than the popular weighted fictitious play and reinforcement learning models. The functions of experience which replace free parameters “self-tune” over time, adjusting in a way that selects a sensible learning rule to capture subjects’ choice dynamics. For instance, the self-tuning EWA model can turn from a weighted fictitious play into an averaging reinforcement learning as subjects equilibrate and learn to ignore inferior foregone payoffs. The theory was tested on seven different games, and compared to the earlier parametric EWA model and a one-parameter stochastic equilibrium theory (QRE). Self-tuning EWA does as well as EWA in predicting behavior in new games, even though it has fewer parameters, and fits reliably better than the QRE equilibrium benchmark