Mean Field Games (MFGs) have been introduced to efficiently approximate games
with very large populations of strategic agents. Recently, the question of
learning equilibria in MFGs has gained momentum, particularly using model-free
reinforcement learning (RL) methods. One limiting factor to further scale up
using RL is that existing algorithms to solve MFGs require the mixing of
approximated quantities such as strategies or q-values. This is far from
being trivial in the case of non-linear function approximation that enjoy good
generalization properties, e.g. neural networks. We propose two methods to
address this shortcoming. The first one learns a mixed strategy from
distillation of historical data into a neural network and is applied to the
Fictitious Play algorithm. The second one is an online mixing method based on
regularization that does not require memorizing historical data or previous
estimates. It is used to extend Online Mirror Descent. We demonstrate
numerically that these methods efficiently enable the use of Deep RL algorithms
to solve various MFGs. In addition, we show that these methods outperform SotA
baselines from the literature