We consider the problem of federated offline reinforcement learning (RL), a
scenario under which distributed learning agents must collaboratively learn a
high-quality control policy only using small pre-collected datasets generated
according to different unknown behavior policies. Naively combining a standard
offline RL approach with a standard federated learning approach to solve this
problem can lead to poorly performing policies. In response, we develop the
Federated Ensemble-Directed Offline Reinforcement Learning Algorithm (FEDORA),
which distills the collective wisdom of the clients using an ensemble learning
approach. We develop the FEDORA codebase to utilize distributed compute
resources on a federated learning platform. We show that FEDORA significantly
outperforms other approaches, including offline RL over the combined data pool,
in various complex continuous control environments and real world datasets.
Finally, we demonstrate the performance of FEDORA in the real-world on a mobile
robot