Action and observation delays commonly occur in many Reinforcement Learning
applications, such as remote control scenarios. We study the anatomy of
randomly delayed environments, and show that partially resampling trajectory
fragments in hindsight allows for off-policy multi-step value estimation. We
apply this principle to derive Delay-Correcting Actor-Critic (DCAC), an
algorithm based on Soft Actor-Critic with significantly better performance in
environments with delays. This is shown theoretically and also demonstrated
practically on a delay-augmented version of the MuJoCo continuous control
benchmark