Offline reinforcement learning (Offline RL) suffers from the innate
distributional shift as it cannot interact with the physical environment during
training. To alleviate such limitation, state-based offline RL leverages a
learned dynamics model from the logged experience and augments the predicted
state transition to extend the data distribution. For exploiting such benefit
also on the image-based RL, we firstly propose a generative model, S2P
(State2Pixel), which synthesizes the raw pixel of the agent from its
corresponding state. It enables bridging the gap between the state and the
image domain in RL algorithms, and virtually exploring unseen image
distribution via model-based transition in the state space. Through
experiments, we confirm that our S2P-based image synthesis not only improves
the image-based offline RL performance but also shows powerful generalization
capability on unseen tasks.Comment: NeurIPS 2022, first two authors contributed equall