Offline reinforcement learning (RL) addresses the problem of learning a
performant policy from a fixed batch of data collected by following some
behavior policy. Model-based approaches are particularly appealing in the
offline setting since they can extract more learning signals from the logged
dataset by learning a model of the environment. However, the performance of
existing model-based approaches falls short of model-free counterparts, due to
the compounding of estimation errors in the learned model. Driven by this
observation, we argue that it is critical for a model-based method to
understand when to trust the model and when to rely on model-free estimates,
and how to act conservatively w.r.t. both. To this end, we derive an elegant
and simple methodology called conservative Bayesian model-based value expansion
for offline policy optimization (CBOP), that trades off model-free and
model-based estimates during the policy evaluation step according to their
epistemic uncertainties, and facilitates conservatism by taking a lower bound
on the Bayesian posterior value estimate. On the standard D4RL continuous
control tasks, we find that our method significantly outperforms previous
model-based approaches: e.g., MOPO by 116.4%, MOReL by 23.2% and COMBO by
23.7%. Further, CBOP achieves state-of-the-art performance on 11 out of
18 benchmark datasets while doing on par on the remaining datasets