Decision-focused (DF) model-based reinforcement learning has recently been
introduced as a powerful algorithm that can focus on learning the MDP dynamics
that are most relevant for obtaining high returns. While this approach
increases the agent's performance by directly optimizing the reward, it does so
by learning less accurate dynamics from a maximum likelihood perspective. We
demonstrate that when the reward function is defined by preferences over
multiple objectives, the DF model may be sensitive to changes in the objective
preferences.In this work, we develop the robust decision-focused (RDF)
algorithm, which leverages the non-identifiability of DF solutions to learn
models that maximize expected returns while simultaneously learning models that
transfer to changes in the preference over multiple objectives. We demonstrate
the effectiveness of RDF on two synthetic domains and two healthcare
simulators, showing that it significantly improves the robustness of DF model
learning to changes in the reward function without compromising training-time
return