Balancing exploration and conservatism in the constrained setting is an
important problem if we are to use reinforcement learning for meaningful tasks
in the real world. In this paper, we propose a principled algorithm for safe
exploration based on the concept of shielding. Previous approaches to shielding
assume access to a safety-relevant abstraction of the environment or a
high-fidelity simulator. Instead, our work is based on latent shielding -
another approach that leverages world models to verify policy roll-outs in the
latent space of a learned dynamics model. Our novel algorithm builds on this
previous work, using safety critics and other additional features to improve
the stability and farsightedness of the algorithm. We demonstrate the
effectiveness of our approach by running experiments on a small set of Atari
games with state dependent safety labels. We present preliminary results that
show our approximate shielding algorithm effectively reduces the rate of safety
violations, and in some cases improves the speed of convergence and quality of
the final agent.Comment: Accepted for presentation at the ALA workshop as part of AAMAS 202