Learning in MDPs with highly complex state representations is currently
possible due to multiple advancements in reinforcement learning algorithm
design. However, this incline in complexity, and furthermore the increase in
the dimensions of the observation came at the cost of volatility that can be
taken advantage of via adversarial attacks (i.e. moving along worst-case
directions in the observation space). To solve this policy instability problem
we propose a novel method to detect the presence of these non-robust directions
via local quadratic approximation of the deep neural policy loss. Our method
provides a theoretical basis for the fundamental cut-off between safe
observations and adversarial observations. Furthermore, our technique is
computationally efficient, and does not depend on the methods used to produce
the worst-case directions. We conduct extensive experiments in the Arcade
Learning Environment with several different adversarial attack techniques. Most
significantly, we demonstrate the effectiveness of our approach even in the
setting where non-robust directions are explicitly optimized to circumvent our
proposed method.Comment: Published in ICML 202