489 research outputs found
Safe Reinforcement Learning as Wasserstein Variational Inference: Formal Methods for Interpretability
Reinforcement Learning or optimal control can provide effective reasoning for
sequential decision-making problems with variable dynamics. Such reasoning in
practical implementation, however, poses a persistent challenge in interpreting
the reward function and corresponding optimal policy. Consequently, formalizing
the sequential decision-making problems as inference has a considerable value,
as probabilistic inference in principle offers diverse and powerful
mathematical tools to infer the stochastic dynamics whilst suggesting a
probabilistic interpretation of the reward design and policy convergence. In
this study, we propose a novel Adaptive Wasserstein Variational Optimization
(AWaVO) to tackle these challenges in sequential decision-making. Our approach
utilizes formal methods to provide interpretations of reward design,
transparency of training convergence, and probabilistic interpretation of
sequential decisions. To demonstrate practicality, we show convergent training
with guaranteed global convergence rates not only in simulation but also in
real robot tasks, and empirically verify a reasonable tradeoff between high
performance and conservative interpretability.Comment: 24 pages, 8 figures, containing Appendi
- …