Euclidean Markov decision processes are a powerful tool for modeling control
problems under uncertainty over continuous domains. Finite state imprecise,
Markov decision processes can be used to approximate the behavior of these
infinite models. In this paper we address two questions: first, we investigate
what kind of approximation guarantees are obtained when the Euclidean process
is approximated by finite state approximations induced by increasingly fine
partitions of the continuous state space. We show that for cost functions over
finite time horizons the approximations become arbitrarily precise. Second, we
use imprecise Markov decision process approximations as a tool to analyse and
validate cost functions and strategies obtained by reinforcement learning. We
find that, on the one hand, our new theoretical results validate basic design
choices of a previously proposed reinforcement learning approach. On the other
hand, the imprecise Markov decision process approximations reveal some
inaccuracies in the learned cost functions