We present a hierarchical reinforcement learning framework that formulates
each task in the hierarchy as a special type of Markov decision process for
which the Bellman equation is linear and has analytical solution. Problems of
this type, called linearly-solvable MDPs (LMDPs) have interesting properties
that can be exploited in a hierarchical setting, such as efficient learning of
the optimal value function or task compositionality. The proposed hierarchical
approach can also be seen as a novel alternative to solving LMDPs with large
state spaces. We derive a hierarchical version of the so-called Z-learning
algorithm that learns different tasks simultaneously and show empirically that
it significantly outperforms the state-of-the-art learning methods in two
classical hierarchical reinforcement learning domains: the taxi domain and an
autonomous guided vehicle task.Comment: 11 pages, 6 figures, 26th International Conference on Automated
Planning and Schedulin
