Real-world planning problems\unicode{x2014}including autonomous driving and
sustainable energy applications like carbon storage and resource
exploration\unicode{x2014}have recently been modeled as partially observable
Markov decision processes (POMDPs) and solved using approximate methods. To
solve high-dimensional POMDPs in practice, state-of-the-art methods use online
planning with problem-specific heuristics to reduce planning horizons and make
the problems tractable. Algorithms that learn approximations to replace
heuristics have recently found success in large-scale problems in the fully
observable domain. The key insight is the combination of online Monte Carlo
tree search with offline neural network approximations of the optimal policy
and value function. In this work, we bring this insight to partially observed
domains and propose BetaZero, a belief-state planning algorithm for POMDPs.
BetaZero learns offline approximations based on accurate belief models to
enable online decision making in long-horizon problems. We address several
challenges inherent in large-scale partially observable domains; namely
challenges of transitioning in stochastic environments, prioritizing action
branching with limited search budget, and representing beliefs as input to the
network. We apply BetaZero to various well-established benchmark POMDPs found
in the literature. As a real-world case study, we test BetaZero on the
high-dimensional geological problem of critical mineral exploration.
Experiments show that BetaZero outperforms state-of-the-art POMDP solvers on a
variety of tasks.Comment: 20 page