14 research outputs found
Quickest change detection approach to optimal control in Markov decision processes with model changes
Optimal control in non-stationary Markov decision processes (MDP) is a challenging problem. The aim in such a control problem is to maximize the long-term discounted reward when the transition dynamics or the reward function can change over time. When a prior knowledge of change statistics is available, the standard Bayesian approach to this problem is to reformulate it as a partially observable MDP (POMDP) and solve it using approximate POMDP solvers, which are typically computationally demanding. In this paper, the problem is analyzed through the viewpoint of quickest change detection (QCD), a set of tools for detecting a change in the distribution of a sequence of random variables. Current methods applying QCD to such problems only passively detect changes by following prescribed policies, without optimizing the choice of actions for long term performance. We demonstrate that ignoring the reward-detection trade-off can cause a significant loss in long term rewards, and propose a two threshold switching strategy to solve the issue. A non-Bayesian problem formulation is also proposed for scenarios where a Bayesian formulation cannot be defined. The performance of the proposed two threshold strategy is examined through numerical analysis on a non-stationary MDP task, and the strategy outperforms the state-of-the-art QCD methods in both Bayesian and non-Bayesian settings.Lincoln LaboratoryNorthrop Grumman Corporatio
Sequential Transfer in Reinforcement Learning with a Generative Model
We are interested in how to design reinforcement learning agents that
provably reduce the sample complexity for learning new tasks by transferring
knowledge from previously-solved ones. The availability of solutions to related
problems poses a fundamental trade-off: whether to seek policies that are
expected to achieve high (yet sub-optimal) performance in the new task
immediately or whether to seek information to quickly identify an optimal
solution, potentially at the cost of poor initial behavior. In this work, we
focus on the second objective when the agent has access to a generative model
of state-action pairs. First, given a set of solved tasks containing an
approximation of the target one, we design an algorithm that quickly identifies
an accurate solution by seeking the state-action pairs that are most
informative for this purpose. We derive PAC bounds on its sample complexity
which clearly demonstrate the benefits of using this kind of prior knowledge.
Then, we show how to learn these approximate tasks sequentially by reducing our
transfer setting to a hidden Markov model and employing spectral methods to
recover its parameters. Finally, we empirically verify our theoretical findings
in simple simulated domains.Comment: ICML 202