Wide sense one-dependent processes with embedded Harris chains and their applications in inventory management

In this paper we consider stochastic processes with an embedded Harris chain. The embedded Harris chain describes the dependence structure of the stochastic process. That is, all the relevant information of the past is contained in the state of the embedded Harris chain. For these processes we proved a powerful reward theorem. Futher, we show how we can control these type of processes and give a formulation similar to semi-Markov decision processes. Finally we discuss a number of applications in inventory management.

Nonparametric Infinite Horizon Kullback-Leibler Stochastic Control

We present two nonparametric approaches to Kullback-Leibler (KL) control, or linearly-solvable Markov decision problem (LMDP) based on Gaussian processes (GP) and Nystr\"{o}m approximation. Compared to recently developed parametric methods, the proposed data-driven frameworks feature accurate function approximation and efficient on-line operations. Theoretically, we derive the mathematical connection of KL control based on dynamic programming with earlier work in control theory which relies on information theoretic dualities for the infinite time horizon case. Algorithmically, we give explicit optimal control policies in nonparametric forms, and propose on-line update schemes with budgeted computational costs. Numerical results demonstrate the effectiveness and usefulness of the proposed frameworks

Bayesian learning of noisy Markov decision processes

We consider the inverse reinforcement learning problem, that is, the problem of learning from, and then predicting or mimicking a controller based on state/action data. We propose a statistical model for such data, derived from the structure of a Markov decision process. Adopting a Bayesian approach to inference, we show how latent variables of the model can be estimated, and how predictions about actions can be made, in a unified framework. A new Markov chain Monte Carlo (MCMC) sampler is devised for simulation from the posterior distribution. This step includes a parameter expansion step, which is shown to be essential for good convergence properties of the MCMC sampler. As an illustration, the method is applied to learning a human controller

DYNAMIC PROGRAMMING: HAS ITS DAY ARRIVED?

Research Methods/ Statistical Methods,