Optimal control and optimal sensor activation for Markov decision problems with costly observations

This paper considers partial observation Markov decision processes. Besides the classical control decisions influencing the transition probabilities of the Markov process, we also consider control actions that can activate the sensors to provide more or less accurate information about the system state, explicitly including the cost of activating sensors. We synthesize control laws that minimize a discounted operating cost of the system over an infinite interval of time, where the instantaneous cost function depends on the current state, the control influencing the transition probabilities, and the control actions activating the sensors. A general computationally efficient optimal solution for this problem is not known. Hence we design supoptimal controllers that only use knowledge of the value function for the full state information Markov decision problem. Our solution guarantees that the discounted cost of operating the plant increases only by a bounded amount with respect to the minimal cost for the full state information problem. A new concept of pinned conditional distributions of the state given the observed history of the plant is required in order to implement these control laws online

Convergence results for continuous-time adaptive stochastic filtering algorithms

AbstractThe adaptive stochastic filtering problem for Gaussian processes is considered. The self-tuning synthesis procedure is used to derive two algorithms for this problem. Almost sure convergence for the parameter estimate and the filtering error will be established. The convergence analysis is based on an almost-supermartingale convergence lemma that allows a stochastic Lyapunov-like approach

On hidden Markov chains and finite stochastic systems

In this paper we study various properties of finite stochastic systems or hidden Markov chains as they are alternatively called. We discuss their construction following different approaches and we also derive recursive filtering formulas for the different systems that we consider. The key tool is a simple lemma on conditional expectations