Actor-Critic Algorithms for Risk-Sensitive MDPs

In many sequential decision-making problems we may want to manage risk by minimizing some measure of variability in rewards in addition to maximizing a standard criterion. Variance-related risk measures are among the most common risk-sensitive criteria in finance and operations research. However, optimizing many such criteria is known to be a hard problem. In this paper, we consider both discounted and average reward Markov decision processes. For each formulation, we first define a measure of variability for a policy, which in turn gives us a set of risk-sensitive criteria to optimize. For each of these criteria, we derive a formula for computing its gradient. We then devise actor-critic algorithms for estimating the gradient and updating the policy parameters in the ascent direction. We establish the convergence of our algorithms to locally risk-sensitive optimal policies. Finally, we demonstrate the usefulness of our algorithms in a traffic signal control application

Two-step Nonnegative Matrix Factorization Algorithm for the Approximate Realization of Hidden Markov Models

We propose a two-step algorithm for the construction of a Hidden Markov Model (HMM) of assigned size, i.e. cardinality of the state space of the underlying Markov chain, whose $n$-dimensional distribution is closest in divergence to a given distribution. The algorithm is based on the factorization of a pseudo Hankel matrix, defined in terms of the given distribution, into the product of a tall and a wide nonnegative matrix. The implementation is based on the nonnegative matrix factorization (NMF) algorithm. To evaluate the performance of our algorithm we produced some numerical simulations in the context of HMM order reduction.Comment: presented at MTNS2010 - Budapest, July 201