The stability of conditional Markov processes and Markov chains in random environments

We consider a discrete time hidden Markov model where the signal is a stationary Markov chain. When conditioned on the observations, the signal is a Markov chain in a random environment under the conditional measure. It is shown that this conditional signal is weakly ergodic when the signal is ergodic and the observations are nondegenerate. This permits a delicate exchange of the intersection and supremum of $\sigma$-fields, which is key for the stability of the nonlinear filter and partially resolves a long-standing gap in the proof of a result of Kunita [J. Multivariate Anal. 1 (1971) 365--393]. A similar result is obtained also in the continuous time setting. The proofs are based on an ergodic theorem for Markov chains in random environments in a general state space.Comment: Published in at http://dx.doi.org/10.1214/08-AOP448 the Annals of Probability (http://www.imstat.org/aop/) by the Institute of Mathematical Statistics (http://www.imstat.org

Maximization of the portfolio growth rate under fixed and proportional transaction costs

This paper considers a discrete-time Markovian model of asset prices with economic factors and transaction costs with proportional and fixed terms. Existence of optimal strategies maximizing average growth rate of portfolio is proved in the case of complete and partial observation of the process modelling the economic factors. The proof is based on a modification of the vanishing discount approach. The main difficulty is the discontinuity of the controlled transition operator of the underlying Markov process

Discrete-time controlled markov processes with average cost criterion: a survey

This work is a survey of the average cost control problem for discrete-time Markov processes. The authors have attempted to put together a comprehensive account of the considerable research on this problem over the past three decades. The exposition ranges from finite to Borel state and action spaces and includes a variety of methodologies to find and characterize optimal policies. The authors have included a brief historical perspective of the research efforts in this area and have compiled a substantial yet not exhaustive bibliography. The authors have also identified several important questions that are still open to investigation

Geometric ergodicity in a weighted sobolev space

For a discrete-time Markov chain $\{X(t)\}$ evolving on $\Re^\ell$ with transition kernel $P$, natural, general conditions are developed under which the following are established: 1. The transition kernel $P$ has a purely discrete spectrum, when viewed as a linear operator on a weighted Sobolev space $L_\infty^{v,1}$ of functions with norm, $$\|f\|_{v,1} = \sup_{x \in \Re^\ell} \frac{1}{v(x)} \max \{|f(x)|, |\partial_1 f(x)|,\ldots,|\partial_\ell f(x)|\},$$ where $v\colon \Re^\ell \to [1,\infty)$ is a Lyapunov function and $\partial_i:=\partial/\partial x_i$. 2. The Markov chain is geometrically ergodic in $L_\infty^{v,1}$: There is a unique invariant probability measure $\pi$ and constants $B<\infty$ and $\delta>0$ such that, for each $f\in L_\infty^{v,1}$, any initial condition $X(0)=x$, and all $t\geq 0$: $$\Big| \text{E}_x[f(X(t))] - \pi(f)\Big| \le Be^{-\delta t}v(x),\quad \|\nabla \text{E}_x[f(X(t))] \|_2 \le Be^{-\delta t} v(x),$$ where $\pi(f)=\int fd\pi$. 3. For any function $f\in L_\infty^{v,1}$ there is a function $h\in L_\infty^{v,1}$ solving Poisson's equation: $$h-Ph = f-\pi(f).$$ Part of the analysis is based on an operator-theoretic treatment of the sensitivity process that appears in the theory of Lyapunov exponents

Large deviations for some fast stochastic volatility models by viscosity methods

We consider the short time behaviour of stochastic systems affected by a stochastic volatility evolving at a faster time scale. We study the asymptotics of a logarithmic functional of the process by methods of the theory of homogenisation and singular perturbations for fully nonlinear PDEs. We point out three regimes depending on how fast the volatility oscillates relative to the horizon length. We prove a large deviation principle for each regime and apply it to the asymptotics of option prices near maturity

Optimal sequential vector quantization of Markov sources

Includes bibliographical references (p. 30-31).Supported by U.S. Army grant. PAAL03-92-G-0115 Supported by a Homi Bhabha Fellowship and the Center for Intelligent Control Systems.V.S. Borkar, Sanjoy K. Mitter, Sekhar Tatikonda

Stock Market Volatility and Learning

We study a standard consumption based asset pricing model with rational investors who entertain subjective prior beliefs about price behavior. Optimal behavior then dictates that investors learn about price behavior from past price observations. We show that this imparts momentum and mean reversion into the equilibrium behavior of the price dividend ratio, similar to what can be observed in the data. Estimating the model on U.S. stock price data using the method of simulated moments, we show that it can quantitatively account for the observed stock price volatility, the persistence of the price-dividend ratio, and the predictability of long-horizon returns. For reasonable degrees of risk aversion, the model also passes a formal statistical test for the overall goodness of fit, provided one excludes the equity premium from the set of moments to be matched

Uniform positive recurrence and long term behavior of Markov decision processes, with applications in sensor scheduling

In this dissertation, we show a number of new results relating to stability, optimal control, and value iteration algorithms for discrete-time Markov decision processes (MDPs). First, we adapt two recent results in controlled diffusion processes to suit countable state MDPs by making assumptions that approximate continuous behavior. We show that if the MDP is stable under any stationary policy, then it must be uniformly so under all policies. This abstract result is very useful in the analysis of optimal control problems, and extends the characterization of uniform stability properties for MDPs. Then we derive two useful local bounds on the discounted value functions for a large class of MDPs, facilitating analysis of the ergodic cost problem via the Arzela-Ascoli theorem. We also examine and exploit the previously underutilized Harnack inequality for discrete Markov chains; one aim of this work was to discover how much can be accomplished for models with this property. Convergence of the value iteration algorithm is typically treated in the literature under blanket stability assumptions. We show two new sufficient conditions for the convergence of the value iteration algorithm without blanket stability, requiring only geometric ergodicity under the optimal policy. These results form the theoretical basis to apply the value iteration to classes of problems previously unavailable. We then consider a discrete-time linear system with Gaussian white noise and quadratic costs, observed via multiple sensors that communicate over a congested network. Observations are lost or received according to a Bernoulli random variable with a loss rate determined by the state of the network and the choice of sensor. We completely analyze the finite horizon, discounted, and long-term average optimal control problems. Assuming that the system is stabilizable, we use a partial separation principle to transform the problem into an MDP on the set of symmetric, positive definite matrices. A special case of these results generalizes a known result for Kalman filters with intermittent observations to the multiple-sensor case, with powerful implications. Finally, we show that the value iteration algorithm converges without additional assumptions, as the structure of the problem guarantees geometric ergodicity under the optimal policy. The results allow the incorporation of adaptive schemes to determine unknown system parameters without affecting stability or long-term average cost. We also show that after only a few steps of the value iteration algorithm, the generated policy is geometrically ergodic and near-optimal.Electrical and Computer Engineerin