Markov two-components processes

We propose Markov two-components processes (M2CP) as a probabilistic model of asynchronous systems based on the trace semantics for concurrency. Considering an asynchronous system distributed over two sites, we introduce concepts and tools to manipulate random trajectories in an asynchronous framework: stopping times, an Asynchronous Strong Markov property, recurrent and transient states and irreducible components of asynchronous probabilistic processes. The asynchrony assumption implies that there is no global totally ordered clock ruling the system. Instead, time appears as partially ordered and random. We construct and characterize M2CP through a finite family of transition matrices. M2CP have a local independence property that guarantees that local components are independent in the probabilistic sense, conditionally to their synchronization constraints. A synchronization product of two Markov chains is introduced, as a natural example of M2CP.Comment: 34 page

Approximating stochastic volatility by recombinant trees

A general method to construct recombinant tree approximations for stochastic volatility models is developed and applied to the Heston model for stock price dynamics. In this application, the resulting approximation is a four tuple Markov process. The first two components are related to the stock and volatility processes and take values in a two-dimensional binomial tree. The other two components of the Markov process are the increments of random walks with simple values in $\{-1,+1\}$. The resulting efficient option pricing equations are numerically implemented for general American and European options including the standard put and calls, barrier, lookback and Asian-type pay-offs. The weak and extended weak convergences are also proved.Comment: Published in at http://dx.doi.org/10.1214/13-AAP977 the Annals of Applied Probability (http://www.imstat.org/aap/) by the Institute of Mathematical Statistics (http://www.imstat.org

Two Timescale Stochastic Approximation with Controlled Markov noise and Off-policy temporal difference learning

We present for the first time an asymptotic convergence analysis of two time-scale stochastic approximation driven by `controlled' Markov noise. In particular, both the faster and slower recursions have non-additive controlled Markov noise components in addition to martingale difference noise. We analyze the asymptotic behavior of our framework by relating it to limiting differential inclusions in both time-scales that are defined in terms of the ergodic occupation measures associated with the controlled Markov processes. Finally, we present a solution to the off-policy convergence problem for temporal difference learning with linear function approximation, using our results.Comment: 23 pages (relaxed some important assumptions from the previous version), accepted in Mathematics of Operations Research in Feb, 201

Convergence Analysis of the Approximate Newton Method for Markov Decision Processes

Recently two approximate Newton methods were proposed for the optimisation of Markov Decision Processes. While these methods were shown to have desirable properties, such as a guarantee that the preconditioner is negative-semidefinite when the policy is $\log$-concave with respect to the policy parameters, and were demonstrated to have strong empirical performance in challenging domains, such as the game of Tetris, no convergence analysis was provided. The purpose of this paper is to provide such an analysis. We start by providing a detailed analysis of the Hessian of a Markov Decision Process, which is formed of a negative-semidefinite component, a positive-semidefinite component and a remainder term. The first part of our analysis details how the negative-semidefinite and positive-semidefinite components relate to each other, and how these two terms contribute to the Hessian. The next part of our analysis shows that under certain conditions, relating to the richness of the policy class, the remainder term in the Hessian vanishes in the vicinity of a local optimum. Finally, we bound the behaviour of this remainder term in terms of the mixing time of the Markov chain induced by the policy parameters, where this part of the analysis is applicable over the entire parameter space. Given this analysis of the Hessian we then provide our local convergence analysis of the approximate Newton framework.Comment: This work has been removed because a more recent piece (A Gauss-Newton method for Markov Decision Processes, T. Furmston & G. Lever) of work has subsumed i

Conditional ergodicity in infinite dimension

The goal of this paper is to develop a general method to establish conditional ergodicity of infinite-dimensional Markov chains. Given a Markov chain in a product space, we aim to understand the ergodic properties of its conditional distributions given one of the components. Such questions play a fundamental role in the ergodic theory of nonlinear filters. In the setting of Harris chains, conditional ergodicity has been established under general nondegeneracy assumptions. Unfortunately, Markov chains in infinite-dimensional state spaces are rarely amenable to the classical theory of Harris chains due to the singularity of their transition probabilities, while topological and functional methods that have been developed in the ergodic theory of infinite-dimensional Markov chains are not well suited to the investigation of conditional distributions. We must therefore develop new measure-theoretic tools in the ergodic theory of Markov chains that enable the investigation of conditional ergodicity for infinite dimensional or weak-* ergodic processes. To this end, we first develop local counterparts of zero-two laws that arise in the theory of Harris chains. These results give rise to ergodic theorems for Markov chains that admit asymptotic couplings or that are locally mixing in the sense of H. F\"{o}llmer, and to a non-Markovian ergodic theorem for stationary absolutely regular sequences. We proceed to show that local ergodicity is inherited by conditioning on a nondegenerate observation process. This is used to prove stability and unique ergodicity of the nonlinear filter. Finally, we show that our abstract results can be applied to infinite-dimensional Markov processes that arise in several settings, including dissipative stochastic partial differential equations, stochastic spin systems and stochastic differential delay equations.Comment: Published in at http://dx.doi.org/10.1214/13-AOP879 the Annals of Probability (http://www.imstat.org/aop/) by the Institute of Mathematical Statistics (http://www.imstat.org

Gaussian Conditionally Markov Sequences: Theory with Application

Markov processes have been widely studied and used for modeling problems. A Markov process has two main components (i.e., an evolution law and an initial distribution). Markov processes are not suitable for modeling some problems, for example, the problem of predicting a trajectory with a known destination. Such a problem has three main components: an origin, an evolution law, and a destination. The conditionally Markov (CM) process is a powerful mathematical tool for generalizing the Markov process. One class of CM processes, called $CM_L$, fits the above components of trajectories with a destination. The CM process combines the Markov property and conditioning. The CM process has various classes that are more general and powerful than the Markov process, are useful for modeling various problems, and possess many Markov-like attractive properties. Reciprocal processes were introduced in connection to a problem in quantum mechanics and have been studied for years. But the existing viewpoint for studying reciprocal processes is not revealing and may lead to complicated results which are not necessarily easy to apply. We define and study various classes of Gaussian CM sequences, obtain their models and characterizations, study their relationships, demonstrate their applications, and provide general guidelines for applying Gaussian CM sequences. We develop various results about Gaussian CM sequences to provide a foundation and tools for general application of Gaussian CM sequences including trajectory modeling and prediction. We initiate the CM viewpoint to study reciprocal processes, demonstrate its significance, obtain simple and easy to apply results for Gaussian reciprocal sequences, and recommend studying reciprocal processes from the CM viewpoint. For example, we present a relationship between CM and reciprocal processes that provides a foundation for studying reciprocal processes from the CM viewpoint. Then, we obtain a model for nonsingular Gaussian reciprocal sequences with white dynamic noise, which is easy to apply. Also, this model is extended to the case of singular sequences and its application is demonstrated. A model for singular sequences has not been possible for years based on the existing viewpoint for studying reciprocal processes. This demonstrates the significance of studying reciprocal processes from the CM viewpoint

Spectral methods for bivariate Markov processes with diffusion and discrete components and a variant of the Wright-Fisher model

The aim of this paper is to study differential and spectral properties of the infinitesimal operator of two dimensional Markov processes with diffusion and discrete components. The infinitesimal operator is now a second-order differential operator with matrix-valued coefficients, from which we can derive backward and forward equations, a spectral representation of the probability density, study recurrence of the process and the corresponding invariant distribution. All these results are applied to an example coming from group representation theory which can be viewed as a variant of the Wright-Fisher model involving only mutation effects.Comment: 6 figure

Marginal process framework: A model reduction tool for Markov jump processes

Markov jump process models have many applications across science. Often, these models are defined on a state-space of product form and only one of the components of the process is of direct interest. In this paper, we extend the marginal process framework, which provides a marginal description of the component of interest, to the case of fully coupled processes. We use entropic matching to obtain a finite-dimensional approximation of the filtering equation, which governs the transition rates of the marginal process. The resulting equations can be seen as a combination of two projection operations applied to the full master equation, so that we obtain a principled model reduction framework. We demonstrate the resulting reduced description on the totally asymmetric exclusion process. An important class of Markov jump processes are stochastic reaction networks, which have applications in chemical and biomolecular kinetics, ecological models and models of social networks. We obtain a particularly simple instantiation of the marginal process framework for mass-action systems by using product-Poisson distributions for the approximate solution of the filtering equation. We investigate the resulting approximate marginal process analytically and numerically.Comment: 16 pages, 5 figures; accepted for publication in Physical Review