Maximizing the probability of attaining a target prior to extinction

We present a dynamic programming-based solution to the problem of maximizing the probability of attaining a target set before hitting a cemetery set for a discrete-time Markov control process. Under mild hypotheses we establish that there exists a deterministic stationary policy that achieves the maximum value of this probability. We demonstrate how the maximization of this probability can be computed through the maximization of an expected total reward until the first hitting time to either the target or the cemetery set. Martingale characterizations of thrifty, equalizing, and optimal policies in the context of our problem are also established.Comment: 22 pages, 1 figure. Revise

New insights on stochastic reachability

In this paper, we give new characterizations of the stochastic reachability problem for stochastic hybrid systems in the language of different theories that can be employed in studying stochastic processes (Markov processes, potential theory, optimal control). These characterizations are further used to obtain the probabilities involved in the context of stochastic reachability as viscosity solutions of some variational inequalities

Coupling, local times, immersions

This paper answers a question of \'{E}mery [In S\'{e}minaire de Probabilit\'{e}s XLII (2009) 383-396 Springer] by constructing an explicit coupling of two copies of the Bene\v{s} et al. [In Applied Stochastic Analysis (1991) 121-156 Gordon & Breach] diffusion (BKR diffusion), neither of which starts at the origin, and whose natural filtrations agree. The paper commences by surveying probabilistic coupling, introducing the formal definition of an immersed coupling (the natural filtration of each component is immersed in a common underlying filtration; such couplings have been described as co-adapted or Markovian in older terminologies) and of an equi-filtration coupling (the natural filtration of each component is immersed in the filtration of the other; consequently the underlying filtration is simultaneously the natural filtration for each of the two coupled processes). This survey is followed by a detailed case-study of the simpler but potentially thematic problem of coupling Brownian motion together with its local time at $0$. This problem possesses its own intrinsic interest as well as being closely related to the BKR coupling construction. Attention focusses on a simple immersed (co-adapted) coupling, namely the reflection/synchronized coupling. It is shown that this coupling is optimal amongst all immersed couplings of Brownian motion together with its local time at $0$, in the sense of maximizing the coupling probability at all possible times, at least when not started at pairs of initial points lying in a certain singular set. However numerical evidence indicates that the coupling is not a maximal coupling, and is a simple but non-trivial instance for which this distinction occurs. It is shown how the reflection/synchronized coupling can be converted into a successful equi-filtration coupling, by modifying the coupling using a deterministic time-delay and then by concatenating an infinite sequence of such modified couplings. The construction of an explicit equi-filtration coupling of two copies of the BKR diffusion follows by a direct generalization, although the proof of success for the BKR coupling requires somewhat more analysis than in the local time case.Comment: Published at http://dx.doi.org/10.3150/14-BEJ596 in the Bernoulli (http://isi.cbs.nl/bernoulli/) by the International Statistical Institute/Bernoulli Society (http://isi.cbs.nl/BS/bshome.htm

Comparing hitting time behaviour of Markov jump processes and their diffusion approximations

Markov jump processes can provide accurate models in many applications, notably chemical and biochemical kinetics, and population dynamics. Stochastic differential equations offer a computationally efficient way to approximate these processes. It is therefore of interest to establish results that shed light on the extent to which the jump and diffusion models agree. In this work we focus on mean hitting time behavior in a thermodynamic limit. We study three simple types of reactions where analytical results can be derived, and we find that the match between mean hitting time behavior of the two models is vastly different in each case. In particular, for a degradation reaction we find that the relative discrepancy decays extremely slowly, namely, as the inverse of the logarithm of the system size. After giving some further computational results, we conclude by pointing out that studying hitting times allows the Markov jump and stochastic differential equation regimes to be compared in a manner that avoids pitfalls that may invalidate other approaches