Reachability in Parametric Interval Markov Chains using Constraints

Parametric Interval Markov Chains (pIMCs) are a specification formalism that extend Markov Chains (MCs) and Interval Markov Chains (IMCs) by taking into account imprecision in the transition probability values: transitions in pIMCs are labeled with parametric intervals of probabilities. In this work, we study the difference between pIMCs and other Markov Chain abstractions models and investigate the two usual semantics for IMCs: once-and-for-all and at-every-step. In particular, we prove that both semantics agree on the maximal/minimal reachability probabilities of a given IMC. We then investigate solutions to several parameter synthesis problems in the context of pIMCs -- consistency, qualitative reachability and quantitative reachability -- that rely on constraint encodings. Finally, we propose a prototype implementation of our constraint encodings with promising results

POMDPs under Probabilistic Semantics

We consider partially observable Markov decision processes (POMDPs) with limit-average payoff, where a reward value in the interval [0,1] is associated to every transition, and the payoff of an infinite path is the long-run average of the rewards. We consider two types of path constraints: (i) quantitative constraint defines the set of paths where the payoff is at least a given threshold lambda_1 in (0,1]; and (ii) qualitative constraint which is a special case of quantitative constraint with lambda_1=1. We consider the computation of the almost-sure winning set, where the controller needs to ensure that the path constraint is satisfied with probability 1. Our main results for qualitative path constraint are as follows: (i) the problem of deciding the existence of a finite-memory controller is EXPTIME-complete; and (ii) the problem of deciding the existence of an infinite-memory controller is undecidable. For quantitative path constraint we show that the problem of deciding the existence of a finite-memory controller is undecidable.Comment: Appears in Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence (UAI2013

Estimating the Sampling Error: Distribution of Transition Matrices and Functions of Transition Matrices for Given Trajectory Data

The problem of estimating a Markov transition matrix to statistically describe the dynamics underlying an observed process is frequently found in the physical and economical sciences. However, little attention has been paid to the fact that such an estimation is associated with statistical uncertainty, which depends on the number of observed transitions between metastable states. In turn, this induces uncertainties in any property computed from the transition matrix, such as stationary probabilities, committor probabilities, or eigenvalues. Assessing these uncertainties is essential for testing the reliability of a given observation and also, if possible, to plan further simulations or measurements in such a way that the most serious uncertainties will be reduced with minimal effort. Here, a rigorous statistical method is proposed to approximate the complete statistical distribution of functions of the transition matrix provided that one can identify discrete states such that the transition process between them may be modeled with a memoryless jump process, i.e., Markov dynamics. The method is based on sampling the statistical distribution of Markov transition matrices that is induced by the observed transition events. It allows the constraint of reversibility to be included, which is physically meaningful in many applications. The method is illustrated on molecular dynamics simulations of a hexapeptide that are modeled by a Markov transition process between the metastable states. For this model the distributions and uncertainties of the stationary probabilities of metastable states, the transition matrix elements, the committor probabilities, and the transition matrix eigenvalues are estimated. It is found that the detailed balance constraint can significantly alter the distribution of some observables

The Fastest Mixing Markov Process on a Graph and a Connection to a Maximum Variance Unfolding Problem

We consider a Markov process on a connected graph, with edges labeled with transition rates between the adjacent vertices. The distribution of the Markov process converges to the uniform distribution at a rate determined by the second smallest eigenvalue lambda_2 of the Laplacian of the weighted graph. In this paper we consider the problem of assigning transition rates to the edges so as to maximize lambda_2 subject to a linear constraint on the rates. This is the problem of finding the fastest mixing Markov process (FMMP) on the graph. We show that the FMMP problem is a convex optimization problem, which can in turn be expressed as a semidefinite program, and therefore effectively solved numerically. We formulate a dual of the FMMP problem and show that it has a natural geometric interpretation as a maximum variance unfolding (MVU) problem, , the problem of choosing a set of points to be as far apart as possible, measured by their variance, while respecting local distance constraints. This MVU problem is closely related to a problem recently proposed by Weinberger and Saul as a method for "unfolding" high-dimensional data that lies on a low-dimensional manifold. The duality between the FMMP and MVU problems sheds light on both problems, and allows us to characterize and, in some cases, find optimal solutions

The space of positive transition measures on a Markov chain

Information geometry of Markov chains has been studied by Nagaoka, Takeuchi and others using the dually flat structure of the space of transition probabilities. In this context, a submanifold of the space is called a Markov model. In the present paper, we seek for a theory of extended spaces of Markov models in the following sense. As a prototype, for the space of probability distributions on a finite set, Amari has introduced the space of positive measures simply by removing the constraint condition that the total mass is equal to $1$ and investigated the extended space by finding the Bregman and $F$-divergence suitably. According to this line, we introduce an extension of the space of transition probabilities equipped with suitable $F$-divergence for a given Markov chain. We regard it as the space of positive transition measures on a Markov chain, and study the dually flat structure on the space. That provides a new insight on the geometry of Markov chains. We also discuss a relation with other existing work.Comment: 12 page

Markov Chain Modeling of Polymer Translocation Through Pores

We solve the Chapman-Kolmogorov equation and study the exact splitting probabilities of the general stochastic process which describes polymer translocation through membrane pores within the broad class of Markov chains. Transition probabilities which satisfy a specific balance constraint provide a refinement of the Chuang-Kantor-Kardar relaxation picture of translocation, allowing us to investigate finite size effects in the evaluation of dynamical scaling exponents. We find that (i) previous Langevin simulation results can be recovered only if corrections to the polymer mobility exponent are taken into account and that (ii) the dynamical scaling exponents have a slow approach to their predicted asymptotic values as the polymer's length increases. We also address, along with strong support from additional numerical simulations, a critical discussion which points in a clear way the viability of the Markov chain approach put forward in this work.Comment: 17 pages, 5 figure

Applying shape and phase restrictions in generalized dynamic categorical models of the business cycle

To match the NBER business cycle features it is necessary to employ Generalised dynamic categorical (GDC) models that impose certain phase restrictions and permit multiple indexes. Theory suggests additional shape restrictions in the form of monotonicity and boundedness of certain transition probabilities. Maximum likelihood and constraint weighted bootstrap estimators are developed to impose these restrictions. In the application these estimators generate improved estimates of how the probability of recession varies with the yield spread.Generalized dynamic categorical model, Business cycle; binary variable, Markov process, probit model, yield curve