35 research outputs found
Approximating Euclidean by Imprecise Markov Decision Processes
Euclidean Markov decision processes are a powerful tool for modeling control
problems under uncertainty over continuous domains. Finite state imprecise,
Markov decision processes can be used to approximate the behavior of these
infinite models. In this paper we address two questions: first, we investigate
what kind of approximation guarantees are obtained when the Euclidean process
is approximated by finite state approximations induced by increasingly fine
partitions of the continuous state space. We show that for cost functions over
finite time horizons the approximations become arbitrarily precise. Second, we
use imprecise Markov decision process approximations as a tool to analyse and
validate cost functions and strategies obtained by reinforcement learning. We
find that, on the one hand, our new theoretical results validate basic design
choices of a previously proposed reinforcement learning approach. On the other
hand, the imprecise Markov decision process approximations reveal some
inaccuracies in the learned cost functions
Syntactic Markovian Bisimulation for Chemical Reaction Networks
In chemical reaction networks (CRNs) with stochastic semantics based on
continuous-time Markov chains (CTMCs), the typically large populations of
species cause combinatorially large state spaces. This makes the analysis very
difficult in practice and represents the major bottleneck for the applicability
of minimization techniques based, for instance, on lumpability. In this paper
we present syntactic Markovian bisimulation (SMB), a notion of bisimulation
developed in the Larsen-Skou style of probabilistic bisimulation, defined over
the structure of a CRN rather than over its underlying CTMC. SMB identifies a
lumpable partition of the CTMC state space a priori, in the sense that it is an
equivalence relation over species implying that two CTMC states are lumpable
when they are invariant with respect to the total population of species within
the same equivalence class. We develop an efficient partition-refinement
algorithm which computes the largest SMB of a CRN in polynomial time in the
number of species and reactions. We also provide an algorithm for obtaining a
quotient network from an SMB that induces the lumped CTMC directly, thus
avoiding the generation of the state space of the original CRN altogether. In
practice, we show that SMB allows significant reductions in a number of models
from the literature. Finally, we study SMB with respect to the deterministic
semantics of CRNs based on ordinary differential equations (ODEs), where each
equation gives the time-course evolution of the concentration of a species. SMB
implies forward CRN bisimulation, a recently developed behavioral notion of
equivalence for the ODE semantics, in an analogous sense: it yields a smaller
ODE system that keeps track of the sums of the solutions for equivalent
species.Comment: Extended version (with proofs), of the corresponding paper published
at KimFest 2017 (http://kimfest.cs.aau.dk/
Probabilistic Reachability for Parametric Markov Models
Abstract. Given a parametric Markov model, we consider the problem of computing the formula expressing the probability of reaching a given set of states. To attack this principal problem, Daws has suggested to first convert the Markov chain into a finite automaton, from which a regular expression is computed. Afterwards, this expression is evaluated to a closed form expression representing the reachability probability. This paper investigates how this idea can be turned into an effective procedure. It turns out that the bottleneck lies in an exponential growth of the regular expression relative to the number of states. We therefore proceed differently, by tightly intertwining the regular expression computation with its evaluation. This allows us to arrive at an effective method that avoids the exponential blow up in most practical cases. We give a detailed account of the approach, also extending to parametric models with rewards and with non-determinism. Experimental evidence is provided, illustrating that our implementation provides meaningful insights on non-trivial models.
Efficient Syntax-Driven Lumping of Differential Equations
We present an algorithm to compute exact aggregations of a class of systems of ordinary differential equations (ODEs). Our approach consists in an extension of Paige and Tarjan鈥檚 seminal solution to the coarsest refinement problem by encoding an ODE system into a suitable discrete-state representation. In particular, we consider a simple extension of the syntax of elementary chemical reaction networks because (i) it can express ODEs with derivatives given by polynomials of degree at most two, which are relevant in many applications in natural sciences and engineering; and (ii) we can build on two recently introduced bisimulations, which yield two complementary notions of ODE lumping. Our algorithm computes the largest bisimulations in O(r鈰卻鈰卨ogs)O(r鈰卻鈰卨og鈦) time, where r is the number of monomials and s is the number of variables in the ODEs. Numerical experiments on real-world models from biochemistry, electrical engineering, and structural mechanics show that our prototype is able to handle ODEs with millions of variables and monomials, providing significant model reductions
Probabilistic timing covert channels: to close or not to close?
We develop a new notion of security against timing attacks where the attacker is able to simultaneously observe the execution time of a program and the probability of the values of low variables. We then propose an algorithm which computes an estimate of the security of a program with respect to this notion in terms of timing leakage and show how to use this estimate for cost optimization
Probabilistic Reachability for Parametric Markov Models
Abstract. Given a parametric Markov model, we consider the problem of computing the rational function expressing the probability of reaching a given set of states. To attack this principal problem, Daws has suggested to first convert the Markov chain into a finite automaton, from which a regular expression is computed. Afterwards, this expression is evaluated to a closed form function representing the reachability probability. This paper investigates how this idea can be turned into an effective procedure. It turns out that the bottleneck lies in the growth of the regular expression relative to the number of states (n螛(logn)). We therefore proceed differently, by tightly intertwining the regular expression computation with its evaluation. This allows us to arrive at an effective method that avoids this blow up in most practical cases. We give a detailed account of the approach, also extending to parametric models with rewards and with non-determinism. Experimental evidence is provided, illustrating that our implementation provides meaningful insights on non-trivial models.
Exact analysis of summary statistics for continuous-time discrete-state Markov processes on networks using graph-automorphism lumping
We propose a unified framework to represent a wide range of continuous-time discrete-state Markov processes on networks, and show how many network dynamics models in the literature can be represented in this unified framework. We show how a particular sub-set of these models, referred to here as single-vertex-transition (SVT) processes, lead to the analysis of quasi-birth-and-death (QBD) processes in the theory of continuous-time Markov chains. We illustrate how to analyse a number of summary statistics for these processes, such as absorption probabilities and first-passage times. We extend the graph-automorphism lumping approach [Kiss, Miller, Simon, Mathematics of Epidemics on Networks, 2017; Simon, Taylor, Kiss, J. Math. Bio. 62(4), 2011], by providing a matrix-oriented representation of this technique, and show how it can be applied to a very wide range of dynamical processes on networks. This approach can be used not only to solve the master equation of the system, but also to analyse the summary statistics of interest. We also show the interplay between the graph-automorphism lumping approach and the QBD structures when dealing with SVT processes. Finally, we illustrate our theoretical results with examples from the areas of opinion dynamics and mathematical epidemiology