Adaptive aggregation methods for infinite horizon dynamic programming

"July 1988."Includes bibliographical references.Work supported by the Office of Naval Research under contract N00014-84-C-0577by Dimitri P. Bertsekas, David A. Castañon

On the effects of using the Grassmann-Taksar-Heyman method in iterative aggregation-disaggregation

Iterative aggregation-disaggregation (IAD) is an effective method for solving finite nearly completely decomposable (NCD) Markov chains. Small perturbations in the transition probabilities of these chains may lead to considerable changes in the stationary probabilities; NCD Markov chains are known to be ill-conditioned. During an IAD step, this undesirable condition is inherited by the coupling matrix and one confronts the problem of finding the stationary probabilities of a stochastic matrix whose diagonal elements are close to 1. In this paper, the effects of using the Grassmann-Taksar-Heyman (GTH) method to solve the coupling matrix formed in the aggregation step are investigated. Then the idea is extended in such a way that the same direct method can be incorporated into the disaggregation step. Finally, the effects of using the GTH method in the IAD algorithm on various examples are demonstrated, and the conditions under which it should be employed are explained

Stable computation of probability densities for metastable dynamical systems

Whenever the invariant stationary density of metastable dynamical systems decomposes into almost invariant partial densities, its computation as eigenvector of some transition probability matrix is an ill-conditioned problem. In order to avoid this computational difficulty, we suggest applying an aggregation/disaggregation method which addresses only well-conditioned subproblems aud thus results in a stable algorithm. In contrast to existing methods, the aggregation step is done via a sampling algorithm which covers only small patches of the sampling space. Finally, the theoretical analysis is illustrated by two biomolecular examples

Comparison of partitioning techniques for two-level iterative solvers on large, sparse Markov chains

Experimental results for large, sparse Markov chains, especially the ill-conditioned nearly completely decomposable (NCD) ones, are few. We believe there is need for further research in this area, specifically to aid in the understanding of the effects of the degree of coupling of NCD Markov chains and their nonzero structure on the convergence characteristics and space requirements of iterative solvers. The work of several researchers has raised the following questions that led to research in a related direction: How must one go about partitioning the global coefficient matrix into blocks when the system is NCD and a two-level iterative solver (such as block SOR) is to be employed? Are block partitionings dictated by the NCD form of the stochastic one-step transition probability matrix necessarily superior to others? Is it worth investigating alternative partitionings? Better yet, for a fixed labeling and partitioning of the states, how does the performance of block SOR (or even that of point SOR) compare to the performance of the iterative aggregation-disaggregation (IAD) algorithm? Finally, is there any merit in using two-level iterative solvers when preconditioned Krylov subspace methods are available? We seek answers to these questions on a test suite of 13 Markov chains arising in 7 applications

Markov chain models of instantaneously coupled intracellular calcium channels

Localized calcium elevations known as calcium puffs or sparks are cellular signals arising from cooperative activity of clusters of inositol 1,4,5-trisphosphate receptors (IP3Rs) or ryanodine receptors (RyRs) located at calcium release sites on the endoplasmic or sarcoplasmic reticulum membrane. When Markov chain models of these intracellular calcium-regulated calcium channels are coupled via a mathematical representation of the calcium microdomain, simulated calcium release sites may exhibit the phenomenon of stochastic calcium excitability where the IP3Rs or RyRs open and close in a concerted fashion. Although the biophysical theory relating the kinetics of single channels to the collective phenomena of puffs and sparks is only beginning to be developed, Markov chain models of coupled intracellular channels give insight into the dynamics of calcium puffs and sparks.;Interestingly, under some conditions simulated puffs and sparks can be observed even when the single channel model used does not include slow calcium inactivation or any long-lived closed state. In this case termination of the localized calcium elevation occurs when all of the intracellular channels at a release site simultaneously close through a process called stochastic attrition. This dissertation investigates the statistical properties of stochastic attrition viewed as an absorption time on a terminating Markov chain that represents a calcium release site composed of two-state channels that are activated by calcium. Assuming that the local calcium concentration experienced by a channel depends only on the number of open channels at the calcium release site, the probability distribution function for the time until stochastic attrition occurs is derived and an analytical formula for the expectation of this random variable is presented. Also explored is how the contribution of stochastic attrition to the termination of calcium puffs and sparks depends on the number of channels at a release site, the source amplitude of the channels, the background calcium concentration, channel kinetics, and the cooperativity of calcium binding.;This dissertation also studies whether single channel models with calcium inactivation are less sensitive to the details of release site ultrastructure than models that lack a slow calcium-inactivation process. Release site dynamics obtained from simulated calcium release sites composed of instantaneously coupled calcium-regulated calcium channels whose random spatial locations were chosen from a uniform distribution on a disc of specified radius are compared to simulations with channels arranged on hexagonal lattices. Analysis of puff/spark statistics confirms that puffs and sparks are less sensitive to the spatial organization of release sites when the single channel model includes a slow inactivation process. The validity of several different mean-field reductions that do not explicitly account for the details of release site ultrastructure is also investigated.;Calcium release site models are stochastic automata networks that involve many functional transitions, that is, the transition probabilities of each channel depend on the local calcium concentration and thus the state of the other channels. A Kronecker structured representation for calcium release site models is presented and benchmark stationary distribution calculations using both exact and approximate iterative numerical solution techniques that leverage this structure are performed. When it is possible to obtain an exact solution, response measures such as the number of channels in a particular state converge more quickly using the iterative numerical methods than occupation measures calculated via Monte Carlo simulation. When an exact solution is not feasible, iterative approximate methods based on the Power method may be used, with performance similar to Monte Carlo estimates

Decompositional analysis of Kronecker structured Markov chains

This contribution proposes a decompositional iterative method with low memory requirements for the steadystate analysis ofKronecker structured Markov chains. The Markovian system is formed by a composition of subsystems using the Kronecker sum operator for local transitions and the Kronecker product operator for synchronized transitions. Even though the interactions among subsystems, which are captured by synchronized transitions, need not be weak, numerical experiments indicate that the solver benefits considerably from weak interactions among subsystems, and is to be recommended specifically in this case. © 2008, Kent State University

Nearly reducible finite Markov chains: theory and algorithms

Finite Markov chains are probabilistic network models that are commonly used as representations of dynamical processes in the physical sciences, biological sciences, economics, and elsewhere. Markov chains that appear in realistic modelling tasks are frequently observed to be nearly reducible, incorporating a mixture of fast and slow processes that leads to ill-conditioning of the underlying matrix of probabilities for transitions between states. Hence, the wealth of established theoretical results that makes Markov chains attractive and convenient models often cannot be used straightforwardly in practice, owing to numerical instability associated with the standard computational procedures to evaluate the expressions. This work is concerned with the development of theory, algorithms, and simulation methods for the efficient and numerically stable analysis of finite Markov chains, with a primary focus on exact approaches that are robust and therefore applicable to nearly reducible networks. New methodologies are presented to determine representative paths, identify the dominant transition mechanisms for a particular process of interest, and analyze the local states that have a strong influence on the characteristics of the global dynamics. The novel approaches yield new insights into the behaviour of Markovian networks, addressing and overcoming numerical challenges. The methodology is applied to example models that are relevant to current problems in chemical physics, including Markov chains representing a protein folding transition, and a configurational transition in an atomic cluster. Relevant classical theory of finite Markov chains and a description of existing robust algorithms for their numerical analysis is given in Chapter 1. The remainder of this thesis considers the problem of investigating a transition from an initial set of states in a Markovian network to an absorbing (target) macrostate. A formal approach to determine a finite set of representative transition paths is proposed in Chapter 2, based on exact pathwise decomposition of the total productive flux. This analysis allows for the importance of competing dynamical processes to be rigorously quantified. A robust state reduction algorithm to compute the expectation of any path property for a transition between two endpoint states is also described in Chapter 2. Chapter 3 reports further numerically stable state reduction algorithms to compute quantities that characterize the features of a transition at a statewise level of detail, allowing for identification of the local states that play a key role in modulating the slow dynamics. An expression is derived for the probability that a state is visited on a path that proceeds directly to the absorbing state without revisiting the initial state, which characterizes the dynamical relevance of an individual state to the overall transition process. In Chapter 4, an unsupervised strategy is proposed to utilize a highly efficient simulation algorithm for sampling paths on a Markov chain. The framework employs a scalable community detection algorithm to obtain an initial clustering of the network into metastable sets of states, which is subsequently refined by a variational optimization procedure. The optimized clustering is then used as the basis for simulating trajectory segments that necessarily escape from the metastable macrostates. The thesis is concluded with an overview of recent related advances that are beyond the scope of the current work (Chapter 5), and a discussion of potential applications where the novel methodology reported herein may be applied to perform insightful analyses that were previously intractable.Cambridge Commonwealth, European and International Trust Engineering and Physical Sciences Research Counci

Quasi lumpability, lower-bounding coupling matrices, and nearly completely decomposable Markov chains

In this paper, it is shown that nearly completely decomposable (NCD) Markov chains are quasi-lumpable. The state space partition is the natural one, and the technique may be used to compute lower and upper bounds on the stationary probability of each NCD block. In doing so, a lower-bounding nonnegative coupling matrix is employed. The nature of the stationary probability bounds is closely related to the structure of this lower-bounding matrix. Irreducible lower-bounding matrices give tighter bounds compared with bounds obtained using reducible lower-bounding matrices. It is also noticed that the quasi-lumped chain of an NCD Markov chain is an ill-conditioned matrix and the bounds obtained generally will not be tight. However, under some circumstances, it is possible to compute the stationary probabilities of some NCD blocks exactly

Experiments with two-stage iterative solvers and preconditioned Krylov subspace methods on nearly completely decomposable Markov chains

Ankara : Department of Computer Engineering and Information Science and the Institute of Engineering and Science of Bilkent University, 1997.Thesis (Master's) -- Bilkent University, 1997.Includes bibliographical references leaves 121-124Gueaieb, WailM.S