Property-driven State-Space Coarsening for Continuous Time Markov Chains

Dynamical systems with large state-spaces are often expensive to thoroughly explore experimentally. Coarse-graining methods aim to define simpler systems which are more amenable to analysis and exploration; most current methods, however, focus on a priori state aggregation based on similarities in transition rates, which is not necessarily reflected in similar behaviours at the level of trajectories. We propose a way to coarsen the state-space of a system which optimally preserves the satisfaction of a set of logical specifications about the system's trajectories. Our approach is based on Gaussian Process emulation and Multi-Dimensional Scaling, a dimensionality reduction technique which optimally preserves distances in non-Euclidean spaces. We show how to obtain low-dimensional visualisations of the system's state-space from the perspective of properties' satisfaction, and how to define macro-states which behave coherently with respect to the specifications. Our approach is illustrated on a non-trivial running example, showing promising performance and high computational efficiency.Comment: 16 pages, 6 figures, 1 tabl

Derivation of mean-field equations for stochastic particle systems

We study stochastic particle systems on a complete graph and derive effective mean-field rate equations in the limit of diverging system size, which are also known from cluster aggregation models. We establish the propagation of chaos under generic growth conditions on particle jump rates, and the limit provides a master equation for the single site dynamics of the particle system, which is a non-linear birth death chain. Conservation of mass in the particle system leads to conservation of the first moment for the limit dynamics, and to non-uniqueness of stationary distributions. Our findings are consistent with recent results on exchange driven growth, and provide a connection between the well studied phenomena of gelation and condensation.Comment: 26 page

Persistence and First-Passage Properties in Non-equilibrium Systems

In this review we discuss the persistence and the related first-passage properties in extended many-body nonequilibrium systems. Starting with simple systems with one or few degrees of freedom, such as random walk and random acceleration problems, we progressively discuss the persistence properties in systems with many degrees of freedom. These systems include spins models undergoing phase ordering dynamics, diffusion equation, fluctuating interfaces etc. Persistence properties are nontrivial in these systems as the effective underlying stochastic process is non-Markovian. Several exact and approximate methods have been developed to compute the persistence of such non-Markov processes over the last two decades, as reviewed in this article. We also discuss various generalisations of the local site persistence probability. Persistence in systems with quenched disorder is discussed briefly. Although the main emphasis of this review is on the theoretical developments on persistence, we briefly touch upon various experimental systems as well.Comment: Review article submitted to Advances in Physics: 149 pages, 21 Figure

Dynamics of condensation in the totally asymmetric inclusion process

We study the dynamics of condensation of the inclusion process on a one-dimensional periodic lattice in the thermodynamic limit, generalising recent results on finite lattices for symmetric dynamics. Our main focus is on totally asymmetric dynamics which have not been studied before, and which we also compare to exact solutions for symmetric systems. We identify all relevant dynamical regimes and corresponding time scales as a function of the system size, including a coarsening regime where clusters move on the lattice and exchange particles, leading to a growing average cluster size. Suitable observables exhibit a power law scaling in this regime before they saturate to stationarity following an exponential decay depending on the system size. Our results are based on heuristic derivations and exact computations for symmetric systems, and are supported by detailed simulation data.Comment: 23 pages, 6 figures, updated references and introductio

Metastability of finite state Markov chains: a recursive procedure to identify slow variables for model reduction

Consider a sequence $(\eta^N(t) :t\ge 0)$ of continuous-time, irreducible Markov chains evolving on a fixed finite set $E$, indexed by a parameter $N$. Denote by $R_N(\eta,\xi)$ the jump rates of the Markov chain $\eta^N_t$, and assume that for any pair of bonds $(\eta,\xi)$, $(\eta',\xi')$ $\arctan \{R_N(\eta,\xi)/R_N(\eta',\xi')\}$ converges as $N\uparrow\infty$. Under a hypothesis slightly more restrictive (cf. \eqref{mhyp} below), we present a recursive procedure which provides a sequence of increasing time-scales \theta^1_N, \dots, \theta^{\mf p}_N, $\theta^j_N \ll \theta^{j+1}_N$, and of coarsening partitions \{\ms E^j_1, \dots, \ms E^j_{\mf n_j}, \Delta^j\}, 1\le j\le \mf p, of the set $E$. Let \phi_j: E \to \{0,1, \dots, \mf n_j\} be the projection defined by \phi_j(\eta) = \sum_{x=1}^{\mf n_j} x \, \mb 1\{\eta \in \ms E^j_x\}. For each 1\le j\le \mf p, we prove that the hidden Markov chain $X^j_N(t) = \phi_j(\eta^N(t\theta^j_N))$ converges to a Markov chain on \{1, \dots, \mf n_j\}

Condensation in stochastic particle systems with stationary product measures

We study stochastic particle systems with stationary product measures that exhibit a condensation transition due to particle interactions or spatial inhomogeneities. We review previous work on the stationary behaviour and put it in the context of the equivalence of ensembles, providing a general characterization of the condensation transition for homogeneous and inhomogeneous systems in the thermodynamic limit. This leads to strengthened results on weak convergence for subcritical systems, and establishes the equivalence of ensembles for spatially inhomogeneous systems under very general conditions, extending previous results which were focused on attractive and finite systems. We use relative entropy techniques which provide simple proofs, making use of general versions of local limit theorems for independent random variables.Comment: 44 pages, 4 figures; improved figures and corrected typographical error

Metastability in a condensing zero-range process in the thermodynamic limit

Zero-range processes with decreasing jump rates are known to exhibit condensation, where a finite fraction of all particles concentrates on a single lattice site when the total density exceeds a critical value. We study such a process on a one-dimensional lattice with periodic boundary conditions in the thermodynamic limit with fixed, super-critical particle density. We show that the process exhibits metastability with respect to the condensate location, i.e. the suitably accelerated process of the rescaled location converges to a limiting Markov process on the unit torus. This process has stationary, independent increments and the rates are characterized by the scaling limit of capacities of a single random walker on the lattice. Our result extends previous work for fixed lattices and diverging density in [J. Beltran, C. Landim, Probab. Theory Related Fields, 152(3-4):781-807, 2012], and we follow the martingale approach developed there and in subsequent publications. Besides additional technical difficulties in estimating error bounds for transition rates, the thermodynamic limit requires new estimates for equilibration towards a suitably defined distribution in metastable wells, corresponding to a typical set of configurations with a particular condensate location. The total exit rates from individual wells turn out to diverge in the limit, which requires an intermediate regularization step using the symmetries of the process and the regularity of the limit generator. Another important novel contribution is a coupling construction to provide a uniform bound on the exit rates from metastable wells, which is of a general nature and can be adapted to other models.Comment: 55 pages, 1 figur

Time scale separation and dynamic heterogeneity in the low temperature East model

We consider the non-equilibrium dynamics of the East model, a linear chain of 0-1 spins evolving under a simple Glauber dynamics in the presence of a kinetic constraint which forbids flips of those spins whose left neighbor is 1. We focus on the glassy effects caused by the kinetic constraint as $q\downarrow 0$, where $q$ is the equilibrium density of the 0's. In the physical literature this limit is equivalent to the zero temperature limit. We first prove that, for any given $L=O(1/q)$, the divergence as $q\downarrow 0$ of three basic characteristic time scales of the East process of length $L$ is the same. Then we examine the problem of dynamic heterogeneity, i.e. non-trivial spatio-temporal fluctuations of the local relaxation to equilibrium, one of the central aspects of glassy dynamics. For any mesoscopic length scale $L=O(q^{-\gamma})$, $\gamma<1$, we show that the characteristic time scale of two East processes of length $L$ and $\lambda L$ respectively are indeed separated by a factor $q^{-a}$, $a=a(\gamma)>0$, provided that $\lambda \geq 2$ is large enough (independent of $q$, $\lambda=2$ for $\gamma<1/2$). In particular, the evolution of mesoscopic domains, i.e. maximal blocks of the form $111..10$, occurs on a time scale which depends sharply on the size of the domain, a clear signature of dynamic heterogeneity. A key result for this part is a very precise computation of the relaxation time of the chain as a function of $(q,L)$, well beyond the current knowledge, which uses induction on length scales on one hand and a novel algorithmic lower bound on the other. Finally we show that no form of time scale separation occurs for $\gamma=1$, i.e. at the equilibrium scale $L=1/q$, contrary to what was assumed in the physical literature based on numerical simulations.Comment: 40 pages, 4 figures; minor typographical corrections and improvement

Multiscale Markov Decision Problems: Compression, Solution, and Transfer Learning

Many problems in sequential decision making and stochastic control often have natural multiscale structure: sub-tasks are assembled together to accomplish complex goals. Systematically inferring and leveraging hierarchical structure, particularly beyond a single level of abstraction, has remained a longstanding challenge. We describe a fast multiscale procedure for repeatedly compressing, or homogenizing, Markov decision processes (MDPs), wherein a hierarchy of sub-problems at different scales is automatically determined. Coarsened MDPs are themselves independent, deterministic MDPs, and may be solved using existing algorithms. The multiscale representation delivered by this procedure decouples sub-tasks from each other and can lead to substantial improvements in convergence rates both locally within sub-problems and globally across sub-problems, yielding significant computational savings. A second fundamental aspect of this work is that these multiscale decompositions yield new transfer opportunities across different problems, where solutions of sub-tasks at different levels of the hierarchy may be amenable to transfer to new problems. Localized transfer of policies and potential operators at arbitrary scales is emphasized. Finally, we demonstrate compression and transfer in a collection of illustrative domains, including examples involving discrete and continuous statespaces.Comment: 86 pages, 15 figure