The cover time of a Markov chain on a finite state space is the expected time
until all states are visited. We show that if the cover time of a discrete-time
Markov chain with rational transitions probabilities is bounded, then it is a
rational number. The result is proved by relating the cover time of the
original chain to the hitting time of a set in another higher dimensional
chain. We also extend this result to the setting where $k\geq 1 $ independent
copies of a Markov chain are run simultaneously on the same state space and the
cover time is the expected time until each state has been visited by at least
one copy of the chain.Comment: 8 pages, 1 figure. The Proof of Proposition 8 has been simplifie

Sylvester, John

English

arXiv

The cover time of a Markov chain on a finite state space is the expected time until all states are visited. We show that if the cover time of a discrete-time Markov chain with rational transitions probabilities is bounded, then it is a rational number. The result is proved by relating the cover time of the original chain to the hitting time of a set in another higher dimensional chain. We prove this result in a more general setting where k ≥ 1 independent copies of a Markov chain are run simultaneously on the same state space

Enlighten

Statistics and Probability Letters 187 (2022) 109534LisM(nsTEiafh0lContents lists available at ScienceDirectStatistics and Probability Lettersjournal homepage: www.elsevier.com/locate/staproThe cover time of a (multiple)Markov chainwith rationaltransition probabilities is rationalJohn SylvesterSchool of Computing Science, University of Glasgow, Glasgow, UKa r t i c l e i n f oArticle history:Received 23 March 2021Received in revised form 22 March 2022Accepted 26 April 2022Available online 7 May 2022Keywords:Cover timeMarkov chainRational numberMultiple random walka b s t r a c tThe cover time of a Markov chain on a finite state space is the expected time until allstates are visited. We show that if the cover time of a discrete-time Markov chain withrational transitions probabilities is bounded, then it is a rational number. The result isproved by relating the cover time of the original chain to the hitting time of a set inanother higher dimensional chain. We prove this result in a more general setting wherek ≥ 1 independent copies of a Markov chain are run simultaneously on the same statespace.CrownCopyright© 2022 Published by Elsevier B.V. This is an open access article under theCC BY license (http://creativecommons.org/licenses/by/4.0/).1. Introduction and resultsLet (Xt )t≥0 be a discrete-time Markov chain with transition matrix P on a state space Ω , see Aldous and Fill (2002),evin and Peres (2017) for background. We say a chain is rational if all its transition probabilities are rational numbers,.e. P(x, y) ∈ Q for all x, y ∈ Ω . The stopping time τcov is the first time all states are visited, that isτcov := inf{t ≥ 0 :t⋃k=0{Xk} = Ω}.For x ∈ Ω , let Ex [ τcov ] = E [ τcov | X0 = x ] be the cover time from x, that is, the expected time for the chain to visit alltates when started from x ∈ Ω .Along with mixing and hitting times, the cover time is one of the most natural and well studied stopping times for aarkov chain and has found applications in the analysis of algorithms, see for example Alon et al. (2011), Aldous and Fill2002, Ch. 6.8) and Levin and Peres (2017, Ch. 11). It is clear that the stopping time τcov is a natural number, however it isot so clear whether the cover time Ex [ τcov ] is rational, even if the transition probabilities are rational. Our main resulthows that, under some natural assumptions, the cover time of a rational Markov chain is rational.heorem 1. Let (Xt )t≥0 be a discrete-time rational Markov chain on a finite state space Ω . Then, for any x ∈ Ω such thatx[τcov] < ∞, we have Ex[τcov] ∈ Q.The assumption that Ω is finite is necessary to ensure the cover time is bounded. Recall that a Markov chain isrreducible if for every x, y ∈ Ω there exists some t ≥ 0 such that Pt (x, y) > 0, where Pt (x, y) denotes the probabilitychain started at x is at state y after t ≥ 1 steps. Theorem 1 does not require irreducibility, just that the cover timerom the given start vertex is bounded. An example of a non-irreducible Markov chain to which we can apply Theorem 1E-mail address: john.sylvester@glasgow.ac.uk.ttps://doi.org/10.1016/j.spl.2022.109534167-7152/Crown Copyright © 2022 Published by Elsevier B.V. This is an open access article under the CC BY license (http://creativecommons.org/icenses/by/4.0/).J. Sylvester Statistics and Probability Letters 187 (2022) 109534iiorhiiCdtsAtftsTfFig. 1. Example of a non-irreducible Markov chain on seven states where the cover time from x is finite and from any other vertex the cover times unbounded/undefined.s given in Fig. 1. In this example the cover time from x is bounded however, the cover time from any other vertex isunbounded/undefined, as if a walk starts from any other vertex, then x (and possibly also the vertex immediately rightf x) cannot be reached.For a concrete example of why rational transition probabilities are necessary in Theorem 1, if one fixes any real number≥ 1 then the two state chain with transition matrix given byP =(1 − 1/r 1/r1/r 1 − 1/r), (1)as cover time r . It is well known, see for example Levin and Peres (2017, Lemma 1.13), that the cover time of finiterreducible Markov chain from any start vertex is bounded. This fact, and restricting the example given by (1) to r ∈ Q,mplies the following corollary to Theorem 1.orollary 2. The set of cover times attainable by finite discrete-time irreducible rational Markov chains is (Q∩ [1, ∞))∪ {0}.We now introduce multiple Markov chains, which have been studied for their applications to parallelising algorithmsriven by random walks, see Alon et al. (2011) and subsequent papers citing it. For any k ≥ 1, let Xt =(X (1)t , . . . , X(k)t)behe k-multiple of a Markov chain P where each X (i)t is an independent copy of the chain P run simultaneously on the sametate space Ω . The k-multiple of P is itself a Markov chain (with transition matrix K) on Ωk with transition probabilitiesK(x, y) =k∏j=1P(x(j), y(j)), for all x, y ∈ Ωk.s before, we denote the conditional expectation E(x(1),...,x(k)) [ · ] := E[·⏐⏐X0 = (x(1), . . . , x(k)) ], where X (i)0 = x(i) ∈ Ω ishe start state of the ith walk for each 1 ≤ i ≤ k. We let the stopping time τ (k)cov = inf{t :⋃ti=0{X(1)i , . . . , X(k)i } = Ω} be theirst time every state in Ω (not Ωk) has been visited by some walk X (i)t . We then let Ex[τ (k)cov]denote the k-walk stoppingime from x ∈ Ωk. Note that this is not simply the cover time of the chain K. The multiple walk cover time can haveubtle dependences on k and the host underlying Markov chain, see Alon et al. (2011).We show that Theorem 1 also holds in the more general setting of k-multiple Markov chains.heorem 3. Let k ≥ 1 and (Xt )t≥0 be the k-multiple of a discrete-time rational Markov chain on a finite state space Ω . Thenor any x ∈ Ωk such that Ex[τ (k)cov]< ∞ we have Ex[τ (k)cov]∈ Q.Theorem 1 is the special case k = 1 of Theorem 3, thus it suffices to prove Theorem 3.2. ProofsIn this section we shall prove Theorem 3. The first part of the proof (covered in Section 2.1) is to show the expectedtime to first visit any set of states (hitting time) in a rational Markov chain is rational. Then, in Section 2.2, we show forany k ≥ 1 and P, the multiple walk with transition matrix P can be coupled with a higher dimensional Markov chain Qon a state space V where |V | ≤ |Ω|k · 2|Ω|. The coupling shows that the first time all states in Ω have been visited by atleast one of the k walks has the same distribution as the first visit time a specific set C ⊂ V is visited in Q.2J. Sylvester Statistics and Probability Letters 187 (2022) 1095342loPBPmLPspmPeTiOsas32pawfSow.1. Rationality of hitting timesFor S ⊆ Ω , a subset of the state space of a Markov chain P, let the stopping timeτS := inf {t ≥ 0 : Xt ∈ S} ,be the first time S is visited. If S = {s} is a singleton set we abuse notation slightly by taking τs to mean τ{s}. For x ∈ Ω ,et Ex [ τS ] be the expected hitting time of S ⊆ Ω for a chain started from x. The next result is the hitting time analoguef Theorem 1.roposition 4. Let P be a discrete-time rational Markov chain on a finite state space Ω . For a non-empty set S ⊆ Ω let(S) = {x ∈ Ω : Ex [ τS ] < ∞}. Then for any S ⊆ Ω and x ∈ B(S) we have Ex[τS] ∈ Q.Observe that if P is irreducible then B(S) = Ω for any S ⊆ Ω by Levin and Peres (2017, Lemma 1.13). Before provingroposition 4 we give some definitions and prove an elementary lemma.For a field F and integers n,m ≥ 1 let Fn and Fm×n denote the set of n-dimensional vectors and m×n-dimensionatrices respectively. Let In denote the n×n identity matrix.emma 5. Let A ∈ Qn×n be non-singular and b ∈ Qn. Then there exists a unique vector x ∈ Qn such that Ax = b.roof. Since A is non-singular there exists a unique solution x ∈ Rn to the linear system given by Ax = b. Also, againince A is non-singular, we can compute A−1 by Gaussian elimination. Since all entries of A are rational, all multiplicationsreformed during the Gaussian elimination will be rational. Thus, as there are only finitely many row additions andultiplications, A−1 ∈ Qn×n. Since b ∈ Qn, we conclude that x = A−1b ∈ Qn. □We now use this lemma to prove Proposition 4.roof of Proposition 4. Observe that B(S) ̸= ∅ since S ⊆ B(S) and Es [ τS ] = 0 for all s ∈ S. Let b := |B(S)|. Now, eachntry of the vector h := (Ex[τS])x∈B(S) is bounded and h is a solution to the following set of linear equationsEx[τS] ={1 +∑y P(x, y) · Ey[τS] if x ̸∈ S0 if x ∈ S.his can be expressed as Ah = b where b ∈ {0, 1}b and A := (Ib − M) ∈ Qb×b for M ∈ Qb×b given by M(i, j) = P(i, j) if, j /∈ S and 0 otherwise. We shall show that(i) all rows i satisfy |A(i, i)| ≥∑j̸=i |A(i, j)|, and(ii) for each row r0, there exists a finite sequence of rows r0, r1 . . . , rt such that A(ri−1, ri) ̸= 0 for all 1 ≤ i ≤ t and|A(rt , rt )| >∑j̸=rt |A(rt , j)|.bserve that Condition (i) holds since M is a sub-matrix of P.For Condition (ii), note that for every row s ∈ S we have∑j M(s, j) = 0. Thus |A(s, s)| >∑j̸=s |A(s, j)| for any row∈ S. The fact that each row r0 corresponds to a state in B(S) implies that, for any row r0, there exists some rt ∈ S andsequence of states/rows r0, r1 . . . , rt such that A(ri−1, ri) = −P(ri−1, ri) ̸= 0 for each 1 ≤ i ≤ t , thus Condition (ii) isatisfied.Since A satisfies (i) and (ii) it is weakly chained diagonally dominant, thus by Azimzadeh and Forsyth (2016, Lemma.2) A is non-singular. Thus, by Lemma 5, h ∈ Qb. □.2. Encoding cover times as hitting timesLet P be a Markov chain on a state space Ω with transition matrix P = (P(x, y))x,y∈Ω and P(Ω) = {S ⊆ Ω} be theower-set of Ω . For k ≥ 1 independent walks with transition matrix P on the same state space Ω we define the k-walkuxiliary chain Q := Q(P, k) to be the Markov Chain on state space V := V (Ω, k) given byV ={((x1, . . . , xk), S) : S ⊆ Ω, xi ∈ S for all 1 ≤ i ≤ k}⊆ Ωk × P(Ω),ith transition matrix specified byQ((x, S), (y, S ∪ {y(1), . . . , y(k)}))= P(x(1), y(1)) · · · P(x(k), y(k)), (2)or any S ⊆ Ω and x, y ∈ Ωk where x = (x(1), . . . , x(k)) and y = (y(1), . . . , y(k)).Fig. 2 shows an example of the auxiliary chain Q of a single Markov chain P on three states, that is the case k = 1.taying within the confines of k = 1 case for simplicity, one may think of Q as inducing a directed graph consistingf many ‘layers’, where each layer is a copy of P restricted to a subset of Ω . These layers are linked by directed edgeshich are crossed when a new state not in the current layer is first visited. Thus, since a sequence x , x , . . . in the first0 13J. Sylvester Statistics and Probability Letters 187 (2022) 109534fctwssgacnCstPawFig. 2. This figure shows an example of a Markov chain P on three states (bottom right) and its associated auxiliary chain Q(P, 1), where the set Crom Lemma 6 is shown in the red shaded ellipse.omponent of V evolves according to P by (2), each layer encodes which states of chain have been visited so far by arajectory in P.Similar constructions to Q(P, 1) were used by the author and co-authors in the study of the Choice and ε-TB randomalks, which are walks where a controller can influence which vertices are visited. In particular they were used tohow that there exist optimal strategies for covering a graph by these walks which are time invariant in a certainense (Georgakopoulos et al., 2022) and to show the computational problem of finding optimal strategies to cover araph by these walks is in PSPACE (Haslegrave et al., 2022).The next result equates the cover time by k ≥ 1 multiple Markov chain with transition matrix P to the hitting time ofspecific set in the auxiliary chain Q(P, k). For clarity we use the notation EP·[ · ] to highlight the chain, in this case P, inwhich the expectation is taken.Lemma 6. Let P be a Markov chain on Ω , and let k ≥ 1 be an integer. Let Q := Q(P, k) be the associated k-walk auxiliaryhain with state space V := V (Ω, k), and set C = {(u, Ω) : u ∈ Ωk} ⊂ W. Then, for any x = (x(1), . . . , x(k)) ∈ Ωk and realumber a, we havePPx[τ (k)cov ≥ a]= PQ(x,{x(1),...,x(k)}) [ τC ≥ a ] .onsequently, EPx[τ (k)cov]= EQ(x,{x(1),...,x(k)}) [ τC ], for any x ∈ Ωk.We must introduce some notation before proving Lemma 6. For real valued random variables X, Y we say that Ytochastically dominates X if P [ Y ≥ a ] ≥ P [ X ≥ a ] for all real a, and we denote this by X ⪯ Y . Thus, if X ⪯ Y and Y ⪯ X ,hen X and Y are equidistributed.roof of Lemma 6. We first show how any trajectory (Xt )t≥0 of a k-multiple of the Markov chain P can be coupled withtrajectory (Yt )t≥0 of the auxiliary Markov chain Q(P, k) given by (2). To begin, given any start vector X0 = x0 ∈ Ωk,here x = (x(1), . . . , x(k)), we set Y = (x , {x(1), . . . , x(k)}) ∈ V . Then, given a trajectory (X )T = (x )T we set0 0 0 0 0 0 0 t t=0 t t=04J. Sylvester Statistics and Probability Letters 187 (2022) 109534Yt =(xt ,⋃ti=0⋃kj=1{x(j)i})for each 0 ≤ t ≤ T . Now by (2),t−1∏i=0k∏j=1P(x(j)i , x(j)i+1) = Q⎛⎝⎛⎝x0, k⋃j=1{x(j)0}⎞⎠ ,⎛⎝x1, 1⋃i=0k⋃j=1{x(j)i}⎞⎠⎞⎠ · · ·· Q⎛⎝⎛⎝xt−1, t−1⋃i=0k⋃j=1{x(j)i}⎞⎠ ,⎛⎝xt , t⋃i=0k⋃j=1{x(j)i}⎞⎠⎞⎠ .(3)Thus given any trajectory (Xt )t≥0 of P we can find a trajectory (Yt )t≥0 of Q(P, k) with the same measure. To couple a giventrajectory (Yt )t≥0 of Q(P, k) to a trajectory (Xt )t≥0 of P is even simpler; given Yt = (yt , St ) we simply ‘forget’ the secondcomponent of Yt and set Xt = yt ∈ Ωk for each t ≥ 0. Again the measure is preserved by (3).Recall the set C = {(u, Ω) : u ∈ Ωk} which is a subset of the state space V (Ω, k) of the auxiliary chain Q(P, k). Tocomplete the proof we show that, for any x ∈ Ωk, the times τcov and τC in the coupled chains Xt and Yt , started from xand (x, {x1, . . . , x(k)}) respectively, are equidistributed.Suppose we take any trajectory (Xt )Tt=0 of length T ≥ 0 such that ∪Ti=0 ∪kj=1 {X(j)i } = Ω . Then by the coupling above, wehave YT =(xT , ∪Ti=0∪kj=1 {X(j)i })= (xT , Ω) ∈ C . Since this holds for any trajectory and any time T such that ∪Tj=0{Xj} = Ω ,we can assume that T is the first such time. That is, we can take T = τcov and then it follows that τC ⪯ τcov.Conversely, let (Yt )Tt=0 be any trajectory in Q where Y0 = (y0, ∪kj=1{y(j)0 }), for some y0 = (y(1)0 , . . . , y(k)0 ) ∈ Ωk andYT ∈ C . Since the only transitions supported by Q are from (y, S) to(z, S ∪(∪kj=1{z(j)}))where∏kj=1 P(y(j), z(j)) > 0, andY0 = (y0, ∪kj=1{y(j)0 }), it follows that ∪Tt=0 ∪kj=1 {y(j)t } = Ω . Thus, by the coupling above, ∪Tt=0 ∪kj=1 {X(j)t } = Ω . Similarly, sincewe can take T = τC to be minimal, we have τcov ⪯ τC .Thus for any pair of coupled trajectories with fixed start vertices x and (x, ∪kj=1{x(j)0 }) the times τcov and τC are thesame. The final statement then follows by taking expectation. □Lemma 6 equates the cover time of any Markov chain P on Ω (not just rational chains) to a hitting time in a higherdimensional chain Q on V . This result may be useful for studying the cover time of an arbitrary Markov chain P on Ω .However, one drawback of this approach is that for many chains |V | is exponential in |Ω|.Having established Proposition 4 and Lemma 6 the proof of Theorem 3 is simple.Proof of Theorem 3. Let Q := Q(P, k) be the auxiliary chain associated with the k-multiple Markov chain with transitionmatrix P. Then EPx[τ (k)cov]= EQ(x,{x(1),...,x(k)}) [ τC ] for any x = (x(1), . . . , x(k)) ∈ Ωk by Lemma 6, where C = {(y, Ω) | y ∈ Ωk}.By assumption we have EPx [ τcov ] < ∞ and so x ∈ B(C). It follows from Proposition 4 that EQ(x,{x(1),...,x(k)}) [ τC ] ∈ Q and soEPx [ τcov ] ∈ Q as claimed. □AcknowledgementsWe thank Parsiad Azimzadeh and Agelos Georgakopoulos for discussions which led to a simplification of the proofof Proposition 4. The author is currently supported by Engineering and Physical Sciences Research Council (ESPRC) grantnumber EP/T004878/1. This work was started while the author was supported by ERC Starting Grant no. 679660 at theUniversity of Cambridge.ReferencesAldous, David, Fill, James Allen, 2002. Reversible Markov chains and random walks on graphs. Unfinished monograph, recompiled 2014, URL:https://www.stat.berkeley.edu/~aldous/RWG/book.html.Alon, Noga, Avin, Chen, Koucký, Michal, Kozma, Gady, Lotker, Zvi, Tuttle, Mark R., 2011. Many random walks are faster than one. Combin. Probab.Comput. (ISSN: 0963-5483) 20 (4), 481–502. http://dx.doi.org/10.1017/S0963548311000125.Azimzadeh, Parizad, Forsyth, Peter A., 2016. Weakly chained matrices, policy iteration, and impulse control. SIAM J. Numer. Anal. (ISSN: 0036-1429)54 (3), 1341–1364. http://dx.doi.org/10.1137/15M1043431.Georgakopoulos, Agelos, Haslegrave, John, Sauerwald, Thomas, Sylvester, John, 2022. The power of two choices for random walks. Combin. Probab.Comput. 31 (1), 73–100. http://dx.doi.org/10.1017/S0963548321000183.Haslegrave, John, Sauerwald, Thomas, Sylvester, John, 2022. Time dependent biased random walks. ACM Trans. Algorithms 18 (2), 12:1–12:30.http://dx.doi.org/10.1145/3498848.Levin, David A., Peres, Yuval, 2017. Markov Chains and Mixing Times, second ed. American Mathematical Society, Providence, RI, p. xvi+447.http://dx.doi.org/10.1090/mbk/107.5

The cover time of a (multiple) Markov chain with rational transition probabilities is rational

The cover time of a Markov chain on a finite state space is the expected time until all states are visited. We show that if the cover time of a discrete-time Markov chain with rational transitions probabilities is bounded, then it is a rational number. The result is proved by relating the cover time of the original chain to the hitting time of a set in another higher dimensional chain. We prove this result in a more general setting where k≥1 independent copies of a Markov chain are run simultaneously on the same state space

The Cover Time of a (Multiple) Markov Chain with Rational Transition Probabilities is Rational

Abstract

Similar works

Full text

Available Versions

Enlighten

University of Liverpool Repository