Convergence rate of linear two-time-scale stochastic approximation

We study the rate of convergence of linear two-time-scale stochastic approximation methods. We consider two-time-scale linear iterations driven by i.i.d. noise, prove some results on their asymptotic covariance and establish asymptotic normality. The well-known result [Polyak, B. T. (1990). Automat. Remote Contr. 51 937-946; Ruppert, D. (1988). Technical Report 781, Cornell Univ.] on the optimality of Polyak-Ruppert averaging techniques specialized to linear stochastic approximation is established as a consequence of the general results in this paper

A two-time-scale phenomenon in a fragmentation-coagulation process

Consider two urns, $A$ and $B$, where initially $A$ contains a large number $n$ of balls and $B$ is empty. At each step, with equal probability, either we pick a ball at random in $A$ and place it in $B$, or vice-versa (provided of course that $A$, or $B$, is not empty). The number of balls in $B$ after $n$ steps is of order $\sqrt n$, and this number remains essentially the same after $\sqrt n$ further steps. Observe that each ball in the urn $B$ after $n$ steps has a probability bounded away from $0$ and $1$ to be placed back in the urn $A$ after $\sqrt n$ further steps. So, even though the number of balls in $B$ does not evolve significantly between $n$ and $n+\sqrt n$, the precise contain of urn $B$ does. This elementary observation is the source of an interesting two-time-scale phenomenon which we illustrate using a simple model of fragmentation-coagulation. Inspired by Pitman's construction of coalescing random forests, we consider for every $n\in \N$ a uniform random tree with $n$ vertices, and at each step, depending on the outcome of an independent fair coin tossing, either we remove one edge chosen uniformly at random amongst the remaining edges, or we replace one edge chosen uniformly at random amongst the edges which have been removed previously. The process that records the sizes of the tree-components evolves by fragmentation and coagulation. It exhibits subaging in the sense that when it is observed after $k$ steps in the regime $k\sim tn+s\sqrt n$ with $t>0$ fixed, it seems to reach a statistical equilibrium as $n\to\infty$; but different values of $t$ yield distinct pseudo-stationary distributions

Two time scale output feedback regulation for ill-conditioned systems

Issues pertaining to the well-posedness of a two time scale approach to the output feedback regulator design problem are examined. An approximate quadratic performance index which reflects a two time scale decomposition of the system dynamics is developed. It is shown that, under mild assumptions, minimization of this cost leads to feedback gains providing a second-order approximation of optimal full system performance. A simplified approach to two time scale feedback design is also developed, in which gains are separately calculated to stabilize the slow and fast subsystem models. By exploiting the notion of combined control and observation spillover suppression, conditions are derived assuring that these gains will stabilize the full-order system. A sequential numerical algorithm is described which obtains output feedback gains minimizing a broad class of performance indices, including the standard LQ case. It is shown that the algorithm converges to a local minimum under nonrestrictive assumptions. This procedure is adapted to and demonstrated for the two time scale design formulations

Convergence rate and averaging of nonlinear two-time-scale stochastic approximation algorithms

The first aim of this paper is to establish the weak convergence rate of nonlinear two-time-scale stochastic approximation algorithms. Its second aim is to introduce the averaging principle in the context of two-time-scale stochastic approximation algorithms. We first define the notion of asymptotic efficiency in this framework, then introduce the averaged two-time-scale stochastic approximation algorithm, and finally establish its weak convergence rate. We show, in particular, that both components of the averaged two-time-scale stochastic approximation algorithm simultaneously converge at the optimal rate $\sqrt{n}$.Comment: Published at http://dx.doi.org/10.1214/105051606000000448 in the Annals of Applied Probability (http://www.imstat.org/aap/) by the Institute of Mathematical Statistics (http://www.imstat.org

Reduction of Markov chains with two-time-scale state transitions

In this paper, we consider a general class of two-time-scale Markov chains whose transition rate matrix depends on a parameter $\lambda>0$. We assume that some transition rates of the Markov chain will tend to infinity as $\lambda\rightarrow\infty$. We divide the state space of the Markov chain $X$ into a fast state space and a slow state space and define a reduced chain $Y$ on the slow state space. Our main result is that the distribution of the original chain $X$ will converge in total variation distance to that of the reduced chain $Y$ uniformly in time $t$ as $\lambda\rightarrow\infty$.Comment: 30 pages, 3 figures; Stochastics: An International Journal of Probability and Stochastic Processes, 201