2 research outputs found
Nonlinear Two-Time-Scale Stochastic Approximation: Convergence and Finite-Time Performance
Two-time-scale stochastic approximation, a generalized version of the popular
stochastic approximation, has found broad applications in many areas including
stochastic control, optimization, and machine learning. Despite its popularity,
theoretical guarantees of this method, especially its finite-time performance,
are mostly achieved for the linear case while the results for the nonlinear
counterpart are very sparse. Motivated by the classic control theory for
singularly perturbed systems, we study in this paper the asymptotic convergence
and finite-time analysis of the nonlinear two-time-scale stochastic
approximation. Under some fairly standard assumptions, we provide a formula
that characterizes the rate of convergence of the main iterates to the desired
solutions. In particular, we show that the method achieves a convergence in
expectation at a rate , where is the number of
iterations. The key idea in our analysis is to properly choose the two step
sizes to characterize the coupling between the fast and slow-time-scale
iterates
Finite-Time Analysis and Restarting Scheme for Linear Two-Time-Scale Stochastic Approximation
Motivated by their broad applications in reinforcement learning, we study the
linear two-time-scale stochastic approximation, an iterative method using two
different step sizes for finding the solutions of a system of two equations.
Our main focus is to characterize the finite-time complexity of this method
under time-varying step sizes and Markovian noise. In particular, we show that
the mean square errors of the variables generated by the method converge to
zero at a sublinear rate \Ocal(k^{2/3}), where is the number of
iterations. We then improve the performance of this method by considering the
restarting scheme, where we restart the algorithm after every predetermined
number of iterations. We show that using this restarting method the complexity
of the algorithm under time-varying step sizes is as good as the one using
constant step sizes, but still achieving an exact converge to the desired
solution. Moreover, the restarting scheme also helps to prevent the step sizes
from getting too small, which is useful for the practical implementation of the
linear two-time-scale stochastic approximation