Stochastic Approximation Methods for Systems Over an Infinite Horizon

The paper develops efficient and general stochastic approximation (SA) methods for improving the operation of parametrized systems of either the continuous- or discrete-event dynamical systems types and which are of interest over a long time period. For example, one might wish to optimize or improve the stationary (or average cost per unit time) performance by adjusting the systems parameters. The number of applications and the associated literature are increasing at a rapid rate. This is partly due to the increasing activity in computing pathwise derivatives and adapting them to the average-cost problem. Although the original motivation and the examples come from an interest in the infinite-horizon problem, the techniques and results are of general applicability in SA. We present an updating and review of powerful ordinary differential equation-type methods, in a fairly general context, and based on weak convergence ideas. The results and proof techniques are applicable to a wide variety of applications. Exploiting the full potential of these ideas can greatly simplify and extend much current work. Their breadth as well as the relative ease of using the basic ideas are illustrated in detail via typical examples drawn from discrete-event dynamical systems, piecewise deterministic dynamical systems, and a stochastic differential equations model. In these particular illustrations, we use either infinitesimal perturbation analysis-type estimators, mean square derivative-type estimators, or finite-difference type estimators. Markov and non-Markov models are discussed. The algorithms for distributed/asynchronous updating as well as the fully synchronous schemes are developed

Congestion management in traffic-light intersections via Infinitesimal Perturbation Analysis

We present a flow-control technique in traffic-light intersections, aiming at regulating queue lengths to given reference setpoints. The technique is based on multivariable integrators with adaptive gains, computed at each control cycle by assessing the IPA gradients of the plant functions. Moreover, the IPA gradients are computable on-line despite the absence of detailed models of the traffic flows. The technique is applied to a two-intersection system where it exhibits robustness with respect to modeling uncertainties and computing errors, thereby permitting us to simplify the on-line computations perhaps at the expense of accuracy while achieving the desired tracking. We compare, by simulation, the performance of a centralized, joint two-intersection control with distributed control of each intersection separately, and show similar performance of the two control schemes for a range of parameters

Approximate IPA: Trading Unbiasedness for Simplicity

When Perturbation Analysis (PA) yields unbiased sensitivity estimators for expected-value performance functions in discrete event dynamic systems, it can be used for performance optimization of those functions. However, when PA is known to be unbiased, the complexity of its estimators often does not scale with the system's size. The purpose of this paper is to suggest an alternative approach to optimization which balances precision with computing efforts by trading off complicated, unbiased PA estimators for simple, biased approximate estimators. Furthermore, we provide guidelines for developing such estimators, that are largely based on the Stochastic Flow Modeling framework. We suggest that if the relative error (or bias) is not too large, then optimization algorithms such as stochastic approximation converge to a (local) minimum just like in the case where no approximation is used. We apply this approach to an example of balancing loss with buffer-cost in a finite-buffer queue, and prove a crucial upper bound on the relative error. This paper presents the initial study of the proposed approach, and we believe that if the idea gains traction then it may lead to a significant expansion of the scope of PA in optimization of discrete event systems.Comment: 8 pages, 8 figure

Sensitivity analysis of oscillator models in the space of phase-response curves: Oscillators as open systems

Oscillator models are central to the study of system properties such as entrainment or synchronization. Due to their nonlinear nature, few system-theoretic tools exist to analyze those models. The paper develops a sensitivity analysis for phase-response curves, a fundamental one-dimensional phase reduction of oscillator models. The proposed theoretical and numerical analysis tools are illustrated on several system-theoretic questions and models arising in the biology of cellular rhythms