An MDP decomposition approach for traffic control at isolated signalized intersections

This article presents a novel approach for the dynamic control of a signalized intersection. At the intersection, there is a number of arrival flows of cars, each having a single queue (lane). The set of all flows is partitioned into disjoint combinations of nonconflicting flows that will receive green together. The dynamic control of the traffic lights is based on the numbers of cars waiting in the queues. The problem concerning when to switch (and which combination to serve next) is modeled as a Markovian decision process in discrete time. For large intersections (i.e., intersections with a large number of flows), the number of states becomes tremendously large, prohibiting straightforward optimization using value iteration or policy iteration. Starting from an optimal (or nearly optimal) fixed-cycle strategy, a one-step policy improvement is proposed that is easy to compute and is shown to give a close to optimal strategy for the dynamic proble

A Survey on Delay-Aware Resource Control for Wireless Systems --- Large Deviation Theory, Stochastic Lyapunov Drift and Distributed Stochastic Learning

In this tutorial paper, a comprehensive survey is given on several major systematic approaches in dealing with delay-aware control problems, namely the equivalent rate constraint approach, the Lyapunov stability drift approach and the approximate Markov Decision Process (MDP) approach using stochastic learning. These approaches essentially embrace most of the existing literature regarding delay-aware resource control in wireless systems. They have their relative pros and cons in terms of performance, complexity and implementation issues. For each of the approaches, the problem setup, the general solution and the design methodology are discussed. Applications of these approaches to delay-aware resource allocation are illustrated with examples in single-hop wireless networks. Furthermore, recent results regarding delay-aware multi-hop routing designs in general multi-hop networks are elaborated. Finally, the delay performance of the various approaches are compared through simulations using an example of the uplink OFDMA systems.Comment: 58 pages, 8 figures; IEEE Transactions on Information Theory, 201

Queue-Aware Distributive Resource Control for Delay-Sensitive Two-Hop MIMO Cooperative Systems

In this paper, we consider a queue-aware distributive resource control algorithm for two-hop MIMO cooperative systems. We shall illustrate that relay buffering is an effective way to reduce the intrinsic half-duplex penalty in cooperative systems. The complex interactions of the queues at the source node and the relays are modeled as an average-cost infinite horizon Markov Decision Process (MDP). The traditional approach solving this MDP problem involves centralized control with huge complexity. To obtain a distributive and low complexity solution, we introduce a linear structure which approximates the value function of the associated Bellman equation by the sum of per-node value functions. We derive a distributive two-stage two-winner auction-based control policy which is a function of the local CSI and local QSI only. Furthermore, to estimate the best fit approximation parameter, we propose a distributive online stochastic learning algorithm using stochastic approximation theory. Finally, we establish technical conditions for almost-sure convergence and show that under heavy traffic, the proposed low complexity distributive control is global optimal.Comment: 30 pages, 7 figure