Marginal productivity index policies for scheduling a multiclass delay-/loss-sensitive queue

We address the problem of scheduling a multiclass M/M/1 queue with a finite dedicated buffer for each class. Some classes are delay-sensitive, modeling real-time traffic (e.g. voice, video), whereas others are loss-sensitive, modeling nonreal-time traffic (e.g. data). Different levels of tolerance to delay and loss are modeled by appropriate linear holding cost and rejection cost rates. The goal is to design well-grounded and tractable scheduling policies which nearly minimize the discounted or long-run average expected cost objective. We develop new dynamic index policies, prescribing to give higher service priority to classes with larger index values, where the priority index of a class measures the marginal productivity of work at its current state. To construct the indices, we deploy the theory of marginal productivity indices (MPIs) and PCL-indexability we have introduced in recent work, and further introduce significant extensions to such theory motivated by phenomena observed in the model of concern. The MPI policies are shown to furnish new, insightful structural results, and to exhibit a nearly optimal performance in a computational study

Restless bandit marginal productivity indices II: multiproject case and scheduling a multiclass make-to-order/-stock M/G/1 queue

This paper develops a framework based on convex optimization and economic ideas to formulate and solve approximately a rich class of dynamic and stochastic resource allocation problems, fitting in a generic discrete-state multi-project restless bandit problem (RBP). It draws on the single-project framework in the author's companion paper "Restless bandit marginal productivity indices I: Single-project case and optimal control of a make-to-stock M/G/1 queue", based on characterization of a project's marginal productivity index (MPI). Our framework significantly expands the scope of Whittle (1988)'s seminal approach to the RBP. Contributions include: (i) Formulation of a generic multi-project RBP, and algorithmic solution via single-project MPIs of a relaxed problem, giving a lower bound on optimal cost performance; (ii) a heuristic MPI-based hedging point and index policy; (iii) application of the MPI policy and bound to the problem of dynamic scheduling for a multiclass combined MTO/MTS M/G/1 queue with convex backorder and stock holding cost rates, under the LRA criterion; and (iv) results of a computational study on the MPI bound and policy, showing the latter's near-optimality across the cases investigated

From Packet to Power Switching: Digital Direct Load Scheduling

At present, the power grid has tight control over its dispatchable generation capacity but a very coarse control on the demand. Energy consumers are shielded from making price-aware decisions, which degrades the efficiency of the market. This state of affairs tends to favor fossil fuel generation over renewable sources. Because of the technological difficulties of storing electric energy, the quest for mechanisms that would make the demand for electricity controllable on a day-to-day basis is gaining prominence. The goal of this paper is to provide one such mechanisms, which we call Digital Direct Load Scheduling (DDLS). DDLS is a direct load control mechanism in which we unbundle individual requests for energy and digitize them so that they can be automatically scheduled in a cellular architecture. Specifically, rather than storing energy or interrupting the job of appliances, we choose to hold requests for energy in queues and optimize the service time of individual appliances belonging to a broad class which we refer to as "deferrable loads". The function of each neighborhood scheduler is to optimize the time at which these appliances start to function. This process is intended to shape the aggregate load profile of the neighborhood so as to optimize an objective function which incorporates the spot price of energy, and also allows distributed energy resources to supply part of the generation dynamically.Comment: Accepted by the IEEE journal of Selected Areas in Communications (JSAC): Smart Grid Communications series, to appea

Optimal Network Control in Partially-Controllable Networks

The effectiveness of many optimal network control algorithms (e.g., BackPressure) relies on the premise that all of the nodes are fully controllable. However, these algorithms may yield poor performance in a partially-controllable network where a subset of nodes are uncontrollable and use some unknown policy. Such a partially-controllable model is of increasing importance in real-world networked systems such as overlay-underlay networks. In this paper, we design optimal network control algorithms that can stabilize a partially-controllable network. We first study the scenario where uncontrollable nodes use a queue-agnostic policy, and propose a low-complexity throughput-optimal algorithm, called Tracking-MaxWeight (TMW), which enhances the original MaxWeight algorithm with an explicit learning of the policy used by uncontrollable nodes. Next, we investigate the scenario where uncontrollable nodes use a queue-dependent policy and the problem is formulated as an MDP with unknown queueing dynamics. We propose a new reinforcement learning algorithm, called Truncated Upper Confidence Reinforcement Learning (TUCRL), and prove that TUCRL achieves tunable three-way tradeoffs between throughput, delay and convergence rate

MARGINAL PRODUCTIVITY INDEX POLICIES FOR SCHEDULING A MULTICLASS DELAY-/LOSS-SENSITIVE QUEUE

We address the problem of scheduling a multiclass M/M/1 queue with a finite dedicated buffer for each class. Some classes are delay-sensitive, modeling real-time traffic (e.g. voice, video), whereas others are loss-sensitive, modeling nonreal-time traffic (e.g. data). Different levels of tolerance to delay and loss are modeled by appropriate linear holding cost and rejection cost rates. The goal is to design well-grounded and tractable scheduling policies which nearly minimize the discounted or long-run average expected cost objective. We develop new dynamic index policies, prescribing to give higher service priority to classes with larger index values, where the priority index of a class measures the marginal productivity of work at its current state. To construct the indices, we deploy the theory of marginal productivity indices (MPIs) and PCLindexability we have introduced in recent work, and further introduce significant extensions to such theory motivated by phenomena observed in the model of concern. The MPI policies are shown to furnish new, insightful structural results, and to exhibit a nearly optimal performance in a computational study.