Structure-Aware Stochastic Control for Transmission Scheduling

In this paper, we consider the problem of real-time transmission scheduling over time-varying channels. We first formulate the transmission scheduling problem as a Markov decision process (MDP) and systematically unravel the structural properties (e.g. concavity in the state-value function and monotonicity in the optimal scheduling policy) exhibited by the optimal solutions. We then propose an online learning algorithm which preserves these structural properties and achieves -optimal solutions for an arbitrarily small . The advantages of the proposed online method are that: (i) it does not require a priori knowledge of the traffic arrival and channel statistics and (ii) it adaptively approximates the state-value functions using piece-wise linear functions and has low storage and computation complexity. We also extend the proposed low-complexity online learning solution to the prioritized data transmission. The simulation results demonstrate that the proposed method achieves significantly better utility (or delay)-energy trade-offs when comparing to existing state-of-art online optimization methods.Comment: 41page

Optimal Routing for Delay-Sensitive Traffic in Overlay Networks

We design dynamic routing policies for an overlay network which meet delay requirements of real-time traffic being served on top of an underlying legacy network, where the overlay nodes do not know the underlay characteristics. We pose the problem as a constrained MDP, and show that when the underlay implements static policies such as FIFO with randomized routing, then a decentralized policy, that can be computed efficiently in a distributed fashion, is optimal. Our algorithm utilizes multi-timescale stochastic approximation techniques, and its convergence relies on the fact that the recursions asymptotically track a nonlinear differential equation, namely the replicator equation. Extensive simulations show that the proposed policy indeed outperforms the existing policies

Structured Optimal Transmission Control in Network-coded Two-way Relay Channels

This paper considers a transmission control problem in network-coded two-way relay channels (NC-TWRC), where the relay buffers random symbol arrivals from two users, and the channels are assumed to be fading. The problem is modeled by a discounted infinite horizon Markov decision process (MDP). The objective is to find a transmission control policy that minimizes the symbol delay, buffer overflow and transmission power consumption and error rate simultaneously and in the long run. By using the concepts of submodularity, multimodularity and L-natural convexity, we study the structure of the optimal policy searched by dynamic programming (DP) algorithm. We show that the optimal transmission policy is nondecreasing in queue occupancies or/and channel states under certain conditions such as the chosen values of parameters in the MDP model, channel modeling method, modulation scheme and the preservation of stochastic dominance in the transitions of system states. The results derived in this paper can be used to relieve the high complexity of DP and facilitate real-time control.Comment: 32 page

Opportunities for Network Coding: To Wait or Not to Wait

It has been well established that wireless network coding can significantly improve the efficiency of multi-hop wireless networks. However, in a stochastic environment some of the packets might not have coding pairs, which limits the number of available coding opportunities. In this context, an important decision is whether to delay packet transmission in hope that a coding pair will be available in the future or transmit a packet without coding. The paper addresses this problem by formulating a stochastic dynamic program whose objective is to minimize the long-run average cost per unit time incurred due to transmissions and delays. In particular, we identify optimal control actions that would balance between costs of transmission against the costs incurred due to the delays. Moreover, we seek to address a crucial question: what should be observed as the state of the system? We analytically show that observing queue lengths suffices if the system can be modeled as a Markov decision process. We also show that a stationary threshold type policy based on queue lengths is optimal. We further substantiate our results with simulation experiments for more generalized settings.Comment: 14 pages, journal version of arXiv:1105.4143v1, submitted to IEEE/ACM Transaction on Networkin

A c/μc/\mu-Rule for Service Resource Allocation in Group-Server Queues

In this paper, we study a dynamic on/off server scheduling problem in a queueing system with multi-class servers, where servers are heterogeneous and can be classified into $K$ groups. Servers in the same group are homogeneous. A scheduling policy determines the number of working servers (servers that are turned on) in each group at every state $n$ (number of customers in the system). Our goal is to find the optimal scheduling policy to minimize the long-run average cost, which consists of an increasing convex holding cost and a linear operating cost. We use the sensitivity-based optimization theory to characterize the optimal policy. A necessary and sufficient condition of the optimal policy is derived. We also prove that the optimal policy has monotone structures and a quasi bang-bang control is optimal. We find that the optimal policy is indexed by the value of $c - \mu G(n)$, where $c$ is the operating cost rate, $\mu$ is the service rate for a server, and $G(n)$ is a computable quantity called perturbation realization factor. Specifically, the group with smaller negative $c - \mu G(n)$ is more preferred to be turned on, while the group with positive $c - \mu G(n)$ should be turned off. However, the preference ranking of each group is affected by $G(n)$ and the preference order may change with the state $n$, the arrival rate, and the cost function. Under a reasonable condition of scale economies, we further prove that the optimal policy obeys a so-called $c$/$\mu$-rule. That is, the servers with smaller $c$/$\mu$ should be turned on with higher priority and the preference order of groups remains unchanged. This rule can be viewed as a sister version of the famous $c\mu$-rule for polling queues. With the monotone property of $G(n)$, we further prove that the optimal policy has a multi-threshold structure when the $c$/$\mu$-rule is applied.Comment: 55 pages, 11 figures, present an optimal rule called $c/\mu$-rule which can be viewed as a sister version of the famous $c\mu$-rule in queueing theor

Optimal Policies for Status Update Generation in a Wireless System with Heterogeneous Traffic

A large body of applications that involve monitoring, decision making, and forecasting require timely status updates for their efficient operation. Age of Information (AoI) is a newly proposed metric that effectively captures this requirement. Recent research on the subject has derived AoI optimal policies for the generation of status updates and AoI optimal packet queueing disciplines. Unlike previous research we focus on low-end devices that typically support monitoring applications in the context of the Internet of Things. We acknowledge that these devices host a diverse set of applications some of which are AoI sensitive while others are not. Furthermore, due to their limited computational resources they typically utilize a simple First-In First-Out (FIFO) queueing discipline. We consider the problem of optimally controlling the status update generation process for a system with a source-destination pair that communicates via a wireless link, whereby the source node is comprised of a FIFO queue and two applications, one that is AoI sensitive and one that is not. We formulate this problem as a dynamic programming problem and utilize the framework of Markov Decision Processes to derive optimal policies for the generation of status update packets. Due to the lack of comparable methods in the literature, we compare the derived optimal policies against baseline policies, such as the zero-wait policy, and investigate the performance of all policies for a variety of network configurations. Results indicate that existing status update policies fail to capture the trade-off between frequent generation of status updates and queueing delay and thus perform poorly

The Value of Service Rate Flexibility in an M/M/1 Queue with Admission Control

We consider a single server queueing system with admission control and the possibility to switch dynamically between a low and a high service rate, and examine the benefit of this service rate flexibility. We formulate a discounted Markov Decision Process model for the problem of joint admission and service control, and show that the optimal policy has a threshold structure for both controls. Regarding the benefit due to flexibility, we show that it is increasing in system congestion, and that its effect on the admission policy is to increase the admission threshold. We also derive a simple approximate condition between the admission reward and the relative cost of service rate increase, so that the service rate flexibility is beneficial. We finally show that the results extend to the expected average reward case

Load balancing with heterogeneous schedulers

Load balancing is a common approach in web server farms or inventory routing problems. An important issue in such systems is to determine the server to which an incoming request should be routed to optimize a given performance criteria. In this paper, we assume the server's scheduling disciplines to be heterogeneous. More precisely, a server implements a scheduling discipline which belongs to the class of limited processor sharing (LPS-$d$) scheduling disciplines. Under LPS-$d$, up to $d$ jobs can be served simultaneously, and hence, includes as special cases First Come First Served ($d=1$) and Processor Sharing ($d=\infty$). In order to obtain efficient heuristics, we model the above load-balancing framework as a multi-armed restless bandit problem. Using the relaxation technique, as first developed in the seminal work of Whittle, we derive Whittle's index policy for general cost functions and obtain a closed-form expression for Whittle's index in terms of the steady-state distribution. Through numerical computations, we investigate the performance of Whittle's index with two different performance criteria: linear cost criterion and a cost criterion that depends on the first and second moment of the throughput. Our results show that \emph{(i)} the structure of Whittle's index policy can strongly depend on the scheduling discipline implemented in the server, i.e., on $d$, and that \emph{(ii)} Whittle's index policy significantly outperforms standard dispatching rules such as Join the Shortest Queue (JSQ), Join the Shortest Expected Workload (JSEW), and Random Server allocation (RSA)

Channels, Remote Estimation and Queueing Systems With A Utilization-Dependent Component: A Unifying Survey Of Recent Results

In this article, we survey the main models, techniques, concepts, and results centered on the design and performance evaluation of engineered systems that rely on a utilization-dependent component (UDC) whose operation may depend on its usage history or assigned workload. Specifically, we report on research themes concentrating on the characterization of the capacity of channels and the design with performance guarantees of remote estimation and queueing systems. Causes for the dependency of a UDC on past utilization include the use of replenishable energy sources to power the transmission of information among the sub-components of a networked system, and the assistance of a human operator for servicing a queue. Our analysis unveils the similarity of the UDC models typically adopted in each of the research themes, and it reveals the differences in the objectives and technical approaches employed. We also identify new challenges and future research directions inspired by the cross-pollination among the central concepts, techniques, and problem formulations of the research themes discussed

Delay Optimal Scheduling of Arbitrarily Bursty Traffic over Multi-State Time-Varying Channels

In this paper, we study joint queue-aware and channel-aware scheduling of arbitrarily bursty traffic over multi-state time-varying channels, where the bursty packet arrival in the network layer, the backlogged queue in the data link layer, and the power adaptive transmission with fixed modulation in the physical layer are jointly considered from a cross-layer perspective. To achieve minimum queueing delay given a power constraint, a probabilistic cross-layer scheduling policy is proposed, and characterized by a Markov chain model. To describe the delay-power tradeoff, we formulate a non-linear optimization problem, which however is very challenging to solve. To handle with this issue, we convert the optimization problem into an equivalent Linear Programming (LP) problem, which allows us to obtain the optimal threshold-based scheduling policy with an optimal threshold imposed on the queue length in accordance with each channel state.Comment: 6 pages, conferenc