Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Resource management in QoS-aware wireless cellular networks

2011 Summer.Includes bibliographical references.Emerging broadband wireless networks that support high speed packet data with heterogeneous quality of service (QoS) requirements demand more flexible and efficient use of the scarce spectral resource. Opportunistic scheduling exploits the time-varying, location-dependent channel conditions to achieve multiuser diversity. In this work, we study two types of resource allocation problems in QoS-aware wireless cellular networks. First, we develop a rigorous framework to study opportunistic scheduling in multiuser OFDM systems. We derive optimal opportunistic scheduling policies under three common QoS/fairness constraints for multiuser OFDM systems--temporal fairness, utilitarian fairness, and minimum-performance guarantees. To implement these optimal policies efficiently, we provide a modified Hungarian algorithm and a simple suboptimal algorithm. We then propose a generalized opportunistic scheduling framework that incorporates multiple mixed QoS/fairness constraints, including providing both lower and upper bound constraints. Next, taking input queues and channel memory into consideration, we reformulate the transmission scheduling problem as a new class of Markov decision processes (MDPs) with fairness constraints. We investigate the throughput maximization and the delay minimization problems in this context. We study two categories of fairness constraints, namely temporal fairness and utilitarian fairness. We consider two criteria: infinite horizon expected total discounted reward and expected average reward. We derive and prove explicit dynamic programming equations for the above constrained MDPs, and characterize optimal scheduling policies based on those equations. An attractive feature of our proposed schemes is that they can easily be extended to fit different objective functions and other fairness measures. Although we only focus on uplink scheduling, the scheme is equally applicable to the downlink case. Furthermore, we develop an efficient approximation method--temporal fair rollout--to reduce the computational cost

Energy-Efficient Transmission Scheduling with Strict Underflow Constraints

We consider a single source transmitting data to one or more receivers/users over a shared wireless channel. Due to random fading, the wireless channel conditions vary with time and from user to user. Each user has a buffer to store received packets before they are drained. At each time step, the source determines how much power to use for transmission to each user. The source's objective is to allocate power in a manner that minimizes an expected cost measure, while satisfying strict buffer underflow constraints and a total power constraint in each slot. The expected cost measure is composed of costs associated with power consumption from transmission and packet holding costs. The primary application motivating this problem is wireless media streaming. For this application, the buffer underflow constraints prevent the user buffers from emptying, so as to maintain playout quality. In the case of a single user with linear power-rate curves, we show that a modified base-stock policy is optimal under the finite horizon, infinite horizon discounted, and infinite horizon average expected cost criteria. For a single user with piecewise-linear convex power-rate curves, we show that a finite generalized base-stock policy is optimal under all three expected cost criteria. We also present the sequences of critical numbers that complete the characterization of the optimal control laws in each of these cases when some additional technical conditions are satisfied. We then analyze the structure of the optimal policy for the case of two users. We conclude with a discussion of methods to identify implementable near-optimal policies for the most general case of M users.Comment: 109 pages, 11 pdf figures, template.tex is main file. We have significantly revised the paper from version 1. Additions include the case of a single receiver with piecewise-linear convex power-rate curves, the case of two receivers, and the infinite horizon average expected cost proble

A Survey on Delay-Aware Resource Control for Wireless Systems --- Large Deviation Theory, Stochastic Lyapunov Drift and Distributed Stochastic Learning

In this tutorial paper, a comprehensive survey is given on several major systematic approaches in dealing with delay-aware control problems, namely the equivalent rate constraint approach, the Lyapunov stability drift approach and the approximate Markov Decision Process (MDP) approach using stochastic learning. These approaches essentially embrace most of the existing literature regarding delay-aware resource control in wireless systems. They have their relative pros and cons in terms of performance, complexity and implementation issues. For each of the approaches, the problem setup, the general solution and the design methodology are discussed. Applications of these approaches to delay-aware resource allocation are illustrated with examples in single-hop wireless networks. Furthermore, recent results regarding delay-aware multi-hop routing designs in general multi-hop networks are elaborated. Finally, the delay performance of the various approaches are compared through simulations using an example of the uplink OFDMA systems.Comment: 58 pages, 8 figures; IEEE Transactions on Information Theory, 201

Low-complexity dynamic resource scheduling for downlink MC-NOMA over fading channels

In this paper, we investigate dynamic resource scheduling (i.e., joint user, subchannel, and power scheduling) for downlink multi-channel non-orthogonal multiple access (MC-NOMA) systems over time-varying fading channels. Specifically, we address the weighted average sum rate maximization problem with quality-of-service (QoS) constraints. In particular, to facilitate fast resource scheduling, we focus on developing a very low-complexity algorithm. To this end, by leveraging Lagrangian duality and the stochastic optimization theory, we first develop an opportunistic MC-NOMA scheduling algorithm whereby the original problem is decomposed into a series of subproblems, one for each time slot. Accordingly, resource scheduling works in an online manner by solving one subproblem per time slot, making it more applicable to practical systems. Then, we further develop a heuristic joint subchannel assignment and power allocation (Joint-SAPA) algorithm with very low computational complexity, called Joint-SAPA-LCC, that solves each subproblem. Finally, through simulation, we show that our Joint-SAPA-LCC algorithm provides good performance comparable to the existing Joint-SAPA algorithms despite requiring much lower computational complexity. We also demonstrate that our opportunistic MC-NOMA scheduling algorithm in which the Joint-SAPA-LCC algorithm is embedded works well while satisfying given QoS requirements.Comment: 39 pages, 11 figure