Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Learning for Cross-layer Resource Allocation in the Framework of Cognitive Wireless Networks

The framework of cognitive wireless networks is expected to endow wireless devices with a cognition-intelligence ability with which they can efficiently learn and respond to the dynamic wireless environment. In this dissertation, we focus on the problem of developing cognitive network control mechanisms without knowing in advance an accurate network model. We study a series of cross-layer resource allocation problems in cognitive wireless networks. Based on model-free learning, optimization and game theory, we propose a framework of self-organized, adaptive strategy learning for wireless devices to (implicitly) build the understanding of the network dynamics through trial-and-error. The work of this dissertation is divided into three parts. In the first part, we investigate a distributed, single-agent decision-making problem for real-time video streaming over a time-varying wireless channel between a single pair of transmitter and receiver. By modeling the joint source-channel resource allocation process for video streaming as a constrained Markov decision process, we propose a reinforcement learning scheme to search for the optimal transmission policy without the need to know in advance the details of network dynamics. In the second part of this work, we extend our study from the single-agent to a multi-agent decision-making scenario, and study the energy-efficient power allocation problems in a two-tier, underlay heterogeneous network and in a self-sustainable green network. For the heterogeneous network, we propose a stochastic learning algorithm based on repeated games to allow individual macro- or femto-users to find a Stackelberg equilibrium without flooding the network with local action information. For the self-sustainable green network, we propose a combinatorial auction mechanism that allows mobile stations to adaptively choose the optimal base station and sub-carrier group for transmission only from local payoff and transmission strategy information. In the third part of this work, we study a cross-layer routing problem in an interweaved Cognitive Radio Network (CRN), where an accurate network model is not available and the secondary users that are distributed within the CRN only have access to local action/utility information. In order to develop a spectrum-aware routing mechanism that is robust against potential insider attackers, we model the uncoordinated interaction between CRN nodes in the dynamic wireless environment as a stochastic game. Through decomposition of the stochastic routing game, we propose two stochastic learning algorithm based on a group of repeated stage games for the secondary users to learn the best-response strategies without the need of information flooding

Service quality assurance for the IPTV networks

The objective of the proposed research is to design and evaluate end-to-end solutions to support the Quality of Experience (QoE) for the Internet Protocol Television (IPTV) service. IPTV is a system that integrates voice, video, and data delivery into a single Internet Protocol (IP) framework to enable interactive broadcasting services at the subscribers. It promises significant advantages for both service providers and subscribers. For instance, unlike conventional broadcasting systems, IPTV broadcasts will not be restricted by the limited number of channels in the broadcast/radio spectrum. Furthermore, IPTV will provide its subscribers with the opportunity to access and interact with a wide variety of high-quality on-demand video content over the Internet. However, these advantages come at the expense of stricter quality of service (QoS) requirements than traditional Internet applications. Since IPTV is considered as a real-time broadcast service over the Internet, the success of the IPTV service depends on the QoE perceived by the end-users. The characteristics of the video traffic as well as the high-quality requirements of the IPTV broadcast impose strict requirements on transmission delay. IPTV framework has to provide mechanisms to satisfy the stringent delay, jitter, and packet loss requirements of the IPTV service over lossy transmission channels with varying characteristics. The proposed research focuses on error recovery and channel change latency problems in IPTV networks. Our specific aim is to develop a content delivery framework that integrates content features, IPTV application requirements, and network characteristics in such a way that the network resource utilization can be optimized for the given constraints on the user perceived service quality. To achieve the desired QoE levels, the proposed research focuses on the design of resource optimal server-based and peer-assisted delivery techniques. First, by analyzing the tradeoffs on the use of proactive and reactive repair techniques, a solution that optimizes the error recovery overhead is proposed. Further analysis on the proposed solution is performed by also focusing on the use of multicast error recovery techniques. By investigating the tradeoffs on the use of network-assisted and client-based channel change solutions, distributed content delivery frameworks are proposed to optimize the error recovery performance. Next, bandwidth and latency tradeoffs associated with the use of concurrent delivery streams to support the IPTV channel change are analyzed, and the results are used to develop a resource-optimal channel change framework that greatly improves the latency performance in the network. For both problems studied in this research, scalability concerns for the IPTV service are addressed by properly integrating peer-based delivery techniques into server-based solutions.Ph.D

Performance of cognitive stop-and-wait hybrid automatic repeat request in the face of imperfect sensing

The cognitive radio (CR) paradigm has the potential of improving the exploitation of the electromagnetic spectrum by detecting instantaneously unoccupied spectrum slots allocated to primary users (PUs). In order to support the process of spectrum reuse, we consider a CR scheme, which senses and opportunistically accesses a PU's spectrum for communication between a pair of nodes relying on the stop-and-wait hybrid automatic repeat request (SW-HARQ) protocol. This arrangement is represented by the cognitive SW-HARQ (CSW-HARQ), where the availability/unavailability of the PU's channel is modeled as a two-state Markov chain having OFF and ON states, respectively. Once the cognitive user (CU) finds that the PU's channel is available (i.e., in the OFF state), the CU transmits data over the PU channel's spectrum, while relying on the principles of SW-HARQ. We investigate both the throughput and the delay of CSW-HARQ, with a special emphasis on the impact of the various system parameters involved in the scenarios of both perfect and imperfect spectrum sensing. Furthermore, we analyze both the throughput as well as the average packet delay and end-to-end packet delay of the CSW-HARQ system. We propose a pair of analytical approaches: 1) the probability-based and 2) the discrete time Markov chain-based. Closed-form expressions are derived for both the throughput and the delay under the perfect and imperfect sensing environments that are validated by simulation. We demonstrate that the activity of PUs, the transmission reliability of the CU, and the sensing environment have a significant impact on both the throughput and the delay of the CR system