Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

ActMesh- A Cognitive Resource Management paradigm for dynamic mobile Internet Access with Reliability Guarantees

Wireless Mesh Networks (WMNs) are going increasing attention as a flexible low-cost networking architecture to provide media Internet access over metropolitan areas to mobile clients requiring multimedia services. In WMNs, Mesh Routers (MRs) from the mesh backbone and accomplish the twofold task of traffic forwarding, as well as providing multimedia access to mobile Mesh Clients (MCs). Due to the intensive bandwidth-resource requested for supporting QoS-demanding multimedia services, performance of the current WMNs is mainly limited by spectrum-crowding and traffic-congestion, as only scarce spectrum-resources is currently licensed for the MCs' access. In principle, this problem could be mitigated by exploiting in a media-friendly (e.g., content-aware) way the context-aware capabilities offered by the Cognitive Radio (CR) paradigm. As integrated exploitation of both content and context-aware system's capabilities is at the basis of our proposed Active Mesh (ActMesh) networking paradigm. This last aims at defining a network-wide architecture for realizing media-friendly Cognitive Mesh nets (e.g., context aware Cognitive Mesh nets). Hence, main contribution of this work is four fold: 1. After introducing main functional blocks of our ActMesh architecture, suitable self-adaptive Belief Propagation and Soft Data Fusion algorithms are designed to provide context-awareness. This is done under both cooperative and noncooperative sensing frameworks. 2. The resulting network-wide resource management problem is modelled as a constrained stochastic Network Utility Maximization (NUM) problem, with the dual (contrasting) objective to maximize spectrum efficiency at the network level, while accounting for the perceived quality of the delivered media flows at the client level. 3. A fully distributed, scalable and self-adaptive implementation of the resulting Active Resource Manager (ARM) is deployed, that explicitly accounts for the energy limits of the battery powered MCs and the effects induced by both fading and client mobility. Due to informationally decentralized architecture of the ActMesh net, the complexity of (possibly, optimal) centralized solutions for resource management becomes prohibitive when number of MCs accessing ActMesh net grow. Furthermore, centralized resource management solutions could required large amounts of time to collect and process the required network information, which, in turn, induce delay that can be unacceptable for delay sensitive media applications, e.g., multimedia streaming. Hence, it is important to develop network-wide ARM policies that are both distributed and scalable by exploiting the radio MCs capabilities to sense, adapt and coordinate themselves. We validate our analytical models via simulation based numerical tests, that support actual effectiveness of the overall ActMesh paradigm, both in terms of objective and subjective performance metrics. In particular, the basic tradeoff among backbone traffic-vs-access traffic arising in the ActMesh net from the bandwidth-efficient opportunistic resource allocation policy pursued by the deployed ARM is numerically characterized. The standardization framework we inspire to is the emerging IEEE 802.16h one

ActMesh- A Cognitive Resource Management paradigm for dynamic mobile Internet Access with Reliability Guarantees

Wireless Mesh Networks (WMNs) are going increasing attention as a flexible low-cost networking architecture to provide media Internet access over metropolitan areas to mobile clients requiring multimedia services. In WMNs, Mesh Routers (MRs) from the mesh backbone and accomplish the twofold task of traffic forwarding, as well as providing multimedia access to mobile Mesh Clients (MCs). Due to the intensive bandwidth-resource requested for supporting QoS-demanding multimedia services, performance of the current WMNs is mainly limited by spectrum-crowding and traffic-congestion, as only scarce spectrum-resources is currently licensed for the MCs' access. In principle, this problem could be mitigated by exploiting in a media-friendly (e.g., content-aware) way the context-aware capabilities offered by the Cognitive Radio (CR) paradigm. As integrated exploitation of both content and context-aware system's capabilities is at the basis of our proposed Active Mesh (ActMesh) networking paradigm. This last aims at defining a network-wide architecture for realizing media-friendly Cognitive Mesh nets (e.g., context aware Cognitive Mesh nets). Hence, main contribution of this work is four fold: 1. After introducing main functional blocks of our ActMesh architecture, suitable self-adaptive Belief Propagation and Soft Data Fusion algorithms are designed to provide context-awareness. This is done under both cooperative and noncooperative sensing frameworks. 2. The resulting network-wide resource management problem is modelled as a constrained stochastic Network Utility Maximization (NUM) problem, with the dual (contrasting) objective to maximize spectrum efficiency at the network level, while accounting for the perceived quality of the delivered media flows at the client level. 3. A fully distributed, scalable and self-adaptive implementation of the resulting Active Resource Manager (ARM) is deployed, that explicitly accounts for the energy limits of the battery powered MCs and the effects induced by both fading and client mobility. Due to informationally decentralized architecture of the ActMesh net, the complexity of (possibly, optimal) centralized solutions for resource management becomes prohibitive when number of MCs accessing ActMesh net grow. Furthermore, centralized resource management solutions could required large amounts of time to collect and process the required network information, which, in turn, induce delay that can be unacceptable for delay sensitive media applications, e.g., multimedia streaming. Hence, it is important to develop network-wide ARM policies that are both distributed and scalable by exploiting the radio MCs capabilities to sense, adapt and coordinate themselves. We validate our analytical models via simulation based numerical tests, that support actual effectiveness of the overall ActMesh paradigm, both in terms of objective and subjective performance metrics. In particular, the basic tradeoff among backbone traffic-vs-access traffic arising in the ActMesh net from the bandwidth-efficient opportunistic resource allocation policy pursued by the deployed ARM is numerically characterized. The standardization framework we inspire to is the emerging IEEE 802.16h one

An energy-efficient spectrum-aware reinforcement learning-based clustering algorithm for cognitive radio sensor networks

It is well-known that clustering partitions network into logical groups of nodes in order to achieve energy efficiency and to enhance dynamic channel access in cognitive radio through cooperative sensing. While the topic of energy efficiency has been well investigated in conventional wireless sensor networks, the latter has not been extensively explored. In this paper, we propose a reinforcement learning-based spectrum-aware clustering algorithm that allows a member node to learn the energy and cooperative sensing costs for neighboring clusters to achieve an optimal solution. Each member node selects an optimal cluster that satisfies pairwise constraints, minimizes network energy consumption and enhances channel sensing performance through an exploration technique. We first model the network energy consumption and then determine the optimal number of clusters for the network. The problem of selecting an optimal cluster is formulated as a Markov Decision Process (MDP) in the algorithm and the obtained simulation results show convergence, learning and adaptability of the algorithm to dynamic environment towards achieving an optimal solution. Performance comparisons of our algorithm with the Groupwise Spectrum Aware (GWSA)-based algorithm in terms of Sum of Square Error (SSE), complexity, network energy consumption and probability of detection indicate improved performance from the proposed approach. The results further reveal that an energy savings of 9% and a significant Primary User (PU) detection improvement can be achieved with the proposed approach