Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Average AoI Minimization for Energy Harvesting Relay-aided Status Update Network Using Deep Reinforcement Learning

A dual-hop status update system aided by energy harvesting (EH) relays with finite data and energy buffers is studied in this work. To achieve timely status updates, the best relays should be selected to minimize the average age of information (AoI), which is a recently proposed metric to evaluate information freshness. The average AoI minimization can be formulated as a Markov decision process (MDP), but the state space for capturing channel and buffer evolution grows exponentially with the number of relays, leading to high solution complexity. We propose a relay selection (RS) scheme based on deep reinforcement learning (DRL) according to the instantaneous channel packet freshness and buffer information of each relay. Simulation results show a significant improvement of the proposed DRL-based RS scheme over state-of-art approaches.Comment: This article has been accepted for publication in IEEE Wireless Communications Letters. Citation information: DOI 10.1109/LWC.2023.327886

Relaying in the Internet of Things (IoT): A Survey

The deployment of relays between Internet of Things (IoT) end devices and gateways can improve link quality. In cellular-based IoT, relays have the potential to reduce base station overload. The energy expended in single-hop long-range communication can be reduced if relays listen to transmissions of end devices and forward these observations to gateways. However, incorporating relays into IoT networks faces some challenges. IoT end devices are designed primarily for uplink communication of small-sized observations toward the network; hence, opportunistically using end devices as relays needs a redesign of both the medium access control (MAC) layer protocol of such end devices and possible addition of new communication interfaces. Additionally, the wake-up time of IoT end devices needs to be synchronized with that of the relays. For cellular-based IoT, the possibility of using infrastructure relays exists, and noncellular IoT networks can leverage the presence of mobile devices for relaying, for example, in remote healthcare. However, the latter presents problems of incentivizing relay participation and managing the mobility of relays. Furthermore, although relays can increase the lifetime of IoT networks, deploying relays implies the need for additional batteries to power them. This can erode the energy efficiency gain that relays offer. Therefore, designing relay-assisted IoT networks that provide acceptable trade-offs is key, and this goes beyond adding an extra transmit RF chain to a relay-enabled IoT end device. There has been increasing research interest in IoT relaying, as demonstrated in the available literature. Works that consider these issues are surveyed in this paper to provide insight into the state of the art, provide design insights for network designers and motivate future research directions

Delay Contributing Factors and Strategies Towards Its Minimization in IoT

Internet of Things (IoT) refers to various interconnected devices, typically supplied with limited computational and communication resources. Most of the devices are designed to operate with limited memory and processing capability, low bandwidth, short range and other characteristics of low cost hardware. The resulting networks are exposed to traffic loss and prone to other vulnerabilities. One of the major concerns is to ensure that the network communication among these deployed devices remains at required level of Quality of Service (QoS) of different IoT applications. The purpose of this paper is to highlight delay contributing factors in Low Power and Lossy Networks (LLNs) since providing low end-to-end delay is a crucial issue in IoT environment especially for mission critical applications. Various research efforts in relevance to this aspect are then presente

Security and Energy Efficiency in Resource-Constrained Wireless Multi-hop Networks

In recent decades, there has been a huge improvement and interest from the research community in wireless multi-hop networks. Such networks have widespread applications in civil, commercial and military applications. Paradigms of this type of networks that are critical for many aspects of human lives are mobile ad-hoc networks, sensor networks, which are used for monitoring buildings and large agricultural areas, and vehicular networks with applications in traffic monitoring and regulation. Internet of Things (IoT) is also envisioned as a multi-hop network consisting of small interconnected devices, called ``things", such as smart meters, smart traffic lights, thermostats etc. Wireless multi-hop networks suffer from resource constraints, because all the devices have limited battery, computational power and memory. Battery level of these devices should be preserved in order to ensure reliability and communication across the network. In addition, these devices are not a priori designed to defend against sophisticated adversaries, which may be deployed across the network in order to disrupt network operation. In addition, the distributed nature of this type of networks introduces another limitation to protocol performance in the presence of adversaries. Hence, the inherit nature of this type of networks poses severe limitations on designing and optimizing protocols and network operations. In this dissertation, we focus on proposing novel techniques for designing more resilient protocols to attackers and more energy efficient protocols. In the first part of the dissertation, we investigate the scenario of multiple adversaries deployed across the network, which reduce significantly the network performance. We adopt a component-based and a cross-layer view of network protocols to make protocols secure and resilient to attacks and to utilize our techniques across existing network protocols. We use the notion of trust between network entities to propose lightweight defense mechanisms, which also satisfy performance requirements. Using cryptographic primitives in our network scenario can introduce significant computational overhead. In addition, behavioral aspects of entities are not captured by cryptographic primitives. Hence, trust metrics provide an efficient security metric in these scenarios, which can be utilized to introduce lightweight defense mechanisms applicable to deployed network protocols. In the second part of the dissertation, we focus on energy efficiency considerations in this type of networks. Our motivation for this work is to extend network lifetime, but at the same time maintain critical performance requirements. We propose a distributed sleep management framework for heterogeneous machine-to-machine networks and two novel energy efficient metrics. This framework and the routing metrics are integrated into existing routing protocols for machine-to-machine networks. We demonstrate the efficiency of our approach in terms of increasing network lifetime and maintaining packet delivery ratio. Furthermore, we propose a novel multi-metric energy efficient routing protocol for dynamic networks (i.e. mobile ad-hoc networks) and illustrate its performance in terms of network lifetime. Finally, we investigate the energy-aware sensor coverage problem and we propose a novel game theoretic approach to capture the tradeoff between sensor coverage efficiency and energy consumption

Efficient radio resource management in next generation wireless networks

The current decade has witnessed a phenomenal growth in mobile wireless communication networks and subscribers. In 2015, mobile wireless devices and connections were reported to have grown to about 7.9 billion, exceeding human population. The explosive growth in mobile wireless communication network subscribers has created a huge demand for wireless network capacity, ubiquitous wireless network coverage, and enhanced Quality of Service (QoS). These demands have led to several challenging problems for wireless communication networks operators and designers. The Next Generation Wireless Networks (NGWNs) will support high mobility communications, such as communication in high-speed rails. Mobile users in such high mobility environment demand reliable QoS, however, such users are plagued with a poor signal-tonoise ratio, due to the high vehicular penetration loss, increased transmission outage and handover information overhead, leading to poor QoS provisioning for the networks' mobile users. Providing a reliable QoS for high mobility users remains one of the unique challenges for NGWNs. The increased wireless network capacity and coverage of NGWNs means that mobile communication users at the cell-edge should have enhanced network performance. However, due to path loss (path attenuation), interference, and radio background noise, mobile communication users at the cell-edge can experience relatively poor transmission channel qualities and subsequently forced to transmit at a low bit transmission rate, even when the wireless communication networks can support high bit transmission rate. Furthermore, the NGWNs are envisioned to be Heterogeneous Wireless Networks (HWNs). The NGWNs are going to be the integration platform of diverse homogeneous wireless communication networks for a convergent wireless communication network. The HWNs support single and multiple calls (group calls), simultaneously. Decision making is an integral core of radio resource management. One crucial decision making in HWNs is network selection. Network selection addresses the problem of how to select the best available access network for a given network user connection. For the integrated platform of HWNs to be truly seamless and efficient, a robust and stable wireless access network selection algorithm is needed. To meet these challenges for the different mobile wireless communication network users, the NGWNs will have to provide a great leap in wireless network capacity, coverage, QoS, and radio resource utilization. Moving wireless communication networks (mobile hotspots) have been proposed as a solution to providing reliable QoS to high mobility users. In this thesis, an Adaptive Thinning Mobility Aware (ATMA) Call Admission Control (CAC) algorithm for improving the QoS and radio resource utilization of the mobile hotspot networks, which are of critical importance for communicating nodes in moving wireless networks is proposed. The performance of proposed ATMA CAC scheme is investigated and compare it with the traditional CAC scheme. The ATMA scheme exploits the mobility events in the highspeed mobility communication environment and the calls (new and handoff calls) generation pattern to enhance the QoS (new call blocking and handoff call dropping probabilities) of the mobile users. The numbers of new and handoff calls in wireless communication networks are dynamic random processes that can be effectively modeled by the Continuous Furthermore, the NGWNs are envisioned to be Heterogeneous Wireless Networks (HWNs). The NGWNs are going to be the integration platform of diverse homogeneous wireless communication networks for a convergent wireless communication network. The HWNs support single and multiple calls (group calls), simultaneously. Decision making is an integral core of radio resource management. One crucial decision making in HWNs is network selection. Network selection addresses the problem of how to select the best available access network for a given network user connection. For the integrated platform of HWNs to be truly seamless and efficient, a robust and stable wireless access network selection algorithm is needed. To meet these challenges for the different mobile wireless communication network users, the NGWNs will have to provide a great leap in wireless network capacity, coverage, QoS, and radio resource utilization. Moving wireless communication networks (mobile hotspots) have been proposed as a solution to providing reliable QoS to high mobility users. In this thesis, an Adaptive Thinning Mobility Aware (ATMA) Call Admission Control (CAC) algorithm for improving the QoS and radio resource utilization of the mobile hotspot networks, which are of critical importance for communicating nodes in moving wireless networks is proposed

Optimization of Energy Harvesting Mobile Nodes Within Scalable Converter System Based on Reinforcement Learning

Microgrid monitoring focusing on power data, such as voltage and current, has become more significant in the development of decentralized power supply system. The power data transmission delay between distributed generator is vital for evaluating the stability and financial outcome of overall grid performance. In this thesis, both hardware and simulation has been discussed for optimizing the data packets transmission delay, energy consumption, and collision rate. To minimize the transmission delay and collision rate, state-action-reward-state-action (SARSA) and Q-learning method based on Markov decision process (MDP) model is used to search the most efficient data transmission scheme for each agent device. A training process comparison between SARSA and Q-learning is given out for representing the training speed of these two methodologies in the scenario of source-relaying-destination model. To balance the exploration and exploitation process involved in these two methods, a parameter is introduced to optimize the cost time of training process. Finally, the simulation result of average throughput and data packets collision rate in the network with 20 agent nodes is presented to indicate the application feasibility of reinforcement learning algorithm in the development of scalable network. The results show that, the average throughput and collision rate stay on the expected ideal performance level for the overall network when the number of nodes is not too large. Also, the hardware development based on Bluetooth Low Energy (BLE) is used to reveal the process of data packets transmission