Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Stochastic Sensor Scheduling via Distributed Convex Optimization

In this paper, we propose a stochastic scheduling strategy for estimating the states of N discrete-time linear time invariant (DTLTI) dynamic systems, where only one system can be observed by the sensor at each time instant due to practical resource constraints. The idea of our stochastic strategy is that a system is randomly selected for observation at each time instant according to a pre-assigned probability distribution. We aim to find the optimal pre-assigned probability in order to minimize the maximal estimate error covariance among dynamic systems. We first show that under mild conditions, the stochastic scheduling problem gives an upper bound on the performance of the optimal sensor selection problem, notoriously difficult to solve. We next relax the stochastic scheduling problem into a tractable suboptimal quasi-convex form. We then show that the new problem can be decomposed into coupled small convex optimization problems, and it can be solved in a distributed fashion. Finally, for scheduling implementation, we propose centralized and distributed deterministic scheduling strategies based on the optimal stochastic solution and provide simulation examples.Comment: Proof errors and typos are fixed. One section is removed from last versio

Wireless Power Transfer and Data Collection in Wireless Sensor Networks

In a rechargeable wireless sensor network, the data packets are generated by sensor nodes at a specific data rate, and transmitted to a base station. Moreover, the base station transfers power to the nodes by using Wireless Power Transfer (WPT) to extend their battery life. However, inadequately scheduling WPT and data collection causes some of the nodes to drain their battery and have their data buffer overflow, while the other nodes waste their harvested energy, which is more than they need to transmit their packets. In this paper, we investigate a novel optimal scheduling strategy, called EHMDP, aiming to minimize data packet loss from a network of sensor nodes in terms of the nodes' energy consumption and data queue state information. The scheduling problem is first formulated by a centralized MDP model, assuming that the complete states of each node are well known by the base station. This presents the upper bound of the data that can be collected in a rechargeable wireless sensor network. Next, we relax the assumption of the availability of full state information so that the data transmission and WPT can be semi-decentralized. The simulation results show that, in terms of network throughput and packet loss rate, the proposed algorithm significantly improves the network performance.Comment: 30 pages, 8 figures, accepted to IEEE Transactions on Vehicular Technolog