Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Development of Energy and Delay Efficient Protocols for WSAN

Wireless sensor-actor network (WSAN) is a collection of resource conservative sensors and few resource-rich actors. It is widely used in various applications such as environmental monitoring, battlefield surveillance, industrial process control, and home applications. In these real-time applications, data should be delivered with minimum delay and energy. In this thesis, delay and energy efficient protocols are designed to achieve these objectives. The first contribution proposes a delay and energy aware coordination protocol (DEACP) to improve the network performance. It consists of two-level hierarchical K-hop clustering and backup cluster head (BCH) selection mechanism to provide coordination among sensors and actors. Further, a priority based event forwarding mechanism has also been proposed to forward the maximum number of packets within the bounded delay. The simulation results demonstrate the effectiveness of DEACP over existing protocols. In the second work, an interference aware multi-channel MAC protocol (IAMMAC) has been suggested to assign channels for the communication among nodes in the DEACP. An actor assigns the static channels to all of its cluster members for sensor-sensor and sensor-actor coordination. Subsequently, a throughput based dynamic channel selection mechanism has been developed for actor-actor coordination. It is inferred from the simulation results that the proposed IAMMAC protocol outperforms its competitive protocols. Even though its performance is superior, it is susceptible to be attacked because it uses a single static channel between two sensors in the entire communication. To overcome this problem, a lightweight dynamic multi-channel MAC protocol (DM-MAC) has been designed for sensor sensor coordination. Each sensor dynamically selects a channel which provides maximum packet reception ratio among the available hannels with the destination. The comparative analysis shows that DM-MAC protocol performs better than the existing MAC protocols in terms of different performance parameters. WSAN is designed to operate in remote and hostile environments and hence, sensors and actors are vulnerable to various attacks. The fourth contribution proposes a secure coordination mechanism (SCM) to handle the data forwarding attacks in DEACP. In the SCM, each sensor computes the trust level of its neighboring sensors based on the experience, recommendation, and knowledge. The actor analyzes the trust values of all its cluster members to identify the malicious node. Secure hash algorithm-3 is used to compute the message authentication code for the data. The sensor selects a neighbor sensor which has the highest trust value among its 1-hop sensors to transfer data to the actor. The SCM approach outperforms the existing security mechanisms

Multi-objective hierarchical algorithms for restoring Wireless Sensor Network connectivity in known environments

A Wireless Sensor Network can become partitioned due to node failure, requiring the deployment of additional relay nodes in order to restore network connectivity. This introduces an optimisation problem involving a tradeoff between the number of additional nodes that are required and the costs of moving through the sensor field for the purpose of node placement. This tradeoff is application-dependent, influenced for example by the relative urgency of network restoration. We propose a family of algorithms based on hierarchical objectives including complete algorithms and heuristics which integrate network design with path planning, recognising the impact of obstacles on mobility and communication. We conduct an empirical evaluation of the algorithms on random connectivity and mobility graphs, showing their relative performance in terms of node and path costs, and assessing their execution speeds. Finally, we examine how the relative importance of the two objectives influences the choice of algorithm. In summary, the algorithms which prioritise the node cost tend to find graphs with fewer nodes, while the algorithm which prioritise the cost of moving find slightly larger solutions but with cheaper mobility costs. The heuristic algorithms are close to the optimal algorithms in node cost, and higher in mobility costs. For fast moving agents, the node algorithms are preferred for total restoration time, and for slow agents, the path algorithms are preferred

Robotic Wireless Sensor Networks

In this chapter, we present a literature survey of an emerging, cutting-edge, and multi-disciplinary field of research at the intersection of Robotics and Wireless Sensor Networks (WSN) which we refer to as Robotic Wireless Sensor Networks (RWSN). We define a RWSN as an autonomous networked multi-robot system that aims to achieve certain sensing goals while meeting and maintaining certain communication performance requirements, through cooperative control, learning and adaptation. While both of the component areas, i.e., Robotics and WSN, are very well-known and well-explored, there exist a whole set of new opportunities and research directions at the intersection of these two fields which are relatively or even completely unexplored. One such example would be the use of a set of robotic routers to set up a temporary communication path between a sender and a receiver that uses the controlled mobility to the advantage of packet routing. We find that there exist only a limited number of articles to be directly categorized as RWSN related works whereas there exist a range of articles in the robotics and the WSN literature that are also relevant to this new field of research. To connect the dots, we first identify the core problems and research trends related to RWSN such as connectivity, localization, routing, and robust flow of information. Next, we classify the existing research on RWSN as well as the relevant state-of-the-arts from robotics and WSN community according to the problems and trends identified in the first step. Lastly, we analyze what is missing in the existing literature, and identify topics that require more research attention in the future

Energy Management in a Cooperative Energy Harvesting Wireless Sensor Network

In this paper, we consider the problem of finding an optimal energy management policy for a network of sensor nodes capable of harvesting their own energy and sharing it with other nodes in the network. We formulate this problem in the discounted cost Markov decision process framework and obtain good energy-sharing policies using the Deep Deterministic Policy Gradient (DDPG) algorithm. Earlier works have attempted to obtain the optimal energy allocation policy for a single sensor and for multiple sensors arranged on a mote with a single centralized energy buffer. Our algorithms, on the other hand, provide optimal policies for a distributed network of sensors individually harvesting energy and capable of sharing energy amongst themselves. Through simulations, we illustrate that the policies obtained by our DDPG algorithm using this enhanced network model outperform algorithms that do not share energy or use a centralized energy buffer in the distributed multi-nodal case.Comment: 11 pages, 4 figure

Intelligent and Secure Underwater Acoustic Communication Networks

Underwater acoustic (UWA) communication networks are promising techniques for medium- to long-range wireless information transfer in aquatic applications. The harsh and dynamic water environment poses grand challenges to the design of UWA networks. This dissertation leverages the advances in machine learning and signal processing to develop intelligent and secure UWA communication networks. Three research topics are studied: 1) reinforcement learning (RL)-based adaptive transmission in UWA channels; 2) reinforcement learning-based adaptive trajectory planning for autonomous underwater vehicles (AUVs) in under-ice environments; 3) signal alignment to secure underwater coordinated multipoint (CoMP) transmissions. First, a RL-based algorithm is developed for adaptive transmission in long-term operating UWA point-to-point communication systems. The UWA channel dynamics are learned and exploited to trade off energy consumption with information delivery latency. The adaptive transmission problem is formulated as a partially observable Markov decision process (POMDP) which is solved by a Monte Carlo sampling-based approach, and an expectation-maximization-type of algorithm is developed to recursively estimate the channel model parameters. The experimental data processing reveals that the proposed algorithm achieves a good balance between energy efficiency and information delivery latency. Secondly, an online learning-based algorithm is developed for adaptive trajectory planning of multiple AUVs in under-ice environments to reconstruct a water parameter field of interest. The field knowledge is learned online to guide the trajectories of AUVs for collection of informative water parameter samples in the near future. The trajectory planning problem is formulated as a Markov decision process (MDP) which is solved by an actor-critic algorithm, where the field knowledge is estimated online using the Gaussian process regression. The simulation results show that the proposed algorithm achieves the performance close to a benchmark method that assumes perfect field knowledge. Thirdly, the dissertation presents a signal alignment method to secure underwater CoMP transmissions of geographically distributed antenna elements (DAEs) against eavesdropping. Exploiting the low sound speed in water and the spatial diversity of DAEs, the signal alignment method is developed such that useful signals will collide at the eavesdropper while stay collision-free at the legitimate user. The signal alignment mechanism is formulated as a mixed integer and nonlinear optimization problem which is solved through a combination of the simulated annealing method and the linear programming. Taking the orthogonal frequency-division multiplexing (OFDM) as the modulation technique, simulation and emulated experimental results demonstrate that the proposed method significantly degrades the eavesdropper\u27s interception capability

Stochastic Sensor Scheduling via Distributed Convex Optimization

In this paper, we propose a stochastic scheduling strategy for estimating the states of N discrete-time linear time invariant (DTLTI) dynamic systems, where only one system can be observed by the sensor at each time instant due to practical resource constraints. The idea of our stochastic strategy is that a system is randomly selected for observation at each time instant according to a pre-assigned probability distribution. We aim to find the optimal pre-assigned probability in order to minimize the maximal estimate error covariance among dynamic systems. We first show that under mild conditions, the stochastic scheduling problem gives an upper bound on the performance of the optimal sensor selection problem, notoriously difficult to solve. We next relax the stochastic scheduling problem into a tractable suboptimal quasi-convex form. We then show that the new problem can be decomposed into coupled small convex optimization problems, and it can be solved in a distributed fashion. Finally, for scheduling implementation, we propose centralized and distributed deterministic scheduling strategies based on the optimal stochastic solution and provide simulation examples.Comment: Proof errors and typos are fixed. One section is removed from last versio