Two Timescale Convergent Q-learning for Sleep--Scheduling in Wireless Sensor Networks

In this paper, we consider an intrusion detection application for Wireless Sensor Networks (WSNs). We study the problem of scheduling the sleep times of the individual sensors to maximize the network lifetime while keeping the tracking error to a minimum. We formulate this problem as a partially-observable Markov decision process (POMDP) with continuous state-action spaces, in a manner similar to (Fuemmeler and Veeravalli [2008]). However, unlike their formulation, we consider infinite horizon discounted and average cost objectives as performance criteria. For each criterion, we propose a convergent on-policy Q-learning algorithm that operates on two timescales, while employing function approximation to handle the curse of dimensionality associated with the underlying POMDP. Our proposed algorithm incorporates a policy gradient update using a one-simulation simultaneous perturbation stochastic approximation (SPSA) estimate on the faster timescale, while the Q-value parameter (arising from a linear function approximation for the Q-values) is updated in an on-policy temporal difference (TD) algorithm-like fashion on the slower timescale. The feature selection scheme employed in each of our algorithms manages the energy and tracking components in a manner that assists the search for the optimal sleep-scheduling policy. For the sake of comparison, in both discounted and average settings, we also develop a function approximation analogue of the Q-learning algorithm. This algorithm, unlike the two-timescale variant, does not possess theoretical convergence guarantees. Finally, we also adapt our algorithms to include a stochastic iterative estimation scheme for the intruder's mobility model. Our simulation results on a 2-dimensional network setting suggest that our algorithms result in better tracking accuracy at the cost of only a few additional sensors, in comparison to a recent prior work

Dynamic Algorithms for Sensor Scheduling and Adversary Path Prediction

In this thesis we describe three new dynamic, real time and robust sensor scheduling algorithms for intruder tracking and sensor scheduling. We call them Tactic Association Based Algorithm (TABA), Tactic Case Based Algorithm (TCBA) and Tactic Weight Based Algorithm (TWBA). The algorithms are encoded, illustrated visually, validated, and tested. The aim of the algorithms is to efficiently track an intruder or multiple intruders while minimizing energy usage in the sensor network by using real time event driven sensor scheduling. What makes these intrusion detection schemes different from other intrusion detection schemes in the literature is the use of historical data in path prediction and sensor scheduling. The TABA uses sequence pattern mining to generate confidences of movement of an intruder from one location to another location in the sensor network. TCBA uses the Case-based reasoning approach to schedule sensors and track intruders in the wireless sensor network. TWBA uses weighted hexagonal representation of the sensor network to schedule sensors and track intruders. In this research we also introduce a novel approach to generate probable intruder paths which are strong representatives of the paths intruders would take when moving through the sensor network

QoS Provision for Wireless Sensor Networks

Wireless sensor network is a fast growing area of research, receiving attention not only within the computer science and electrical engineering communities, but also in relation to network optimization, scheduling, risk and reliability analysis within industrial and system engineering. The availability of micro-sensors and low-power wireless communications will enable the deployment of densely distributed sensor/actuator networks. And an integration of such system plays critical roles in many facets of human life ranging from intelligent assistants in hospitals to manufacturing process, to rescue agents in large scale disaster response, to sensor networks tracking environment phenomena, and others. The sensor nodes will perform significant signal processing, computation, and network self-configuration to achieve scalable, secure, robust and long-lived networks. More specifically, sensor nodes will do local processing to reduce energy costs, and key exchanges to ensure robust communications. These requirements pose interesting challenges for networking research. The most important technical challenge arises from the development of an integrated system which is 1)energy efficient because the system must be long-lived and operate without manual intervention, 2)reliable for data communication and robust to attackers because information security and system robustness are important in sensitive applications, such as military. Based on the above challenges, this dissertation provides Quality of Service (QoS) implementation and evaluation for the wireless sensor networks. It includes the following 3 modules, 1) energy-efficient routing, 2) energy-efficient coverage, 3). communication security. Energy-efficient routing combines the features of minimum energy consumption routing protocols with minimum computational cost routing protocols. Energy-efficient coverage provides on-demand sensing and measurement. Information security needs a security key exchange scheme to ensure reliable and robust communication links. QoS evaluation metrics and results are presented based on the above requirements

Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Wake-up receivers for wireless sensor networks: benefits and challenges

For successful data delivery, the destination nodes should be listening to the medium to receive data when the sender node starts data communication. To achieve this synchronization, there are different rendezvous schemes, among which the most energy-efficient is utilizing wakeup receivers. Current hardware technologies of wake-up receivers enable us to evaluate them as a promising solution for wireless sensor networks. In this article the benefits achieved with wake-up receivers are investigated along with the challenges observed. In addition, an overview of state-of-the-art hardware and networking protocol proposals is presented. As wake-up receivers offer new opportunities, new potential application areas are also presented and discussed.Peer ReviewedPostprint (published version