Two Timescale Convergent Q-learning for Sleep--Scheduling in Wireless Sensor Networks

In this paper, we consider an intrusion detection application for Wireless Sensor Networks (WSNs). We study the problem of scheduling the sleep times of the individual sensors to maximize the network lifetime while keeping the tracking error to a minimum. We formulate this problem as a partially-observable Markov decision process (POMDP) with continuous state-action spaces, in a manner similar to (Fuemmeler and Veeravalli [2008]). However, unlike their formulation, we consider infinite horizon discounted and average cost objectives as performance criteria. For each criterion, we propose a convergent on-policy Q-learning algorithm that operates on two timescales, while employing function approximation to handle the curse of dimensionality associated with the underlying POMDP. Our proposed algorithm incorporates a policy gradient update using a one-simulation simultaneous perturbation stochastic approximation (SPSA) estimate on the faster timescale, while the Q-value parameter (arising from a linear function approximation for the Q-values) is updated in an on-policy temporal difference (TD) algorithm-like fashion on the slower timescale. The feature selection scheme employed in each of our algorithms manages the energy and tracking components in a manner that assists the search for the optimal sleep-scheduling policy. For the sake of comparison, in both discounted and average settings, we also develop a function approximation analogue of the Q-learning algorithm. This algorithm, unlike the two-timescale variant, does not possess theoretical convergence guarantees. Finally, we also adapt our algorithms to include a stochastic iterative estimation scheme for the intruder's mobility model. Our simulation results on a 2-dimensional network setting suggest that our algorithms result in better tracking accuracy at the cost of only a few additional sensors, in comparison to a recent prior work

Multiple abstraction levels in performance analysis of WSN monitoring systems

In this paper, we illustrate the use of different methods to support the design of a Wireless Sensor Network (WSN), by using as a case study a monitoring system that must track a moving object within a given area. The goal of the study is to find a good trade-off between the power consumption and the object tracking reliability. Power saving can be achieved by periodically powering off some of the nodes for a given time interval. Of course nodes can detect the moving object only when they are on, so that the power management strategy can affect the ability to accurately track the object movements. We propose two models and the corresponding analysis and simulation tools, that can be used in a synergistic way: the first model is based on the Markov Decision Well-formed Net (MDWN) formalism while the second one is based on the Stochastic Activity Network (SAN) formalism. The MDWN model is more abstract and is used to compute an optimal power management strategy by solving a Markov Decision Process (MDP); the SAN model is more detailed and is used to perform extensive simulation (using the Mobius tool) in order to analyze different performance indices, both when applying the power management policy derived from the first model and when using different policies