Two Timescale Convergent Q-learning for Sleep--Scheduling in Wireless Sensor Networks

In this paper, we consider an intrusion detection application for Wireless Sensor Networks (WSNs). We study the problem of scheduling the sleep times of the individual sensors to maximize the network lifetime while keeping the tracking error to a minimum. We formulate this problem as a partially-observable Markov decision process (POMDP) with continuous state-action spaces, in a manner similar to (Fuemmeler and Veeravalli [2008]). However, unlike their formulation, we consider infinite horizon discounted and average cost objectives as performance criteria. For each criterion, we propose a convergent on-policy Q-learning algorithm that operates on two timescales, while employing function approximation to handle the curse of dimensionality associated with the underlying POMDP. Our proposed algorithm incorporates a policy gradient update using a one-simulation simultaneous perturbation stochastic approximation (SPSA) estimate on the faster timescale, while the Q-value parameter (arising from a linear function approximation for the Q-values) is updated in an on-policy temporal difference (TD) algorithm-like fashion on the slower timescale. The feature selection scheme employed in each of our algorithms manages the energy and tracking components in a manner that assists the search for the optimal sleep-scheduling policy. For the sake of comparison, in both discounted and average settings, we also develop a function approximation analogue of the Q-learning algorithm. This algorithm, unlike the two-timescale variant, does not possess theoretical convergence guarantees. Finally, we also adapt our algorithms to include a stochastic iterative estimation scheme for the intruder's mobility model. Our simulation results on a 2-dimensional network setting suggest that our algorithms result in better tracking accuracy at the cost of only a few additional sensors, in comparison to a recent prior work

Intensity based interrogation of optical fibre sensors for industrial automation and intrusion detection systems

In this study, the use of optical fibre sensors for intrusion detection and industrial automation systems has been demonstrated, with a particular focus on low cost, intensity-based, interrogation techniques. The use of optical fibre sensors for intrusion detection systems to secure residential, commercial, and industrial premises against potential security breaches has been extensively reviewed in this thesis. Fibre Bragg grating (FBG) sensing is one form of optical fibre sensing that has been underutilised in applications such as in-ground, in-fence, and window and door monitoring, and addressing that opportunity has been a major goal of this thesis. Both security and industrial sensor systems must include some centralised intelligence (electronic controller) and ideally both automation and security sensor systems would be controlled and monitored by the same centralised system. Optical fibre sensor systems that could be used for either application have been designed, developed, and tested in this study, and optoelectronic interfaces for integrating these sensors with electronic controllers have been demonstrated. The versatility of FBG sensors means that they are also ideal for certain mainstream industrial applications. Two novel transducers have been developed in this work; a highly sensitive low pressure FBG diaphragm transducer and a FBG load cell transducer. Both have been designed to allow interrogation of the optical signal could occur within the housing of the individual sensors themselves. This is achieved in a simple and low cost manner that enables the output of the transducers to be easily connected to standard electronic controllers, such as programmable logic controllers. Furthermore, some of the nonlinear characteristics of FBG sensors have been explored with the aim of developing transducers that are inherently decoupled from strain and temperature interference. One of the major advantages of optical fibre sensors is their ability to be both time division and wavelength division multiplexed. The intensity-based interrogation techniques used here complement this attribute and are a major consideration when developing the transducers and optoelectronic circuits. A time division multiplexing technique, using transmit-reflect detection and incorporating a dual bus, has also been developed. This system architecture enables all the different optical fibre transducers on the network to have the same Bragg wavelength and hence the number of spare replacement transducers required is minimal. Moreover, sensors can be replaced in an online control system without disrupting the network. In addition, by analysing both the transmitted and reflected signals, problems associated with optical power fluctuations are eliminated and the intensity of the sensor signals is increased through differential amplification. Overall, the research addresses the limitations of conventional electrical sensors, such as susceptibility to corrosive damage in wet and corrosive environments, and risk of causing an explosion in hazardous environments, as well as the limitations of current stand-alone optical fibre sensor systems. This thesis supports more alert, reliable, affordable, and coordinated, control and monitoring systems in an on-line environment