Two Timescale Convergent Q-learning for Sleep--Scheduling in Wireless Sensor Networks

In this paper, we consider an intrusion detection application for Wireless Sensor Networks (WSNs). We study the problem of scheduling the sleep times of the individual sensors to maximize the network lifetime while keeping the tracking error to a minimum. We formulate this problem as a partially-observable Markov decision process (POMDP) with continuous state-action spaces, in a manner similar to (Fuemmeler and Veeravalli [2008]). However, unlike their formulation, we consider infinite horizon discounted and average cost objectives as performance criteria. For each criterion, we propose a convergent on-policy Q-learning algorithm that operates on two timescales, while employing function approximation to handle the curse of dimensionality associated with the underlying POMDP. Our proposed algorithm incorporates a policy gradient update using a one-simulation simultaneous perturbation stochastic approximation (SPSA) estimate on the faster timescale, while the Q-value parameter (arising from a linear function approximation for the Q-values) is updated in an on-policy temporal difference (TD) algorithm-like fashion on the slower timescale. The feature selection scheme employed in each of our algorithms manages the energy and tracking components in a manner that assists the search for the optimal sleep-scheduling policy. For the sake of comparison, in both discounted and average settings, we also develop a function approximation analogue of the Q-learning algorithm. This algorithm, unlike the two-timescale variant, does not possess theoretical convergence guarantees. Finally, we also adapt our algorithms to include a stochastic iterative estimation scheme for the intruder's mobility model. Our simulation results on a 2-dimensional network setting suggest that our algorithms result in better tracking accuracy at the cost of only a few additional sensors, in comparison to a recent prior work

Space-Time Sampling for Network Observability

Designing sparse sampling strategies is one of the important components in having resilient estimation and control in networked systems as they make network design problems more cost-effective due to their reduced sampling requirements and less fragile to where and when samples are collected. It is shown that under what conditions taking coarse samples from a network will contain the same amount of information as a more finer set of samples. Our goal is to estimate initial condition of linear time-invariant networks using a set of noisy measurements. The observability condition is reformulated as the frame condition, where one can easily trace location and time stamps of each sample. We compare estimation quality of various sampling strategies using estimation measures, which depend on spectrum of the corresponding frame operators. Using properties of the minimal polynomial of the state matrix, deterministic and randomized methods are suggested to construct observability frames. Intrinsic tradeoffs assert that collecting samples from fewer subsystems dictates taking more samples (in average) per subsystem. Three scalable algorithms are developed to generate sparse space-time sampling strategies with explicit error bounds.Comment: Submitted to IEEE TAC (Revised Version

Adaptive smartphone-based sensor fusion for estimating competitive rowing kinematic metrics.

Competitive rowing highly values boat position and velocity data for real-time feedback during training, racing and post-training analysis. The ubiquity of smartphones with embedded position (GPS) and motion (accelerometer) sensors motivates their possible use in these tasks. In this paper, we investigate the use of two real-time digital filters to achieve highly accurate yet reasonably priced measurements of boat speed and distance traveled. Both filters combine acceleration and location data to estimate boat distance and speed; the first using a complementary frequency response-based filter technique, the second with a Kalman filter formalism that includes adaptive, real-time estimates of effective accelerometer bias. The estimates of distance and speed from both filters were validated and compared with accurate reference data from a differential GPS system with better than 1 cm precision and a 5 Hz update rate, in experiments using two subjects (an experienced club-level rower and an elite rower) in two different boats on a 300 m course. Compared with single channel (smartphone GPS only) measures of distance and speed, the complementary filter improved the accuracy and precision of boat speed, boat distance traveled, and distance per stroke by 44%, 42%, and 73%, respectively, while the Kalman filter improved the accuracy and precision of boat speed, boat distance traveled, and distance per stroke by 48%, 22%, and 82%, respectively. Both filters demonstrate promise as general purpose methods to substantially improve estimates of important rowing performance metrics

Gossip Algorithms for Distributed Signal Processing

Gossip algorithms are attractive for in-network processing in sensor networks because they do not require any specialized routing, there is no bottleneck or single point of failure, and they are robust to unreliable wireless network conditions. Recently, there has been a surge of activity in the computer science, control, signal processing, and information theory communities, developing faster and more robust gossip algorithms and deriving theoretical performance guarantees. This article presents an overview of recent work in the area. We describe convergence rate results, which are related to the number of transmitted messages and thus the amount of energy consumed in the network for gossiping. We discuss issues related to gossiping over wireless links, including the effects of quantization and noise, and we illustrate the use of gossip algorithms for canonical signal processing tasks including distributed estimation, source localization, and compression.Comment: Submitted to Proceedings of the IEEE, 29 page