Two Timescale Convergent Q-learning for Sleep--Scheduling in Wireless Sensor Networks

In this paper, we consider an intrusion detection application for Wireless Sensor Networks (WSNs). We study the problem of scheduling the sleep times of the individual sensors to maximize the network lifetime while keeping the tracking error to a minimum. We formulate this problem as a partially-observable Markov decision process (POMDP) with continuous state-action spaces, in a manner similar to (Fuemmeler and Veeravalli [2008]). However, unlike their formulation, we consider infinite horizon discounted and average cost objectives as performance criteria. For each criterion, we propose a convergent on-policy Q-learning algorithm that operates on two timescales, while employing function approximation to handle the curse of dimensionality associated with the underlying POMDP. Our proposed algorithm incorporates a policy gradient update using a one-simulation simultaneous perturbation stochastic approximation (SPSA) estimate on the faster timescale, while the Q-value parameter (arising from a linear function approximation for the Q-values) is updated in an on-policy temporal difference (TD) algorithm-like fashion on the slower timescale. The feature selection scheme employed in each of our algorithms manages the energy and tracking components in a manner that assists the search for the optimal sleep-scheduling policy. For the sake of comparison, in both discounted and average settings, we also develop a function approximation analogue of the Q-learning algorithm. This algorithm, unlike the two-timescale variant, does not possess theoretical convergence guarantees. Finally, we also adapt our algorithms to include a stochastic iterative estimation scheme for the intruder's mobility model. Our simulation results on a 2-dimensional network setting suggest that our algorithms result in better tracking accuracy at the cost of only a few additional sensors, in comparison to a recent prior work

Analysis, classification and comparison of scheduling techniques for software transactional memories

Transactional Memory (TM) is a practical programming paradigm for developing concurrent applications. Performance is a critical factor for TM implementations, and various studies demonstrated that specialised transaction/thread scheduling support is essential for implementing performance-effective TM systems. After one decade of research, this article reviews the wide variety of scheduling techniques proposed for Software Transactional Memories. Based on peculiarities and differences of the adopted scheduling strategies, we propose a classification of the existing techniques, and we discuss the specific characteristics of each technique. Also, we analyse the results of previous evaluation and comparison studies, and we present the results of a new experimental study encompassing techniques based on different scheduling strategies. Finally, we identify potential strengths and weaknesses of the different techniques, as well as the issues that require to be further investigated

Security in networks of unmanned aerial vehicles for surveillance with an agent-based approach inspired by the principles of blockchain

Unmanned aerial vehicles (UAVs) can support surveillance even in areas without network infrastructure. However, UAV networks raise security challenges because of its dynamic topology. This paper proposes a technique for maintaining security in UAV networks in the context of surveillance, by corroborating information about events from different sources. In this way, UAV networks can conform peer-to-peer information inspired by the principles of blockchain, and detect compromised UAVs based on trust policies. The proposed technique uses a secure asymmetric encryption with a pre-shared list of official UAVs. Using this technique, the wrong information can be detected when an official UAV is physically hijacked. The novel agent based simulator ABS-SecurityUAV is used to validate the proposed approach. In our experiments, around 90% of UAVs were able to corroborate information about a person walking in a controlled area, while none of the UAVs corroborated fake information coming from a hijacked UAV

Adapting Search Theory to Networks

The CSE is interested in the general problem of locating objects in networks. Because of their exposure to search theory, the problem they brought to the workshop was phrased in terms of adapting search theory to networks. Thus, the first step was the introduction of an already existing healthy literature on searching graphs. T. D. Parsons, who was then at Pennsylvania State University, was approached in 1977 by some local spelunkers who asked his aid in optimizing a search for someone lost in a cave in Pennsylvania. Parsons quickly formulated the problem as a search problem in a graph. Subsequent papers led to two divergent problems. One problem dealt with searching under assumptions of fairly extensive information, while the other problem dealt with searching under assumptions of essentially zero information. These two topics are developed in the next two sections