Deep Reinforcement Learning for Wireless Sensor Scheduling in Cyber-Physical Systems

In many Cyber-Physical Systems, we encounter the problem of remote state estimation of geographically distributed and remote physical processes. This paper studies the scheduling of sensor transmissions to estimate the states of multiple remote, dynamic processes. Information from the different sensors have to be transmitted to a central gateway over a wireless network for monitoring purposes, where typically fewer wireless channels are available than there are processes to be monitored. For effective estimation at the gateway, the sensors need to be scheduled appropriately, i.e., at each time instant one needs to decide which sensors have network access and which ones do not. To address this scheduling problem, we formulate an associated Markov decision process (MDP). This MDP is then solved using a Deep Q-Network, a recent deep reinforcement learning algorithm that is at once scalable and model-free. We compare our scheduling algorithm to popular scheduling algorithms such as round-robin and reduced-waiting-time, among others. Our algorithm is shown to significantly outperform these algorithms for many example scenarios

Optimal Energy Allocation for Kalman Filtering over Packet Dropping Links with Imperfect Acknowledgments and Energy Harvesting Constraints

This paper presents a design methodology for optimal transmission energy allocation at a sensor equipped with energy harvesting technology for remote state estimation of linear stochastic dynamical systems. In this framework, the sensor measurements as noisy versions of the system states are sent to the receiver over a packet dropping communication channel. The packet dropout probabilities of the channel depend on both the sensor's transmission energies and time varying wireless fading channel gains. The sensor has access to an energy harvesting source which is an everlasting but unreliable energy source compared to conventional batteries with fixed energy storages. The receiver performs optimal state estimation with random packet dropouts to minimize the estimation error covariances based on received measurements. The receiver also sends packet receipt acknowledgments to the sensor via an erroneous feedback communication channel which is itself packet dropping. The objective is to design optimal transmission energy allocation at the energy harvesting sensor to minimize either a finite-time horizon sum or a long term average (infinite-time horizon) of the trace of the expected estimation error covariance of the receiver's Kalman filter. These problems are formulated as Markov decision processes with imperfect state information. The optimal transmission energy allocation policies are obtained by the use of dynamic programming techniques. Using the concept of submodularity, the structure of the optimal transmission energy policies are studied. Suboptimal solutions are also discussed which are far less computationally intensive than optimal solutions. Numerical simulation results are presented illustrating the performance of the energy allocation algorithms.Comment: Submitted to IEEE Transactions on Automatic Control. arXiv admin note: text overlap with arXiv:1402.663

An Optimal Transmission Strategy for Kalman Filtering over Packet Dropping Links with Imperfect Acknowledgements

This paper presents a novel design methodology for optimal transmission policies at a smart sensor to remotely estimate the state of a stable linear stochastic dynamical system. The sensor makes measurements of the process and forms estimates of the state using a local Kalman filter. The sensor transmits quantized information over a packet dropping link to the remote receiver. The receiver sends packet receipt acknowledgments back to the sensor via an erroneous feedback communication channel which is itself packet dropping. The key novelty of this formulation is that the smart sensor decides, at each discrete time instant, whether to transmit a quantized version of either its local state estimate or its local innovation. The objective is to design optimal transmission policies in order to minimize a long term average cost function as a convex combination of the receiver's expected estimation error covariance and the energy needed to transmit the packets. The optimal transmission policy is obtained by the use of dynamic programming techniques. Using the concept of submodularity, the optimality of a threshold policy in the case of scalar systems with perfect packet receipt acknowledgments is proved. Suboptimal solutions and their structural results are also discussed. Numerical results are presented illustrating the performance of the optimal and suboptimal transmission policies.Comment: Conditionally accepted in IEEE Transactions on Control of Network System

Proactive Caching for Energy-Efficiency in Wireless Networks: A Markov Decision Process Approach

Content caching in wireless networks provides a substantial opportunity to trade off low cost memory storage with energy consumption, yet finding the optimal causal policy with low computational complexity remains a challenge. This paper models the Joint Pushing and Caching (JPC) problem as a Markov Decision Process (MDP) and provides a solution to determine the optimal randomized policy. A novel approach to decouple the influence from buffer occupancy and user requests is proposed to turn the high-dimensional optimization problem into three low-dimensional ones. Furthermore, a non-iterative algorithm to solve one of the sub-problems is presented, exploiting a structural property we found as \textit{generalized monotonicity}, and hence significantly reduces the computational complexity. The result attains close performance in comparison with theoretical bounds from non-practical policies, while benefiting from higher time efficiency than the unadapted MDP solution.Comment: 6 pages, 6 figures, submitted to IEEE International Conference on Communications 201

Computation-Communication Trade-offs and Sensor Selection in Real-time Estimation for Processing Networks

Recent advances in electronics are enabling substantial processing to be performed at each node (robots, sensors) of a networked system. Local processing enables data compression and may mitigate measurement noise, but it is still slower compared to a central computer (it entails a larger computational delay). However, while nodes can process the data in parallel, the centralized computational is sequential in nature. On the other hand, if a node sends raw data to a central computer for processing, it incurs communication delay. This leads to a fundamental communication-computation trade-off, where each node has to decide on the optimal amount of preprocessing in order to maximize the network performance. We consider a network in charge of estimating the state of a dynamical system and provide three contributions. First, we provide a rigorous problem formulation for optimal real-time estimation in processing networks in the presence of delays. Second, we show that, in the case of a homogeneous network (where all sensors have the same computation) that monitors a continuous-time scalar linear system, the optimal amount of local preprocessing maximizing the network estimation performance can be computed analytically. Third, we consider the realistic case of a heterogeneous network monitoring a discrete-time multi-variate linear system and provide algorithms to decide on suitable preprocessing at each node, and to select a sensor subset when computational constraints make using all sensors suboptimal. Numerical simulations show that selecting the sensors is crucial. Moreover, we show that if the nodes apply the preprocessing policy suggested by our algorithms, they can largely improve the network estimation performance.Comment: 15 pages, 16 figures. Accepted journal versio

Optimal Sensor Collaboration for Parameter Tracking Using Energy Harvesting Sensors

In this paper, we design an optimal sensor collaboration strategy among neighboring nodes while tracking a time-varying parameter using wireless sensor networks in the presence of imperfect communication channels. The sensor network is assumed to be self-powered, where sensors are equipped with energy harvesters that replenish energy from the environment. In order to minimize the mean square estimation error of parameter tracking, we propose an online sensor collaboration policy subject to real-time energy harvesting constraints. The proposed energy allocation strategy is computationally light and only relies on the second-order statistics of the system parameters. For this, we first consider an offline non-convex optimization problem, which is solved exactly using semidefinite programming. Based on the offline solution, we design an online power allocation policy that requires minimal online computation and satisfies the dynamics of energy flow at each sensor. We prove that the proposed online policy is asymptotically equivalent to the optimal offline solution and show its convergence rate and robustness. We empirically show that the estimation performance of the proposed online scheme is better than that of the online scheme when channel state information about the dynamical system is available in the low SNR regime. Numerical results are conducted to demonstrate the effectiveness of our approach