Deep Reinforcement Learning for Wireless Sensor Scheduling in Cyber-Physical Systems

In many Cyber-Physical Systems, we encounter the problem of remote state estimation of geographically distributed and remote physical processes. This paper studies the scheduling of sensor transmissions to estimate the states of multiple remote, dynamic processes. Information from the different sensors have to be transmitted to a central gateway over a wireless network for monitoring purposes, where typically fewer wireless channels are available than there are processes to be monitored. For effective estimation at the gateway, the sensors need to be scheduled appropriately, i.e., at each time instant one needs to decide which sensors have network access and which ones do not. To address this scheduling problem, we formulate an associated Markov decision process (MDP). This MDP is then solved using a Deep Q-Network, a recent deep reinforcement learning algorithm that is at once scalable and model-free. We compare our scheduling algorithm to popular scheduling algorithms such as round-robin and reduced-waiting-time, among others. Our algorithm is shown to significantly outperform these algorithms for many example scenarios

Stochastic Sensor Scheduling via Distributed Convex Optimization

In this paper, we propose a stochastic scheduling strategy for estimating the states of N discrete-time linear time invariant (DTLTI) dynamic systems, where only one system can be observed by the sensor at each time instant due to practical resource constraints. The idea of our stochastic strategy is that a system is randomly selected for observation at each time instant according to a pre-assigned probability distribution. We aim to find the optimal pre-assigned probability in order to minimize the maximal estimate error covariance among dynamic systems. We first show that under mild conditions, the stochastic scheduling problem gives an upper bound on the performance of the optimal sensor selection problem, notoriously difficult to solve. We next relax the stochastic scheduling problem into a tractable suboptimal quasi-convex form. We then show that the new problem can be decomposed into coupled small convex optimization problems, and it can be solved in a distributed fashion. Finally, for scheduling implementation, we propose centralized and distributed deterministic scheduling strategies based on the optimal stochastic solution and provide simulation examples.Comment: Proof errors and typos are fixed. One section is removed from last versio

Timer-Based Distributed Channel Access in Networked Control Systems over Known and Unknown Gilbert-Elliott Channels

In this paper, we consider a system consisting of multiple (possibly heterogeneous) decoupled control subsystems which aim at communicating with their corresponding controllers via shared (possibly) time-varying wireless channels. To address the resource allocation problem in a distributed fashion, we propose a timer-based channel access mechanism in which the subsystem with the smallest timer value, in a channel, claims the slot for transmission in that specific channel. The value of the timer is inversely proportional to a cost which is a function of the temporal correlation in the channel variation and the subsystem state. This cost can be calculated individually and does not require explicit communication between the subsystems, since it is based on locally available information only. The temporal correlation in the channel variation may be unknown and, in such cases, each subsystem tries to deduce it via machine learning techniques. The performance of our proposed mechanism is demonstrated via simulations

Real-time Sampling and Estimation on Random Access Channels: Age of Information and Beyond

Efficient sampling and remote estimation are critical for a plethora of wireless-empowered applications in the Internet of Things and cyber-physical systems. Motivated by such applications, this work proposes decentralized policies for the real-time monitoring and estimation of autoregressive processes over random access channels. Two classes of policies are investigated: (i) oblivious schemes in which sampling and transmission policies are independent of the processes that are monitored, and (ii) non-oblivious schemes in which transmitters causally observe their corresponding processes for decision making. In the class of oblivious policies, we show that minimizing the expected time-average estimation error is equivalent to minimizing the expected age of information. Consequently, we prove lower and upper bounds on the minimum achievable estimation error in this class. Next, we consider non-oblivious policies and design a threshold policy, called error-based thinning, in which each source node becomes active if its instantaneous error has crossed a fixed threshold (which we optimize). Active nodes then transmit stochastically following a slotted ALOHA policy. A closed-form, approximately optimal, solution is found for the threshold as well as the resulting estimation error. It is shown that non-oblivious policies offer a multiplicative gain close to $3$ compared to oblivious policies. Moreover, it is shown that oblivious policies that use the age of information for decision making improve the state-of-the-art at least by the multiplicative factor $2$. The performance of all discussed policies is compared using simulations. The numerical comparison shows that the performance of the proposed decentralized policy is very close to that of centralized greedy scheduling