Multitarget tracking via restless bandit marginal productivity indices and Kalman Filter in discrete time

This paper designs, evaluates, and tests a tractable priority-index policy for scheduling target updates in a discrete-time multitarget tracking model, which aims to be close to optimal relative to a discounted or average performance objective accounting for tracking-error variance and measurement costs. The policy is to be used by a sensor system composed of M phased-array radars coordinated to track the positions of N targets moving according to independent scalar Gauss-Markov linear dynamics, which therefore allows for the use of the Kalman Filter for track estimation. The paper exploits the natural problem formulation as a multiarmed restless bandit problem (MARBP) with real-state projects subject to deterministic dynamics by deploying Whittle's (1988) index policy for the MARBP. The challenging issues of indexability (existence of the index) and index evaluation are resolved by applying a method recently introduced by the first author for the analysis of real-state restless bandits. Computational results are reported demonstrating the tractability of index evaluation, the substantial performance gains that the Whittle's marginal productivity (MP) index policy achieves against myopic policies advocated in previous work and the resulting index policies suboptimality gaps. Further, a preliminary small scale computational study shows that the (MP) index policy exhibits a nearly optimal behavior as the number of distinct objective targets grows with the number of radars per target constant

Caching Contents with Varying Popularity using Restless Bandits

We study content caching in a wireless network in which the users are connected through a base station that is equipped with a finite-capacity cache. We assume a fixed set of contents whose popularity varies with time. Users' requests for the content depend on their instantaneous popularity levels. Proactively caching contents at the base station incurs a cost but not having requested contents at the base station also incurs a cost. We propose to proactively cache contents at the base station so as to minimize content missing and caching costs. We formulate the problem as a discounted cost Markov decision problem that is a restless multi-armed bandit problem. We provide conditions under which the problem is indexable and also propose a novel approach to maneuver a few parameters to render the problem indexable. We demonstrate the efficacy of the Whittle index policy via numerical evaluation

An Index Policy for Minimizing the Uncertainty-of-Information of Markov Sources

This paper focuses on the information freshness of finite-state Markov sources, using the uncertainty of information (UoI) as the performance metric. Measured by Shannon's entropy, UoI can capture not only the transition dynamics of the Markov source but also the different evolutions of information quality caused by the different values of the last observation. We consider an information update system with M finite-state Markov sources transmitting information to a remote monitor via m communication channels. Our goal is to explore the optimal scheduling policy to minimize the sum-UoI of the Markov sources. The problem is formulated as a restless multi-armed bandit (RMAB). We relax the RMAB and then decouple the relaxed problem into M single bandit problems. Analyzing the single bandit problem provides useful properties with which the relaxed problem reduces to maximizing a concave and piecewise linear function, allowing us to develop a gradient method to solve the relaxed problem and obtain its optimal policy. By rounding up the optimal policy for the relaxed problem, we obtain an index policy for the original RMAB problem. Notably, the proposed index policy is universal in the sense that it applies to general RMABs with bounded cost functions.Comment: 55 page