Experimental Analysis of Reinforcement Learning Techniques for Spectrum Sharing Radar

In this work, we first describe a framework for the application of Reinforcement Learning (RL) control to a radar system that operates in a congested spectral setting. We then compare the utility of several RL algorithms through a discussion of experiments performed on Commercial off-the-shelf (COTS) hardware. Each RL technique is evaluated in terms of convergence, radar detection performance achieved in a congested spectral environment, and the ability to share 100MHz spectrum with an uncooperative communications system. We examine policy iteration, which solves an environment posed as a Markov Decision Process (MDP) by directly solving for a stochastic mapping between environmental states and radar waveforms, as well as Deep RL techniques, which utilize a form of Q-Learning to approximate a parameterized function that is used by the radar to select optimal actions. We show that RL techniques are beneficial over a Sense-and-Avoid (SAA) scheme and discuss the conditions under which each approach is most effective.Comment: Accepted for publication at IEEE Intl. Radar Conference, Washington DC, Apr. 2020. This is the author's version of the wor

26th Annual Computational Neuroscience Meeting (CNS*2017): Part 3 - Meeting Abstracts - Antwerp, Belgium. 15–20 July 2017

This work was produced as part of the activities of FAPESP Research,\ud Disseminations and Innovation Center for Neuromathematics (grant\ud 2013/07699-0, S. Paulo Research Foundation). NLK is supported by a\ud FAPESP postdoctoral fellowship (grant 2016/03855-5). ACR is partially\ud supported by a CNPq fellowship (grant 306251/2014-0)

Distributed Online Learning for Coexistence in Cognitive Radar Networks

This work addresses the coexistence problem for radar networks. Specifically, we model a network of cooperative, independent, and non-communicating radar nodes which must share resources within the network as well as with non-cooperative nearby emitters. We approach this problem using online Machine Learning (ML) techniques. Online learning approaches are specifically preferred due to the fact that each radar node has no prior knowledge of the environment nor of the positions of the other radar nodes, and due to the sequential nature of the problem. For this task we specifically select the multi-player multi-armed bandit (MMAB) model, which poses the problem as a sequential game, where each radar node in a network makes independent selections of center frequency and waveform with the same goal of improving tracking performance for the network as a whole. For accurate tracking, each radar node communicates observations to a fusion center on set intervals. The fusion center has knowledge of the radar node placement, but cannot communicate to the individual nodes fast enough for waveform control. Every radar node in the network must learn the behavior of the environment, which includes rewards, interferer behavior, and target behavior. Each independent and identical node must choose one of many waveforms to transmit in each Pulse Repetition Interval (PRI) while avoiding \emph{collisions} with other nodes and interference from the environment. The goal for the network as a whole is to minimize target tracking error, which relies on obtaining high SINR in each time step. Our contributions include a mathematical description of the MMAB framework adapted to the radar network scenario. We conclude with a simulation study of several different network configurations. Experimental results show that iterative, online learning using MMAB outperforms the more traditional sense-and-avoid (SAA) and fixed-allocation approaches

Timely Target Tracking: Distributed Updating in Cognitive Radar Networks

Cognitive radar networks are capable of optimizing operating parameters in order to provide actionable information to an operator or secondary system. CRNs have been proposed to answer the need for low-cost devices tracking potentially large numbers of targets in geographically diverse regions. Networks of small-scale devices have also been shown to outperform legacy, large scale, high price, single-device installations. In this work, we consider a CRN tracking multiple targets with a goal of providing information which is both fresh and accurate to a measurement fusion center. We show that under a constraint on the update rate of each radar node, the network is able to utilize Age of Information metrics to maximize the resource utilization and minimize error per track. Since information freshness is critical to decision-making, this structure enables a CRN to provide the highest-quality information possible to a downstream system or operator. We discuss centralized and distributed approaches to solving this problem, taking into account the quality of node observations, the maneuverability of each target, and a limit on the rate at which any node may provide updates to the FC. We present a centralized AoI-inspired node selection metric, where a FC requests updates from specific nodes. We compare this against several alternative techniques. Further, we provide a distributed approach which utilizes the Age of Incorrect Information metric, allowing each independent node to provide updates according to the targets it can observe. We provide mathematical analysis of the rate limits defined for the centralized and distributed approaches, showing that they are equivalent. We conclude with numerical simulations demonstrating that the performance of the algorithms exceeds that of alternative approaches, both in resource utilization and in tracking performance.Comment: 12 pages, double column, 13 figure

Adversarial Multi-Player Bandits for Cognitive Radar Networks

We model a radar network as an adversarial bandit problem, where the environment pre-selects reward sequences for each of several actions available to the network. This excludes environments which vary rewards in response to the learner’s actions. Adversarial environments include those with third party emitters which enter and exit the environment according to some criteria which does not depend on the radar network. The network consists of several independent radar nodes, which attempt to attain the highest possible SINR in each of many time steps. We show that in such an environment, simple sub-band selection algorithms are unable to consistently attain high SINR. However, through the use of adversarial multi-player bandit algorithms, a radar network can continue to track targets without a loss in tracking precisio

A Bayesian Network for the Classification of Human Motion as Observed by Distributed Radar

In this article, a statistical model of human motion as observed by a network of radar sensors is presented where knowledge on the position and heading of the target provides information on the observation conditions of each sensor node. Sequences of motions are estimated from measurements of instantaneous Doppler frequency, which captures informative micromotions exhibited by the human target. A closed-form Bayesian estimation algorithm is presented that jointly estimates the state of the target and its exhibited motion class which are described by a hidden Markov model. To correct errors in the estimated motion class distribution introduced by faulty modeling assumptions, calibration of the probability distribution and measurement likelihood is performed by isotonic regression. It is shown, by modeling sensor observation conditions and by isotonic calibration of the measurement likelihood that a cognitive resource management system is able to increase classification accuracy by 5%-10% while utilizing sensor resources in accordance with defined mission objectives.Green Open Access added to TU Delft Institutional Repository ‘You share, we take care!’ – Taverne project https://www.openaccess.nl/en/you-share-we-take-care Otherwise as indicated in the copyright section: the publisher is the copyright holder of this work and the author uses the Dutch legislation to make this work public.Microwave Sensing, Signals & System