Multitarget tracking via restless bandit marginal productivity indices and Kalman Filter in discrete time

This paper designs, evaluates, and tests a tractable priority-index policy for scheduling target updates in a discrete-time multitarget tracking model, which aims to be close to optimal relative to a discounted or average performance objective accounting for tracking-error variance and measurement costs. The policy is to be used by a sensor system composed of M phased-array radars coordinated to track the positions of N targets moving according to independent scalar Gauss-Markov linear dynamics, which therefore allows for the use of the Kalman Filter for track estimation. The paper exploits the natural problem formulation as a multiarmed restless bandit problem (MARBP) with real-state projects subject to deterministic dynamics by deploying Whittle's (1988) index policy for the MARBP. The challenging issues of indexability (existence of the index) and index evaluation are resolved by applying a method recently introduced by the first author for the analysis of real-state restless bandits. Computational results are reported demonstrating the tractability of index evaluation, the substantial performance gains that the Whittle's marginal productivity (MP) index policy achieves against myopic policies advocated in previous work and the resulting index policies suboptimality gaps. Further, a preliminary small scale computational study shows that the (MP) index policy exhibits a nearly optimal behavior as the number of distinct objective targets grows with the number of radars per target constant.Multitarget tracking, Sensor management, Phased array radar, Radar scheduling, Scaled track-error variance (STEV), Kalman Filter, Index policy, Marginal productivity (MP) index, Real-state multiarmed restless bandit problems (MARBP)

Sensor scheduling for hunting elusive hiding targets: a restless bandit index policy

We consider a sensor scheduling model where a set of identical sensors are used to hunt a larger set of heterogeneous targets, each of which is located at a corresponding site. Target states change randomly over discrete time slots between “exposed” and ‘hidden,” according to Markovian transition probabilities that depend on whether sites are searched or not, so as to make the targets elusive. Sensors are imperfect, failing to detect an exposed target when searching its site with a positive misdetection probability. We formulate as a partially observable Markov decision process the problem of scheduling the sensors to search the sites so as to maximize the expected total discounted value of rewards earned (when targets are hunted) minus search costs incurred. Given the intractability of finding an optimal policy, we introduce a tractable heuristic search policy of priority index type based on the Whittle’s index for restless bandits. Preliminary computational results are reported showing that such a policy is nearly optimal and can substantially outperform the myopic policy and other simple heuristics.This work has been supported in part by the Spanish Ministry of Education and Science project MTM2007- 63140 and by the Ministry of Science and Innovation project MTM2010-2080

Multitarget tracking via restless bandit marginal productivity indices and Kalman Filter in discrete time

This paper designs, evaluates, and tests a tractable priority-index policy for scheduling target updates in a discrete-time multitarget tracking model, which aims to be close to optimal relative to a discounted or average performance objective accounting for tracking-error variance and measurement costs. The policy is to be used by a sensor system composed of M phased-array radars coordinated to track the positions of N targets moving according to independent scalar Gauss-Markov linear dynamics, which therefore allows for the use of the Kalman Filter for track estimation. The paper exploits the natural problem formulation as a multiarmed restless bandit problem (MARBP) with real-state projects subject to deterministic dynamics by deploying Whittle's (1988) index policy for the MARBP. The challenging issues of indexability (existence of the index) and index evaluation are resolved by applying a method recently introduced by the first author for the analysis of real-state restless bandits. Computational results are reported demonstrating the tractability of index evaluation, the substantial performance gains that the Whittle's marginal productivity (MP) index policy achieves against myopic policies advocated in previous work and the resulting index policies suboptimality gaps. Further, a preliminary small scale computational study shows that the (MP) index policy exhibits a nearly optimal behavior as the number of distinct objective targets grows with the number of radars per target constant

Response-adaptive designs for binary responses: How to offer patient benefit while being robust to time trends?

Response-adaptive randomisation (RAR) can considerably improve the chances of a successful treatment outcome for patients in a clinical trial by skewing the allocation probability towards better performing treatments as data accumulates. There is considerable interest in using RAR designs in drug development for rare diseases, where traditional designs are not either feasible or ethically questionable. In this paper, we discuss and address a major criticism levelled at RAR: namely, type I error inflation due to an unknown time trend over the course of the trial. The most common cause of this phenomenon is changes in the characteristics of recruited patients-referred to as patient drift. This is a realistic concern for clinical trials in rare diseases due to their lengthly accrual rate. We compute the type I error inflation as a function of the time trend magnitude to determine in which contexts the problem is most exacerbated. We then assess the ability of different correction methods to preserve type I error in these contexts and their performance in terms of other operating characteristics, including patient benefit and power. We make recommendations as to which correction methods are most suitable in the rare disease context for several RAR rules, differentiating between the 2-armed and the multi-armed case. We further propose a RAR design for multi-armed clinical trials, which is computationally efficient and robust to several time trends considered

Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges.

Multi-armed bandit problems (MABPs) are a special type of optimal control problem well suited to model resource allocation under uncertainty in a wide variety of contexts. Since the first publication of the optimal solution of the classic MABP by a dynamic index rule, the bandit literature quickly diversified and emerged as an active research topic. Across this literature, the use of bandit models to optimally design clinical trials became a typical motivating application, yet little of the resulting theory has ever been used in the actual design and analysis of clinical trials. To this end, we review two MABP decision-theoretic approaches to the optimal allocation of treatments in a clinical trial: the infinite-horizon Bayesian Bernoulli MABP and the finite-horizon variant. These models possess distinct theoretical properties and lead to separate allocation rules in a clinical trial design context. We evaluate their performance compared to other allocation rules, including fixed randomization. Our results indicate that bandit approaches offer significant advantages, in terms of assigning more patients to better treatments, and severe limitations, in terms of their resulting statistical power. We propose a novel bandit-based patient allocation rule that overcomes the issue of low power, thus removing a potential barrier for their use in practice

A Bayesian adaptive design for clinical trials in rare diseases

Development of treatments for rare diseases is challenging due to the limited number of patients available for participation. Learning about treatment effectiveness with a view to treat patients in the larger outside population, as in the traditional fixed randomised design, may not be a plausible goal. An alternative goal is to treat the patients within the trial as effectively as possible. Using the framework of finite-horizon Markov decision processes and dynamic programming (DP), a novel randomised response-adaptive design is proposed which maximises the total number of patient successes in the trial and penalises if a minimum number of patients are not recruited to each treatment arm. Several performance measures of the proposed design are evaluated and compared to alternative designs through extensive simulation studies using a recently published trial as motivation. For simplicity, a two-armed trial with binary endpoints and immediate responses is considered. Simulation results for the proposed design show that: (i) the percentage of patients allocated to the superior arm is much higher than in the traditional fixed randomised design; (ii) relative to the optimal DP design, the power is largely improved upon and (iii) it exhibits only a very small bias and mean squared error of the treatment effect estimator. Furthermore, this design is fully randomised which is an advantage from a practical point of view because it protects the trial against various sources of bias. As such, the proposed design addresses some of the key issues that have been suggested as preventing so-called bandit models from being implemented in clinical practice.We gratefully acknowledge the support of the EPSRC funded STOR-i Centre for Doctoral Training, Grant No. EP/H023151/1. Sofía S. Villar is grateful to the Biometrika Trust for its research funding. This report is independent research arising in part from Professor Jaki’s Senior Research Fellowship (NIHR-SRF-2015-08-001) supported by the National Institute for Health Research

Response-adaptive randomization for multi-arm clinical trials using the forward looking Gittins index rule.

The Gittins index provides a well established, computationally attractive, optimal solution to a class of resource allocation problems known collectively as the multi-arm bandit problem. Its development was originally motivated by the problem of optimal patient allocation in multi-arm clinical trials. However, it has never been used in practice, possibly for the following reasons: (1) it is fully sequential, i.e., the endpoint must be observable soon after treating a patient, reducing the medical settings to which it is applicable; (2) it is completely deterministic and thus removes randomization from the trial, which would naturally protect against various sources of bias. We propose a novel implementation of the Gittins index rule that overcomes these difficulties, trading off a small deviation from optimality for a fully randomized, adaptive group allocation procedure which offers substantial improvements in terms of patient benefit, especially relevant for small populations. We report the operating characteristics of our approach compared to existing methods of adaptive randomization using a recently published trial as motivation

A novel statistical test for treatment differences in clinical trials using a response-adaptive forward-looking Gittins Index Rule.

The most common objective for response-adaptive clinical trials is to seek to ensure that patients within a trial have a high chance of receiving the best treatment available by altering the chance of allocation on the basis of accumulating data. Approaches that yield good patient benefit properties suffer from low power from a frequentist perspective when testing for a treatment difference at the end of the study due to the high imbalance in treatment allocations. In this work we develop an alternative pairwise test for treatment difference on the basis of allocation probabilities of the covariate-adjusted response-adaptive randomization with forward-looking Gittins Index (CARA-FLGI) Rule for binary responses. The performance of the novel test is evaluated in simulations for two-armed studies and then its applications to multiarmed studies are illustrated. The proposed test has markedly improved power over the traditional Fisher exact test when this class of nonmyopic response adaptation is used. We also find that the test's power is close to the power of a Fisher exact test under equal randomization

Medicines and Healthcare products Regulatory Agency’s “Consultation on proposals for legislative changes for clinical trials”: a response from the Trials Methodology Research Partnership Adaptive Designs Working Group, with a focus on data sharing

In the UK, the Medicines and Healthcare products Regulatory Agency consulted on proposals “to improve and strengthen the UK clinical trials legislation to help us make the UK the best place to research and develop safe and innovative medicines”. The purpose of the consultation was to help finalise the proposals and contribute to the drafting of secondary legislation. We discussed these proposals as members of the Trials Methodology Research Partnership Adaptive Designs Working Group, which is jointly funded by the Medical Research Council and the National Institute for Health and Care Research. Two topics arose frequently in the discussion: the emphasis on legislation, and the absence of questions on data sharing. It is our opinion that the proposals rely heavily on legislation to change practice. However, clinical trials are heterogeneous, and as a result some trials will struggle to comply with all of the proposed legislation. Furthermore, adaptive design clinical trials are even more heterogeneous than their non-adaptive counterparts, and face more challenges. Consequently, it is possible that increased legislation could have a greater negative impact on adaptive designs than non-adaptive designs. Overall, we are sceptical that the introduction of legislation will achieve the desired outcomes, with some exceptions. Meanwhile the topic of data sharing — making anonymised individual-level clinical trial data available to other investigators for further use — is entirely absent from the proposals and the consultation in general. However, as an aspect of the wider concept of open science and reproducible research, data sharing is an increasingly important aspect of clinical trials. The benefits of data sharing include faster innovation, improved surveillance of drug safety and effectiveness and decreasing participant exposure to unnecessary risk. There are already a number of UK-focused documents that discuss and encourage data sharing, for example, the Concordat on Open Research Data and the Medical Research Council’s Data Sharing Policy. We strongly suggest that data sharing should be the norm rather than the exception, and hope that the forthcoming proposals on clinical trials invite discussion on this important topic

Point OutWords: protocol for a feasibility randomised controlled trial of a motor skills intervention to promote communicative development in non-verbal children with autism

Abstract: Background: Point OutWords is a caregiver-delivered, iPad-assisted intervention for non-verbal or minimally verbal children with autism. It aims to develop prerequisite skills for communication such as manual and oral motor skills, sequencing, and symbolic representation. This feasibility trial aims to determine the viability of evaluating the clinical efficacy of Point OutWords. Methodology: We aim to recruit 46 non-verbal or minimally verbal children with autism and their families, approximately 23 per arm. Children in the intervention group will use Point OutWords for half an hour, five times a week, for 8 weeks. Children in the control group will have equal caregiver-led contact time with the iPad using a selection of control apps (e.g. sensory apps, drawing apps). Communication, motor, and daily living skills are assessed at baseline and post-intervention. Parents will keep diaries during the intervention period and will take part in focus groups when the intervention is completed. Discussion: Point OutWords was developed in collaboration with children with autism and their caregivers, to provide an intervention for a subgroup of autism that has been historically underserved. As autism is a heterogeneous condition, it is unlikely that one style of intervention will address all aspects of its symptomatology; the motor skills approach of Point OutWords can complement other therapies that address core autistic symptoms of social cognition and communication more directly. The current feasibility trial can inform the selection of outcome measures and design for future full-scale randomised controlled trials of Point OutWords and of other early interventions in autism. Trial registration: ISRCTN, ISRCTN12808402. Prospectively registered on 12 March 2019