Optimal Sequential Multi-class Diagnosis

Sequential multi-class diagnosis, also known as multi-hypothesis testing, is a classical sequential decision problem with broad applications. However, the optimal solution remains, in general, unknown as the dynamic program suffers from the curse of dimensionality in the posterior belief space. We consider a class of practical problems in which the observation distributions associated with different classes are related through exponential tilting, and show that the reachable beliefs could be restricted on, or near, a set of low-dimensional, time-dependent manifolds with closed-form expressions. This sparsity is driven by the low dimensionality of the observation distributions (which is intuitive) as well as by specific structural interrelations among them (which is less intuitive). We use a matrix factorization approach to uncover the potential low dimensionality hidden in high-dimensional beliefs and reconstruct the beliefs using a diagnostic statistic in lower dimension. For common univariate distributions, e.g., normal, binomial, and Poisson, the belief reconstruction is exact, and the optimal policies can be efficiently computed for a large number of classes. We also characterize the structure of the optimal policy in the reduced dimension. For multivariate distributions, we propose a low-rank matrix approximation scheme that works well when the beliefs are near the low-dimensional manifolds. The optimal policy significantly outperforms the state-of-the-art heuristic policy in quick diagnosis with noisy data. (forthcoming in Operations Research)Comment: 68 pages, 22 figures, 10 table

An Asymptotically Optimal Policy for Uniform Bandits of Unknown Support

Consider the problem of a controller sampling sequentially from a finite number of $N \geq 2$ populations, specified by random variables $X^i_k$, $i = 1,\ldots , N,$ and $k = 1, 2, \ldots$; where $X^i_k$ denotes the outcome from population $i$ the $k^{th}$ time it is sampled. It is assumed that for each fixed $i$, $\{ X^i_k \}_{k \geq 1}$ is a sequence of i.i.d. uniform random variables over some interval $[a_i, b_i]$, with the support (i.e., $a_i, b_i$) unknown to the controller. The objective is to have a policy $\pi$ for deciding, based on available data, from which of the $N$ populations to sample from at any time $n=1,2,\ldots$ so as to maximize the expected sum of outcomes of $n$ samples or equivalently to minimize the regret due to lack on information of the parameters $\{ a_i \}$ and $\{ b_i \}$. In this paper, we present a simple inflated sample mean (ISM) type policy that is asymptotically optimal in the sense of its regret achieving the asymptotic lower bound of Burnetas and Katehakis (1996). Additionally, finite horizon regret bounds are given.Comment: arXiv admin note: text overlap with arXiv:1504.0582

Normal Bandits of Unknown Means and Variances: Asymptotic Optimality, Finite Horizon Regret Bounds, and a Solution to an Open Problem

Consider the problem of sampling sequentially from a finite number of $N \geq 2$ populations, specified by random variables $X^i_k$, $i = 1,\ldots , N,$ and $k = 1, 2, \ldots$; where $X^i_k$ denotes the outcome from population $i$ the $k^{th}$ time it is sampled. It is assumed that for each fixed $i$, $\{ X^i_k \}_{k \geq 1}$ is a sequence of i.i.d. normal random variables, with unknown mean $\mu_i$ and unknown variance $\sigma_i^2$. The objective is to have a policy $\pi$ for deciding from which of the $N$ populations to sample form at any time $n=1,2,\ldots$ so as to maximize the expected sum of outcomes of $n$ samples or equivalently to minimize the regret due to lack on information of the parameters $\mu_i$ and $\sigma_i^2$. In this paper, we present a simple inflated sample mean (ISM) index policy that is asymptotically optimal in the sense of Theorem 4 below. This resolves a standing open problem from Burnetas and Katehakis (1996). Additionally, finite horizon regret bounds are given.Comment: 15 pages 3 figure

Optimal Index Policies for Anomaly Localization in Resource-Constrained Cyber Systems

The problem of anomaly localization in a resource-constrained cyber system is considered. Each anomalous component of the system incurs a cost per unit time until its anomaly is identified and fixed. Different anomalous components may incur different costs depending on their criticality to the system. Due to resource constraints, only one component can be probed at each given time. The observations from a probed component are realizations drawn from two different distributions depending on whether the component is normal or anomalous. The objective is a probing strategy that minimizes the total expected cost, incurred by all the components during the detection process, under reliability constraints. We consider both independent and exclusive models. In the former, each component can be abnormal with a certain probability independent of other components. In the latter, one and only one component is abnormal. We develop optimal simple index policies under both models. The proposed index policies apply to a more general case where a subset (more than one) of the components can be probed simultaneously and have strong performance as demonstrated by simulation examples. The problem under study also finds applications in spectrum scanning in cognitive radio networks and event detection in sensor networks.Comment: 30 pages, 4 figures, accepted for publication in the IEEE Transactions on Signal Processing, part of this work was presented at the IEEE GlobalSIP 201

Data-Efficient Minimax Quickest Change Detection in a Decentralized System

A sensor network is considered where a sequence of random variables is observed at each sensor. At each time step, a processed version of the observations is transmitted from the sensors to a common node called the fusion center. At some unknown point in time the distribution of the observations at all the sensor nodes changes. The objective is to detect this change in distribution as quickly as possible, subject to constraints on the false alarm rate and the cost of observations taken at each sensor. Minimax problem formulations are proposed for the above problem. A data-efficient algorithm is proposed in which an adaptive sampling strategy is used at each sensor to control the cost of observations used before change. To conserve the cost of communication an occasional binary digit is transmitted from each sensor to the fusion center. It is shown that the proposed algorithm is globally asymptotically optimal for the proposed formulations, as the false alarm rate goes to zero.Comment: Submitted to Sequential Analysis, Aug. 201

Asymptotically Optimal Sequential Design for Rank Aggregation

A sequential design problem for rank aggregation is commonly encountered in psychology, politics, marketing, sports, etc. In this problem, a decision maker is responsible for ranking $K$ items by sequentially collecting pairwise noisy comparison from judges. The decision maker needs to choose a pair of items for comparison in each step, decide when to stop data collection, and make a final decision after stopping, based on a sequential flow of information. Due to the complex ranking structure, existing sequential analysis methods are not suitable. In this paper, we formulate the problem under a Bayesian decision framework and propose sequential procedures that are asymptotically optimal. These procedures achieve asymptotic optimality by seeking for a balance between exploration (i.e. finding the most indistinguishable pair of items) and exploitation (i.e. comparing the most indistinguishable pair based on the current information). New analytical tools are developed for proving the asymptotic results, combining advanced change of measure techniques for handling the level crossing of likelihood ratios and classic large deviation results for martingales, which are of separate theoretical interest in solving complex sequential design problems. A mirror-descent algorithm is developed for the computation of the proposed sequential procedures

Minimum Risk Point Estimation of Gini Index

This paper develops a theory and methodology for estimation of Gini index such that both cost of sampling and estimation error are minimum. Methods in which sample size is fixed in advance, cannot minimize estimation error and sampling cost at the same time. In this article, a purely sequential procedure is proposed which provides an estimate of the sample size required to achieve a sufficiently smaller estimation error and lower sampling cost. Characteristics of the purely sequential procedure are examined and asymptotic optimality properties are proved without assuming any specific distribution of the data. Performance of our method is examined through extensive simulation study

Finite-time Analysis for the Knowledge-Gradient Policy

We consider sequential decision problems in which we adaptively choose one of finitely many alternatives and observe a stochastic reward. We offer a new perspective of interpreting Bayesian ranking and selection problems as adaptive stochastic multi-set maximization problems and derive the first finite-time bound of the knowledge-gradient policy for adaptive submodular objective functions. In addition, we introduce the concept of prior-optimality and provide another insight into the performance of the knowledge gradient policy based on the submodular assumption on the value of information. We demonstrate submodularity for the two-alternative case and provide other conditions for more general problems, bringing out the issue and importance of submodularity in learning problems. Empirical experiments are conducted to further illustrate the finite time behavior of the knowledge gradient policy

Multistage Adaptive Estimation of Sparse Signals

This paper considers sequential adaptive estimation of sparse signals under a constraint on the total sensing effort. The advantage of adaptivity in this context is the ability to focus more resources on regions of space where signal components exist, thereby improving performance. A dynamic programming formulation is derived for the allocation of sensing effort to minimize the expected estimation loss. Based on the method of open-loop feedback control, allocation policies are then developed for a variety of loss functions. The policies are optimal in the two-stage case, generalizing an optimal two-stage policy proposed by Bashan et al., and improve monotonically thereafter with the number of stages. Numerical simulations show gains up to several dB as compared to recently proposed adaptive methods, and dramatic gains compared to non-adaptive estimation. An application to radar imaging is also presented.Comment: 15 pages, 8 figures, minor revision

Active Hypothesis Testing for Quickest Anomaly Detection

The problem of quickest detection of an anomalous process among M processes is considered. At each time, a subset of the processes can be observed, and the observations from each chosen process follow two different distributions, depending on whether the process is normal or abnormal. The objective is a sequential search strategy that minimizes the expected detection time subject to an error probability constraint. This problem can be considered as a special case of active hypothesis testing first considered by Chernoff in 1959 where a randomized strategy, referred to as the Chernoff test, was proposed and shown to be asymptotically (as the error probability approaches zero) optimal. For the special case considered in this paper, we show that a simple deterministic test achieves asymptotic optimality and offers better performance in the finite regime. We further extend the problem to the case where multiple anomalous processes are present. In particular, we examine the case where only an upper bound on the number of anomalous processes is known.Comment: 46 pages, 3 figures, part of this work was presented at the Information Theory and Applications (ITA) Workshop, Feb. 201