81 research outputs found
Asymptotically Optimal Anomaly Detection via Sequential Testing
Sequential detection of independent anomalous processes among K processes is
considered. At each time, only M processes can be observed, and the
observations from each chosen process follow two different distributions,
depending on whether the process is normal or abnormal. Each anomalous process
incurs a cost per unit time until its anomaly is identified and fixed.
Switching across processes and state declarations are allowed at all times,
while decisions are based on all past observations and actions. The objective
is a sequential search strategy that minimizes the total expected cost incurred
by all the processes during the detection process under reliability
constraints. Low-complexity algorithms are established to achieve
asymptotically optimal performance as the error constraints approach zero.
Simulation results demonstrate strong performance in the finite regime.Comment: 28 pages, 5 figures, part of this work will be presented at the 52nd
Annual Allerton Conference on Communication, Control, and Computing, 201
Quickest anomaly detection: A case of active hypothesis testing
Abstract — The problem of quickest detection of an anomalous process among M processes is considered. At each time, a subset of the processes can be observed, and the observations follow two different distributions, depending on whether the process is normal or abnormal. The objective is a sequential search strategy that minimizes the expected detection time subject to an error probability constraint. This problem can be considered as a special case of active hypothesis testing first considered by Chernoff in 1959, where a randomized test was proposed and shown to be asymptotically optimal. For the special case considered in this paper, we show that a simple deterministic test achieves asymptotic optimality and offers better performance in the finite regime. Index Terms—Sequential detection, hypothesis testing, dy-namic search. I
Active Anomaly Detection in Heterogeneous Processes
An active inference problem of detecting anomalies among heterogeneous
processes is considered. At each time, a subset of processes can be probed. The
objective is to design a sequential probing strategy that dynamically
determines which processes to observe at each time and when to terminate the
search so that the expected detection time is minimized under a constraint on
the probability of misclassifying any process. This problem falls into the
general setting of sequential design of experiments pioneered by Chernoff in
1959, in which a randomized strategy, referred to as the Chernoff test, was
proposed and shown to be asymptotically optimal as the error probability
approaches zero. For the problem considered in this paper, a low-complexity
deterministic test is shown to enjoy the same asymptotic optimality while
offering significantly better performance in the finite regime and faster
convergence to the optimal rate function, especially when the number of
processes is large. The computational complexity of the proposed test is also
of a significantly lower order.Comment: This work has been accepted for publication on IEEE Transactions on
Information Theor
Dynamic Intrusion Detection in Resource-Constrained Cyber Networks
We consider a large-scale cyber network with N components (e.g., paths,
servers, subnets). Each component is either in a healthy state (0) or an
abnormal state (1). Due to random intrusions, the state of each component
transits from 0 to 1 over time according to certain stochastic process. At each
time, a subset of K (K < N) components are checked and those observed in
abnormal states are fixed. The objective is to design the optimal scheduling
for intrusion detection such that the long-term network cost incurred by all
abnormal components is minimized. We formulate the problem as a special class
of Restless Multi-Armed Bandit (RMAB) process. A general RMAB suffers from the
curse of dimensionality (PSPACE-hard) and numerical methods are often
inapplicable. We show that, for this class of RMAB, Whittle index exists and
can be obtained in closed form, leading to a low-complexity implementation of
Whittle index policy with a strong performance. For homogeneous components,
Whittle index policy is shown to have a simple structure that does not require
any prior knowledge on the intrusion processes. Based on this structure,
Whittle index policy is further shown to be optimal over a finite time horizon
with an arbitrary length. Beyond intrusion detection, these results also find
applications in queuing networks with finite-size buffers.Comment: 9 pages, 5 figure
- …