2,335 research outputs found
An Improved Composite Hypothesis Test for Markov Models with Applications in Network Anomaly Detection
Recent work has proposed the use of a composite hypothesis Hoeffding test for
statistical anomaly detection. Setting an appropriate threshold for the test
given a desired false alarm probability involves approximating the false alarm
probability. To that end, a large deviations asymptotic is typically used
which, however, often results in an inaccurate setting of the threshold,
especially for relatively small sample sizes. This, in turn, results in an
anomaly detection test that does not control well for false alarms. In this
paper, we develop a tighter approximation using the Central Limit Theorem (CLT)
under Markovian assumptions. We apply our result to a network anomaly detection
application and demonstrate its advantages over earlier work.Comment: 6 pages, 6 figures; final version for CDC 201
Robust measurement-based buffer overflow probability estimators for QoS provisioning and traffic anomaly prediction applications
Suitable estimators for a class of Large Deviation approximations of rare event probabilities based on sample realizations of random processes have been proposed in our earlier work. These estimators are expressed as non-linear multi-dimensional optimization problems of a special structure. In this paper, we develop an algorithm to solve these optimization problems very efficiently based on their characteristic structure. After discussing the nature of the objective function and constraint set and their peculiarities, we provide a formal proof that the developed algorithm is guaranteed to always converge. The existence of efficient and provably convergent algorithms for solving these problems is a prerequisite for using the proposed estimators in real time problems such as call admission control, adaptive modulation and coding with QoS constraints, and traffic anomaly detection in high data rate communication networks
Robust measurement-based buffer overflow probability estimators for QoS provisioning and traffic anomaly prediction applicationm
Suitable estimators for a class of Large Deviation approximations of rare
event probabilities based on sample realizations of random processes have been
proposed in our earlier work. These estimators are expressed as non-linear
multi-dimensional optimization problems of a special structure. In this paper,
we develop an algorithm to solve these optimization problems very efficiently
based on their characteristic structure. After discussing the nature of the
objective function and constraint set and their peculiarities, we provide a
formal proof that the developed algorithm is guaranteed to always converge. The
existence of efficient and provably convergent algorithms for solving these
problems is a prerequisite for using the proposed estimators in real time
problems such as call admission control, adaptive modulation and coding with
QoS constraints, and traffic anomaly detection in high data rate communication
networks
HYPA: Efficient Detection of Path Anomalies in Time Series Data on Networks
The unsupervised detection of anomalies in time series data has important
applications in user behavioral modeling, fraud detection, and cybersecurity.
Anomaly detection has, in fact, been extensively studied in categorical
sequences. However, we often have access to time series data that represent
paths through networks. Examples include transaction sequences in financial
networks, click streams of users in networks of cross-referenced documents, or
travel itineraries in transportation networks. To reliably detect anomalies, we
must account for the fact that such data contain a large number of independent
observations of paths constrained by a graph topology. Moreover, the
heterogeneity of real systems rules out frequency-based anomaly detection
techniques, which do not account for highly skewed edge and degree statistics.
To address this problem, we introduce HYPA, a novel framework for the
unsupervised detection of anomalies in large corpora of variable-length
temporal paths in a graph. HYPA provides an efficient analytical method to
detect paths with anomalous frequencies that result from nodes being traversed
in unexpected chronological order.Comment: 11 pages with 8 figures and supplementary material. To appear at SIAM
Data Mining (SDM 2020
Robust Anomaly Detection in Dynamic Networks
We propose two robust methods for anomaly detection in dynamic networks in
which the properties of normal traffic are time-varying. We formulate the
robust anomaly detection problem as a binary composite hypothesis testing
problem and propose two methods: a model-free and a model-based one, leveraging
techniques from the theory of large deviations. Both methods require a family
of Probability Laws (PLs) that represent normal properties of traffic. We
devise a two-step procedure to estimate this family of PLs. We compare the
performance of our robust methods and their vanilla counterparts, which assume
that normal traffic is stationary, on a network with a diurnal normal pattern
and a common anomaly related to data exfiltration. Simulation results show that
our robust methods perform better than their vanilla counterparts in dynamic
networks.Comment: 6 pages. MED conferenc
- …