34,119 research outputs found
Mid-level feature set for specific event and anomaly detection in crowded scenes
Proceedings of: 20th IEEE International Conference on Image Processing (ICIP 2013). Melbourne, Australia, September 15-18, 2013.In this paper we propose a system for automatic detection of specific events and abnormal behaviors in crowded scenes. In particular, we focus on the parametrization by proposing a set of mid-level spatio-temporal features that successfully model the characteristic motion of typical events in crowd behaviors. Furthermore, due to the fact that some features are more suitable than others to model specific events of interest, we also present an automatic process for feature selection. Our experiments prove that the suggested feature set works successfully for both explicit event detection and distance-based anomaly detection tasks. The results on PETS for explicit event detection are generally better than those previously reported. Regarding anomaly detection, the proposed method performance is comparable to those of state-of-the-art method for PETS and substantially better than that reported for Web dataset.Publicad
Payload-based anomaly detection in HTTP traffic
University of Technology, Sydney. Faculty of Engineering and Information Technology.Internet provides quality and convenience to human life but at the same time it provides a platform for network hackers and criminals. Intrusion Detection Systems (IDSs) have been proven to be powerful methods for detecting anomalies in the network. Traditional IDSs based on signatures are unable to detect new (zero days) attacks. Anomaly-based systems are alternative to signature based systems. However, present anomaly detection systems suffer from three major setbacks:
(a) Large number of false alarms,
(b) Very high volume of network traffic due to high data rates (Gbps), and
(c) Inefficiency in operation.
In this thesis, we address above issues and develop efficient intrusion detection frameworks and models which can be used in detecting a wide variety of attacks including web-based attacks. Our proposed methods are designed to have very few false alarms. We also address Intrusion Detection as a Pattern Recognition problem and discuss all aspects that are important in realizing an anomaly-based IDS.
We present three payload-based anomaly detectors, including Geometrical Structure Anomaly Detection (GSAD), Two-Tier Intrusion Detection system using Linear Discriminant Analysis (LDA), and Real-time Payload-based Intrusion Detection System (RePIDS), for intrusion detection. These detectors perform deep-packet analysis and examine payload content using n-gram text categorization and Mahalanobis Distance Map (MDM) techniques. An MDM extracts hidden correlations between the features within each payload and among packet payloads. GSAD generates model of normal network payload as geometrical structure using MDMs in a fully automatic and unsupervised manner. We have implemented the GSAD model in HTTP environment for web-based applications.
For efficient operation of IDSs, the detection speed is a key point. Current IDSs examine a large number of data features to detect intrusions and misuse patterns. Hence, for quickly and accurately identifying anomalies of Internet traffic, feature reduction becomes mandatory. We have proposed two models to address this issue, namely two-tier intrusion detection model and RePIDS.
Two-tier intrusion detection model uses Linear Discriminant Analysis approach for feature reduction and optimal feature selection. It uses MDM technique to create a model of normal network payload using an extracted feature set.
RePIDS uses a 3-tier Iterative Feature Selection Engine (IFSEng) to reduce dimensionality of the raw dataset using Principal Component Analysis (PCA) technique. IFSEng extracts the most significant features from the original feature set and uses mathematical and graphical methods for optimal feature subset selection. Like two-tier intrusion detection model, RePIDS then uses MDM technique to generate a model of normal network payload using extracted features.
We test the proposed IDSs on two publicly available datasets of attacks and normal traffic. Experimental results confirm the effectiveness and validation of our proposed solutions in terms of detection rate, false alarm rate and computational complexity
Refining the Optimization Target for Automatic Univariate Time Series Anomaly Detection in Monitoring Services
Time series anomaly detection is crucial for industrial monitoring services
that handle a large volume of data, aiming to ensure reliability and optimize
system performance. Existing methods often require extensive labeled resources
and manual parameter selection, highlighting the need for automation. This
paper proposes a comprehensive framework for automatic parameter optimization
in time series anomaly detection models. The framework introduces three
optimization targets: prediction score, shape score, and sensitivity score,
which can be easily adapted to different model backbones without prior
knowledge or manual labeling efforts. The proposed framework has been
successfully applied online for over six months, serving more than 50,000 time
series every minute. It simplifies the user's experience by requiring only an
expected sensitive value, offering a user-friendly interface, and achieving
desired detection results. Extensive evaluations conducted on public datasets
and comparison with other methods further confirm the effectiveness of the
proposed framework.Comment: Accepted by 2023 IJCAI Worksho
The model of an anomaly detector for HiLumi LHC magnets based on Recurrent Neural Networks and adaptive quantization
This paper focuses on an examination of an applicability of Recurrent Neural
Network models for detecting anomalous behavior of the CERN superconducting
magnets. In order to conduct the experiments, the authors designed and
implemented an adaptive signal quantization algorithm and a custom GRU-based
detector and developed a method for the detector parameters selection. Three
different datasets were used for testing the detector. Two artificially
generated datasets were used to assess the raw performance of the system
whereas the 231 MB dataset composed of the signals acquired from HiLumi magnets
was intended for real-life experiments and model training. Several different
setups of the developed anomaly detection system were evaluated and compared
with state-of-the-art OC-SVM reference model operating on the same data. The
OC-SVM model was equipped with a rich set of feature extractors accounting for
a range of the input signal properties. It was determined in the course of the
experiments that the detector, along with its supporting design methodology,
reaches F1 equal or very close to 1 for almost all test sets. Due to the
profile of the data, the best_length setup of the detector turned out to
perform the best among all five tested configuration schemes of the detection
system. The quantization parameters have the biggest impact on the overall
performance of the detector with the best values of input/output grid equal to
16 and 8, respectively. The proposed solution of the detection significantly
outperformed OC-SVM-based detector in most of the cases, with much more stable
performance across all the datasets.Comment: Related to arXiv:1702.0083
A Methodology for the Diagnostic of Aircraft Engine Based on Indicators Aggregation
Aircraft engine manufacturers collect large amount of engine related data
during flights. These data are used to detect anomalies in the engines in order
to help companies optimize their maintenance costs. This article introduces and
studies a generic methodology that allows one to build automatic early signs of
anomaly detection in a way that is understandable by human operators who make
the final maintenance decision. The main idea of the method is to generate a
very large number of binary indicators based on parametric anomaly scores
designed by experts, complemented by simple aggregations of those scores. The
best indicators are selected via a classical forward scheme, leading to a much
reduced number of indicators that are tuned to a data set. We illustrate the
interest of the method on simulated data which contain realistic early signs of
anomalies.Comment: Proceedings of the 14th Industrial Conference, ICDM 2014, St.
Petersburg : Russian Federation (2014
Interpretable Aircraft Engine Diagnostic via Expert Indicator Aggregation
Detecting early signs of failures (anomalies) in complex systems is one of
the main goal of preventive maintenance. It allows in particular to avoid
actual failures by (re)scheduling maintenance operations in a way that
optimizes maintenance costs. Aircraft engine health monitoring is one
representative example of a field in which anomaly detection is crucial.
Manufacturers collect large amount of engine related data during flights which
are used, among other applications, to detect anomalies. This article
introduces and studies a generic methodology that allows one to build automatic
early signs of anomaly detection in a way that builds upon human expertise and
that remains understandable by human operators who make the final maintenance
decision. The main idea of the method is to generate a very large number of
binary indicators based on parametric anomaly scores designed by experts,
complemented by simple aggregations of those scores. A feature selection method
is used to keep only the most discriminant indicators which are used as inputs
of a Naive Bayes classifier. This give an interpretable classifier based on
interpretable anomaly detectors whose parameters have been optimized indirectly
by the selection process. The proposed methodology is evaluated on simulated
data designed to reproduce some of the anomaly types observed in real world
engines.Comment: arXiv admin note: substantial text overlap with arXiv:1408.6214,
arXiv:1409.4747, arXiv:1407.088
Anomaly Detection Based on Indicators Aggregation
Automatic anomaly detection is a major issue in various areas. Beyond mere
detection, the identification of the source of the problem that produced the
anomaly is also essential. This is particularly the case in aircraft engine
health monitoring where detecting early signs of failure (anomalies) and
helping the engine owner to implement efficiently the adapted maintenance
operations (fixing the source of the anomaly) are of crucial importance to
reduce the costs attached to unscheduled maintenance. This paper introduces a
general methodology that aims at classifying monitoring signals into normal
ones and several classes of abnormal ones. The main idea is to leverage expert
knowledge by generating a very large number of binary indicators. Each
indicator corresponds to a fully parametrized anomaly detector built from
parametric anomaly scores designed by experts. A feature selection method is
used to keep only the most discriminant indicators which are used at inputs of
a Naive Bayes classifier. This give an interpretable classifier based on
interpretable anomaly detectors whose parameters have been optimized indirectly
by the selection process. The proposed methodology is evaluated on simulated
data designed to reproduce some of the anomaly types observed in real world
engines.Comment: International Joint Conference on Neural Networks (IJCNN 2014),
Beijing : China (2014). arXiv admin note: substantial text overlap with
arXiv:1407.088
Anomaly Detection Based on Aggregation of Indicators
Automatic anomaly detection is a major issue in various areas. Beyond mere
detection, the identification of the origin of the problem that produced the
anomaly is also essential. This paper introduces a general methodology that can
assist human operators who aim at classifying monitoring signals. The main idea
is to leverage expert knowledge by generating a very large number of
indicators. A feature selection method is used to keep only the most
discriminant indicators which are used as inputs of a Naive Bayes classifier.
The parameters of the classifier have been optimized indirectly by the
selection process. Simulated data designed to reproduce some of the anomaly
types observed in real world engines.Comment: 23rd annual Belgian-Dutch Conference on Machine Learning (Benelearn
2014), Bruxelles : Belgium (2014
ATLANTIDES: Automatic Configuration for Alert Verification in Network Intrusion Detection Systems
We present an architecture designed for alert verification (i.e., to reduce false positives) in network intrusion-detection systems. Our technique is based on a systematic (and automatic) anomaly-based analysis of the system output, which provides useful context information regarding the network services. The false positives raised by the NIDS analyzing the incoming traffic (which can be either signature- or anomaly-based) are reduced by correlating them with the output anomalies. We designed our architecture for TCP-based network services which have a client/server architecture (such as HTTP). Benchmarks show a substantial reduction of false positives between 50% and 100%
- …