1,196 research outputs found
A performance study of anomaly detection using entropy method
An experiment to study the entropy method for an anomaly detection system has
been performed. The study has been conducted using real data generated from the
distributed sensor networks at the Intel Berkeley Research Laboratory. The
experimental results were compared with the elliptical method and has been
analyzed in two dimensional data sets acquired from temperature and humidity
sensors across 52 micro controllers. Using the binary classification to
determine the upper and lower boundaries for each series of sensors, it has
been shown that the entropy method are able to detect more number of out
ranging sensor nodes than the elliptical methods. It can be argued that the
better result was mainly due to the lack of elliptical approach which is
requiring certain correlation between two sensor series, while in the entropy
approach each sensor series is treated independently. This is very important in
the current case where both sensor series are not correlated each other.Comment: Proceeding of the International Conference on Computer, Control,
Informatics and its Applications (2017) pp. 137-14
Network anomaly detection research: a survey
Data analysis to identifying attacks/anomalies is a crucial task in anomaly detection and network anomaly detection itself is an important issue in network security. Researchers have developed methods and algorithms for the improvement of the anomaly detection system. At the same time, survey papers on anomaly detection researches are available. Nevertheless, this paper attempts to analyze futher and to provide alternative taxonomy on anomaly detection researches focusing on methods, types of anomalies, data repositories, outlier identity and the most used data type. In addition, this paper summarizes information on application network categories of the existing studies
Enhancing structural health monitoring with vehicle identification and tracking
Traffic load monitoring and structural health monitoring (SHM) have been gaining increasing attention over the last decade. However, most of the current installations treat the two monitoring types as separated problems, thereby using dedicated installed sensors, such as smart cameras for traffic load or accelerometers for Structural Health Monitoring (SHM). This paper presents a new framework aimed at leveraging the data collected by a SHM system for a second use, namely, monitoring vehicles passing on the structure being monitored (a viaduct). Our framework first processes the raw three-axial acceleration signals through a series of transformations and extracts its energy. Then, an anomaly detection algorithm is used to detect peaks from 90 installed sensors, and a linear regression together with a simple threshold filters out false detection by estimating the speed of the vehicles. Initial results in conditions of moderate traffic load are promising, demonstrating the detection of vehicles and realistic characterization of their speed. Moreover, a k-means clustering analysis distinguishes two groups of peaks with statistically different features such as amplitude and damping duration that could be likely associated with heavy vehicles and cars, respectively
Comparison of new anomaly detection technique for wind turbine condition monitoring using gearbox SCADA data
Anomaly detection for wind turbine condition monitoring is an active area of research within the wind energy operations and maintenance (O&M) community. In this paper three models were compared for multi-megawatt operational wind turbine SCADA data. The models used for comparison were One-Class Support Vector Machine (OCSVM), Isolation Forest (IF), and Elliptical Envelope (EE). Each of these were compared for the same fault, and tested under various different data configurations. IF and EE have not previously been used for fault detection for wind turbines, and OCSVM has not been used for SCADA data. This paper presents a novel method of condition monitoring that only requires two months of data per turbine. These months were separated by a year, the first being healthy and the second unhealthy. The number of anomalies is compared, with a greater number in the unhealthy month being considered correct. It was found that for accuracy IF and OCSVM had similar performances in both training regimes presented. OCSVM performed better for generic training, and IF performed better for specific training. Overall, IF and OCSVM had an average accuracy of 82% for all configurations considered, compared to 77% for EE
IoT Anomaly Detection Methods and Applications: A Survey
Ongoing research on anomaly detection for the Internet of Things (IoT) is a
rapidly expanding field. This growth necessitates an examination of application
trends and current gaps. The vast majority of those publications are in areas
such as network and infrastructure security, sensor monitoring, smart home, and
smart city applications and are extending into even more sectors. Recent
advancements in the field have increased the necessity to study the many IoT
anomaly detection applications. This paper begins with a summary of the
detection methods and applications, accompanied by a discussion of the
categorization of IoT anomaly detection algorithms. We then discuss the current
publications to identify distinct application domains, examining papers chosen
based on our search criteria. The survey considers 64 papers among recent
publications published between January 2019 and July 2021. In recent
publications, we observed a shortage of IoT anomaly detection methodologies,
for example, when dealing with the integration of systems with various sensors,
data and concept drifts, and data augmentation where there is a shortage of
Ground Truth data. Finally, we discuss the present such challenges and offer
new perspectives where further research is required.Comment: 22 page
Performance Evaluation of Network Anomaly Detection Systems
Nowadays, there is a huge and growing concern about security in information and communication
technology (ICT) among the scientific community because any attack or anomaly in
the network can greatly affect many domains such as national security, private data storage,
social welfare, economic issues, and so on. Therefore, the anomaly detection domain is a broad
research area, and many different techniques and approaches for this purpose have emerged
through the years.
Attacks, problems, and internal failures when not detected early may badly harm an
entire Network system. Thus, this thesis presents an autonomous profile-based anomaly detection
system based on the statistical method Principal Component Analysis (PCADS-AD). This
approach creates a network profile called Digital Signature of Network Segment using Flow Analysis
(DSNSF) that denotes the predicted normal behavior of a network traffic activity through
historical data analysis. That digital signature is used as a threshold for volume anomaly detection
to detect disparities in the normal traffic trend. The proposed system uses seven traffic flow
attributes: Bits, Packets and Number of Flows to detect problems, and Source and Destination IP
addresses and Ports, to provides the network administrator necessary information to solve them.
Via evaluation techniques, addition of a different anomaly detection approach, and
comparisons to other methods performed in this thesis using real network traffic data, results
showed good traffic prediction by the DSNSF and encouraging false alarm generation and detection
accuracy on the detection schema.
The observed results seek to contribute to the advance of the state of the art in methods
and strategies for anomaly detection that aim to surpass some challenges that emerge from
the constant growth in complexity, speed and size of today’s large scale networks, also providing
high-value results for a better detection in real time.Atualmente, existe uma enorme e crescente preocupação com segurança em tecnologia
da informação e comunicação (TIC) entre a comunidade científica. Isto porque qualquer
ataque ou anomalia na rede pode afetar a qualidade, interoperabilidade, disponibilidade, e integridade
em muitos domínios, como segurança nacional, armazenamento de dados privados,
bem-estar social, questões econômicas, e assim por diante. Portanto, a deteção de anomalias
é uma ampla área de pesquisa, e muitas técnicas e abordagens diferentes para esse propósito
surgiram ao longo dos anos.
Ataques, problemas e falhas internas quando não detetados precocemente podem prejudicar
gravemente todo um sistema de rede. Assim, esta Tese apresenta um sistema autônomo
de deteção de anomalias baseado em perfil utilizando o método estatístico Análise de Componentes
Principais (PCADS-AD). Essa abordagem cria um perfil de rede chamado Assinatura Digital
do Segmento de Rede usando Análise de Fluxos (DSNSF) que denota o comportamento normal
previsto de uma atividade de tráfego de rede por meio da análise de dados históricos. Essa
assinatura digital é utilizada como um limiar para deteção de anomalia de volume e identificar
disparidades na tendência de tráfego normal. O sistema proposto utiliza sete atributos de fluxo
de tráfego: bits, pacotes e número de fluxos para detetar problemas, além de endereços IP e
portas de origem e destino para fornecer ao administrador de rede as informações necessárias
para resolvê-los.
Por meio da utilização de métricas de avaliação, do acrescimento de uma abordagem
de deteção distinta da proposta principal e comparações com outros métodos realizados nesta
tese usando dados reais de tráfego de rede, os resultados mostraram boas previsões de tráfego
pelo DSNSF e resultados encorajadores quanto a geração de alarmes falsos e precisão de deteção.
Com os resultados observados nesta tese, este trabalho de doutoramento busca contribuir
para o avanço do estado da arte em métodos e estratégias de deteção de anomalias,
visando superar alguns desafios que emergem do constante crescimento em complexidade, velocidade
e tamanho das redes de grande porte da atualidade, proporcionando também alta
performance. Ainda, a baixa complexidade e agilidade do sistema proposto contribuem para
que possa ser aplicado a deteção em tempo real
A survey of outlier detection methodologies
Outlier detection has been used for centuries to detect and, where appropriate, remove anomalous observations from data. Outliers arise due to mechanical faults, changes in system behaviour, fraudulent behaviour, human error, instrument error or simply through natural deviations in populations. Their detection can identify system faults and fraud before they escalate with potentially catastrophic consequences. It can identify errors and remove their contaminating effect on the data set and as such to purify the data for processing. The original outlier detection methods were arbitrary but now, principled and systematic techniques are used, drawn from the full gamut of Computer Science and Statistics. In this paper, we introduce a survey of contemporary techniques for outlier detection. We identify their respective motivations and distinguish their advantages and disadvantages in a comparative review
ANOMALY DETECTION - REVIEW OF METHODS, TOOLS AND ALGORITHMS
This paper contains review of algorithms, methods and tools nowadays used for anomaly detection.Anomaly detection is used in many domains of science and industry, some authors classify anomaly detection as data mining and data science tool, others state it is decision support tool under artificial intelligence domain and indeed the use cases of anomaly detection are very different.The article describes the main algorithms used for anomaly detection from perspective of theory of computer science and practical use cases of anomaly detection in different domains of industry. Several paragraphs are dedicated to the frameworks used by one of the most popular and powerful anomaly detection tools available in the market - Microsoft Anomaly detector.
Copula-based anomaly scoring and localization for large-scale, high-dimensional continuous data
The anomaly detection method presented by this paper has a special feature:
it does not only indicate whether an observation is anomalous or not but also
tells what exactly makes an anomalous observation unusual. Hence, it provides
support to localize the reason of the anomaly.
The proposed approach is model-based; it relies on the multivariate
probability distribution associated with the observations. Since the rare
events are present in the tails of the probability distributions, we use copula
functions, that are able to model the fat-tailed distributions well. The
presented procedure scales well; it can cope with a large number of
high-dimensional samples. Furthermore, our procedure can cope with missing
values, too, which occur frequently in high-dimensional data sets.
In the second part of the paper, we demonstrate the usability of the method
through a case study, where we analyze a large data set consisting of the
performance counters of a real mobile telecommunication network. Since such
networks are complex systems, the signs of sub-optimal operation can remain
hidden for a potentially long time. With the proposed procedure, many such
hidden issues can be isolated and indicated to the network operator.Comment: 27 pages, 12 figures, accepted at ACM Transactions on Intelligent
Systems and Technolog
Unsupervised anomaly detection for unlabelled wireless sensor networks data
With the advances in sensor technology, sensor nodes, the tiny yet powerful device are used to collect data from the various domain. As the sensor nodes communicate continuously from the target areas to base station, hundreds of thousands of data are collected to be used for the decision making. Unfortunately, the big amount of unlabeled data collected and stored at the base station. In most cases, data are not reliable due to several reasons. Therefore, this paper will use the unsupervised one-class SVM (OCSVM) to build the anomaly detection schemes for better decision making. Unsupervised OCSVM is preferable to be used in WSNs domain due to the one class of data training is used to build normal reference model. Furthermore, the dimension reduction is used to minimize the resources usage due to resource constraint incurred in WSNs domain. Therefore one of the OCSVM variants namely Centered Hyper-ellipsoidal Support Vector Machine (CESVM) is used as classifier while Candid-Covariance Free Incremental Principal Component Analysis (CCIPCA) algorithm is served as dimension reduction for proposed anomaly detection scheme. Environmental dataset collected from available WSNs data is used to evaluate the performance measures of the proposed scheme. As the results, the proposed scheme shows comparable results for all datasets in term of detection rate, detection accuracy and false alarm rate as compared with other related methods
- …