230 research outputs found
Big Data in Critical Infrastructures Security Monitoring: Challenges and Opportunities
Critical Infrastructures (CIs), such as smart power grids, transport systems,
and financial infrastructures, are more and more vulnerable to cyber threats,
due to the adoption of commodity computing facilities. Despite the use of
several monitoring tools, recent attacks have proven that current defensive
mechanisms for CIs are not effective enough against most advanced threats. In
this paper we explore the idea of a framework leveraging multiple data sources
to improve protection capabilities of CIs. Challenges and opportunities are
discussed along three main research directions: i) use of distinct and
heterogeneous data sources, ii) monitoring with adaptive granularity, and iii)
attack modeling and runtime combination of multiple data analysis techniques.Comment: EDCC-2014, BIG4CIP-201
An Unsupervised Anomaly Detection Framework for Detecting Anomalies in Real Time through Network System’s Log Files Analysis
Nowadays, in almost every computer system, log files are used to keep records of occurring events. Those log files are then used for analyzing and debugging system failures. Due to this important utility, researchers have worked on finding fast and efficient ways to detect anomalies in a computer system by analyzing its log records. Research in log-based anomaly detection can be divided into two main categories: batch log-based anomaly detection and streaming logbased anomaly detection. Batch log-based anomaly detection is computationally heavy and does not allow us to instantaneously detect anomalies. On the other hand, streaming anomaly detection allows for immediate alert. However, current streaming approaches are mainly supervised. In this work, we propose a fully unsupervised framework which can detect anomalies in real time. We test our framework on hdfs log files and successfully detect anomalies with an F- 1 score of 83%
LogBERT: Log Anomaly Detection via BERT
When systems break down, administrators usually check the produced logs to diagnose the failures. Nowadays, systems grow larger and more complicated. It is labor-intensive to manually detect abnormal behaviors in logs. Therefore, it is necessary to develop an automated anomaly detection on system logs. Automated anomaly detection not only identifies malicious patterns promptly but also requires no prior domain knowledge. Many existing log anomaly detection approaches apply natural language models such as Recurrent Neural Network (RNN) to log analysis since both are based on sequential data. The proposed model, LogBERT, a BERT-based neural network, can capture the contextual information in log sequences.
LogBERT is trained on normal log data considering the scarcity of labeled abnormal data in reality. Intuitively, LogBERT learns normal patterns in training data and flags test data that are deviated from prediction as anomalies. We compare LogBERT with four traditional machine learning models and two deep learning models in terms of precision, recall, and F1 score on three public datasets, HDFS, BGL, and Thunderbird. Overall, LogBERT outperforms the state-of-art models for log anomaly detection
LogGD:Detecting Anomalies from System Logs by Graph Neural Networks
Log analysis is one of the main techniques engineers use to troubleshoot
faults of large-scale software systems. During the past decades, many log
analysis approaches have been proposed to detect system anomalies reflected by
logs. They usually take log event counts or sequential log events as inputs and
utilize machine learning algorithms including deep learning models to detect
system anomalies. These anomalies are often identified as violations of
quantitative relational patterns or sequential patterns of log events in log
sequences. However, existing methods fail to leverage the spatial structural
relationships among log events, resulting in potential false alarms and
unstable performance. In this study, we propose a novel graph-based log anomaly
detection method, LogGD, to effectively address the issue by transforming log
sequences into graphs. We exploit the powerful capability of Graph Transformer
Neural Network, which combines graph structure and node semantics for log-based
anomaly detection. We evaluate the proposed method on four widely-used public
log datasets. Experimental results show that LogGD can outperform
state-of-the-art quantitative-based and sequence-based methods and achieve
stable performance under different window size settings. The results confirm
that LogGD is effective in log-based anomaly detection.Comment: 12 pages, 12 figure
Log-based Anomaly Detection of CPS Using a Statistical Method
Detecting anomalies of a cyber physical system (CPS), which is a complex
system consisting of both physical and software parts, is important because a
CPS often operates autonomously in an unpredictable environment. However,
because of the ever-changing nature and lack of a precise model for a CPS,
detecting anomalies is still a challenging task. To address this problem, we
propose applying an outlier detection method to a CPS log. By using a log
obtained from an actual aquarium management system, we evaluated the
effectiveness of our proposed method by analyzing outliers that it detected. By
investigating the outliers with the developer of the system, we confirmed that
some outliers indicate actual faults in the system. For example, our method
detected failures of mutual exclusion in the control system that were unknown
to the developer. Our method also detected transient losses of functionalities
and unexpected reboots. On the other hand, our method did not detect anomalies
that were too many and similar. In addition, our method reported rare but
unproblematic concurrent combinations of operations as anomalies. Thus, our
approach is effective at finding anomalies, but there is still room for
improvement
- …