28 research outputs found
LogLG: Weakly Supervised Log Anomaly Detection via Log-Event Graph Construction
Fully supervised log anomaly detection methods suffer the heavy burden of
annotating massive unlabeled log data. Recently, many semi-supervised methods
have been proposed to reduce annotation costs with the help of parsed
templates. However, these methods consider each keyword independently, which
disregards the correlation between keywords and the contextual relationships
among log sequences. In this paper, we propose a novel weakly supervised log
anomaly detection framework, named LogLG, to explore the semantic connections
among keywords from sequences. Specifically, we design an end-to-end iterative
process, where the keywords of unlabeled logs are first extracted to construct
a log-event graph. Then, we build a subgraph annotator to generate pseudo
labels for unlabeled log sequences. To ameliorate the annotation quality, we
adopt a self-supervised task to pre-train a subgraph annotator. After that, a
detection model is trained with the generated pseudo labels. Conditioned on the
classification results, we re-extract the keywords from the log sequences and
update the log-event graph for the next iteration. Experiments on five
benchmarks validate the effectiveness of LogLG for detecting anomalies on
unlabeled log data and demonstrate that LogLG, as the state-of-the-art weakly
supervised method, achieves significant performance improvements compared to
existing methods.Comment: 12 page
An Unsupervised Anomaly Detection Framework for Detecting Anomalies in Real Time through Network System’s Log Files Analysis
Nowadays, in almost every computer system, log files are used to keep records of occurring events. Those log files are then used for analyzing and debugging system failures. Due to this important utility, researchers have worked on finding fast and efficient ways to detect anomalies in a computer system by analyzing its log records. Research in log-based anomaly detection can be divided into two main categories: batch log-based anomaly detection and streaming logbased anomaly detection. Batch log-based anomaly detection is computationally heavy and does not allow us to instantaneously detect anomalies. On the other hand, streaming anomaly detection allows for immediate alert. However, current streaming approaches are mainly supervised. In this work, we propose a fully unsupervised framework which can detect anomalies in real time. We test our framework on hdfs log files and successfully detect anomalies with an F- 1 score of 83%
LogUAD: Log unsupervised anomaly detection based on word2Vec
System logs record detailed information about system operation and are important for analyzing the system\u27s operational status and performance. Rapid and accurate detection of system anomalies is of great significance to ensure system stability. However, large-scale distributed systems are becoming more and more complex, and the number of system logs gradually increases, which brings challenges to analyze system logs. Some recent studies show that logs can be unstable due to the evolution of log statements and noise introduced by log collection and parsing. Moreover, deep learning-based detection methods take a long time to train models. Therefore, to reduce the computational cost and avoid log instability we propose a new Word2Vec-based log unsupervised anomaly detection method (LogUAD). LogUAD does not require a log parsing step and takes original log messages as input to avoid the noise. LogUAD uses Word2Vec to generate word vectors and generates weighted log sequence feature vectors with TF-IDF to handle the evolution of log statements. At last, a computationally efficient unsupervised clustering is exploited to detect the anomaly. We conducted extensive experiments on the public dataset from Blue Gene/L (BGL). Experimental results show that the F1-score of LogUAD can be improved by 67.25% compared to LogCluster
CSCLog: A Component Subsequence Correlation-Aware Log Anomaly Detection Method
Anomaly detection based on system logs plays an important role in intelligent
operations, which is a challenging task due to the extremely complex log
patterns. Existing methods detect anomalies by capturing the sequential
dependencies in log sequences, which ignore the interactions of subsequences.
To this end, we propose CSCLog, a Component Subsequence Correlation-Aware Log
anomaly detection method, which not only captures the sequential dependencies
in subsequences, but also models the implicit correlations of subsequences.
Specifically, subsequences are extracted from log sequences based on components
and the sequential dependencies in subsequences are captured by Long Short-Term
Memory Networks (LSTMs). An implicit correlation encoder is introduced to model
the implicit correlations of subsequences adaptively. In addition, Graph
Convolution Networks (GCNs) are employed to accomplish the information
interactions of subsequences. Finally, attention mechanisms are exploited to
fuse the embeddings of all subsequences. Extensive experiments on four publicly
available log datasets demonstrate the effectiveness of CSCLog, outperforming
the best baseline by an average of 7.41% in Macro F1-Measure.Comment: submitted to TKDD, 18 pages and 7 figure
Robust Multimodal Failure Detection for Microservice Systems
Proactive failure detection of instances is vitally essential to microservice
systems because an instance failure can propagate to the whole system and
degrade the system's performance. Over the years, many single-modal (i.e.,
metrics, logs, or traces) data-based nomaly detection methods have been
proposed. However, they tend to miss a large number of failures and generate
numerous false alarms because they ignore the correlation of multimodal data.
In this work, we propose AnoFusion, an unsupervised failure detection approach,
to proactively detect instance failures through multimodal data for
microservice systems. It applies a Graph Transformer Network (GTN) to learn the
correlation of the heterogeneous multimodal data and integrates a Graph
Attention Network (GAT) with Gated Recurrent Unit (GRU) to address the
challenges introduced by dynamically changing multimodal data. We evaluate the
performance of AnoFusion through two datasets, demonstrating that it achieves
the F1-score of 0.857 and 0.922, respectively, outperforming the
state-of-the-art failure detection approaches