Log Message Anomaly Detection using Positive and Unlabeled Learning

Abstract

Log messages are widely used in cloud servers and software systems. Anomaly detection of log messages is important as millions of logs are generated each day. However, besides having a complex and unstructured form, log messages are large unlabeled datasets which makes classification very difficult. In this thesis, a log message anomaly detection technique is proposed which employs Positive and Unlabeled Learning (PU Learning) to detect anomalies. Aggregated reliable negative logs are selected using the Isolation Forest, PU Learning, and Random Forest algorithms. Then, anomaly detection is conducted using deep learning Long Short-Term Memory (LSTM) network. The proposed model is evaluated using the commonly employed Openstack, BGL, and Thunderbird datasets and the results obtained indicate that the proposed model performs better than several well-known approaches in the literature.Graduat