A refined filter for UHAD to improve anomaly detection

Filtering is used in intrusion detection to remove the insignificant events from a log to facilitate the analysis method to focus on the significant events and to minimize processing overhead. Generally, filtering is performed using filtering rules, which are framed using a set of data training data, or the known facts on anomalous events. This knowledge-dependent nature confines the filterer to filter-in only the recognized anomalies in the logs, making the rest unavailable for further scrutiny. This problem has been addressed earlier by designing a filterer that manipulates the tested log data based on the patterns and volume of events to calculate the filtering threshold. Even though this filtering threshold was able to retain the anomalous events in most heterogeneous logs, it failed when such events were of high volume and also due to the inaccuracies in cluster formation. Therefore, this paper proposes a refined filterer for unsupervised heterogeneous anomaly detection that retains most anomalous events irrespective of its volume in the logs and also discusses the impact of the refined filterer in supporting the detection. The experiment conducted reveals that the refined filterer retained almost all the abnormal events thereby enabling the detection of maximum anomalies