Temporally adaptive monitoring procedures with applications in enterprise cyber-security

Due to the perpetual threat of cyber-attacks, enterprises must employ and develop new methods of detection as attack vectors evolve and advance. Enterprise computer networks produce a large volume and variety of data including univariate data streams, time series and network graph streams. Motivated by cyber-security, this thesis develops adaptive monitoring tools for univariate and network graph data streams, however, they are not limited to this domain. In all domains, real data streams present several challenges for monitoring including trend, periodicity and change points. Streams often also have high volume and frequency. To deal with the non-stationarity in the data, the methods applied must be adaptive. Adaptability in the proposed procedures throughout the thesis is introduced using forgetting factors, weighting the data accordingly to recency. Secondly, methods applied must be computationally fast with a small or fixed computation burden and fixed storage requirements for timely processing. Throughout this thesis, sequential or sliding window approaches are employed to achieve this. The first part of the thesis is centred around univariate monitoring procedures. A sequential adaptive parameter estimator is proposed using a Bayesian framework. This procedure is then extended for multiple change point detection, where, unlike existing change point procedures, the proposed method is capable of detecting abrupt changes in the presence of trend. We additionally present a time series model which combines short-term and long-term behaviours of a series for improved anomaly detection. Unlike existing methods which primarily focus on point anomalies detection (extreme outliers), our method is capable of also detecting contextual anomalies, when the data deviates from persistent patterns of the series such as seasonality. Finally, a novel multi-type relational clustering methodology is proposed. As multiple relations exist between the different entities within a network (computers, users and ports), multiple network graphs can be generated. We propose simultaneously clustering over all graphs to produce a single clustering for each entity using Non-Negative Matrix Tri-Factorisation. Through simplifications, the proposed procedure is fast and scalable for large network graphs. Additionally, this methodology is extended for graph streams. This thesis provides an assortment of tools for enterprise network monitoring with a focus on adaptability and scalability making them suitable for intrusion detection and situational awareness.Open Acces

Unsupervised methods to discover events from spatio-temporal data

University of Minnesota Ph.D. dissertation. May 2016. Major: Computer Science. Advisor: Vipin Kumar. 1 computer file (PDF); ix, 110 pages.Unsupervised event detection in spatio-temporal data aims to autonomously identify when and/or where events occurred with little or no human supervision. It is an active field of research with notable applications in social, Earth, and medical sciences. While event detection has enjoyed tremendous success in many domains, it is still a challenging problem due to the vastness of data points, presence of noise and missing values, the heterogeneous nature of spatio-temporal signals, and the large variety of event types. Unsupervised event detection is a broad and yet open research area. Instead of exploring every aspect in this area, this dissertation focuses on four novel algorithms that covers two types of important events in spatio-temporal data: change-points and moving regions. The first algorithm in this dissertation is the Persistence-Consistency (PC) framework. It is a general framework that can increase the robustness of change-point detection algorithms to noise and outliers. The major advantage of the PC framework is that it can work with most modeling-based change-point detection algorithms and improve their performance without modifying the selected change-point detection algorithm. We use two real-world applications, forest fire detection using a satellite dataset and activity segmentation from a mobile health dataset, to test the effectiveness of this framework. The second and third algorithms in this dissertation are proposed to detect a novel type of change point, which is named as contextual change points. While most existing change points more or less indicate that the time series is different from what it was before, a contextual change point typically suggests an event that causes the relationship of several time series changes. Each of these two algorithms introduces one type of contextual change point and also presents an algorithm to detect the corresponding type of change point. We demonstrate the unique capabilities of these approaches with two applications: event detection in stock market data and forest fire detection using remote sensing data. The final algorithm in this dissertation is a clustering method that discovers a particular type of moving regions (or dynamic spatio-temporal patterns) in noisy, incomplete, and heterogeneous data. This task faces two major challenges: First, the regions (or clusters) are dynamic and may change in size, shape, and statistical properties over time. Second, numerous spatio-temporal data are incomplete, noisy, heterogeneous, and highly variable (over space and time). Our proposed approach fully utilizes the spatial contiguity and temporal similarity in the spatio-temporal data and, hence, can address the above two challenges. We demonstrate the performance of the proposed method on a real-world application of monitoring in-land water bodies on a global scale

Spatio-temporal landslide inventory and susceptibility assessment using Sentinel-2 in the Himalayan mountainous region of Pakistan

The 2005 Kashmir earthquake has triggered widespread landslides in the Himalayan mountains in northern Pakistan and surrounding areas, some of which are active and are still posing a significant risk. Landslides triggered by the 2005 Kashmir earthquake are extensively studied; nevertheless, spatio-temporal landslide susceptibility assessment is lacking. This can be partially attributed to the limited availability of high temporal resolution remote sensing data. We present a semi-automated technique to use the Sentinel-2 MSI data for co-seismic landslide detection, landslide activities monitoring, spatio-temporal change detection, and spatio-temporal susceptibility mapping. Time series Sentinel-2 MSI images for the period of 2016–2021 and ALOS PALSAR DEM are used for semi-automated landslide inventory map development and temporal change analysis. Spectral information combined with topographical, contextual, textural, and morphological characteristics of the landslide in Sentinel-2 images is applied for landslide detection. Subsequently, spatio-temporal landslide susceptibility maps are developed utilizing the weight of evidence statistical modeling with seven causative factors, i.e., elevation, slope, geology, aspect, distance to fault, distance to roads, and distance to streams. The results reveal that landslide occurrence increased from 2016 to 2021 and that the coverage of areas of relatively high susceptibility has increased in the study area

Enrichment of raw sensor data to enable high-level queries

Sensor networks are increasingly used across various application domains. Their usage has the advantage of automated, often continuous, monitoring of activities and events. Ubiquitous sensor networks detect location of people and objects and their movement. In our research, we employ a ubiquitous sensor network to track the movement of players in a tennis match. By doing so, our goal is to create a detailed analysis of how the match progressed, recording points scored, games and sets, and in doing so, greatly reduce the eort of coaches and players who are required to study matches afterwards. The sensor network is highly efficient as it eliminates the need for manual recording of the match. However, it generates raw data that is unusable by domain experts as it contains no frame of reference or context and cannot be analyzed or queried. In this work, we present the UbiQuSE system of data transformers which bridges the gap between raw sensor data and the high-level requirements of domain specialists such as the tennis coach

Adding Contextual Information to Intrusion Detection Systems Using Fuzzy Cognitive Maps

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link.In the last few years there has been considerable increase in the efficiency of Intrusion Detection Systems (IDSs). However, networks are still the victim of attacks. As the complexity of these attacks keeps increasing, new and more robust detection mechanisms need to be developed. The next generation of IDSs should be designed incorporating reasoning engines supported by contextual information about the network, cognitive information and situational awareness to improve their detection results. In this paper, we propose the use of a Fuzzy Cognitive Map (FCM) in conjunction with an IDS to incorporate contextual information into the detection process. We have evaluated the use of FCMs to adjust the Basic Probability Assignment (BPA) values defined prior to the data fusion process, which is crucial for the IDS that we have developed. The experimental results that we present verify that FCMs can improve the efficiency of our IDS by reducing the number of false alarms, while not affecting the number of correct detections