Securing Enterprise Networks with Statistical Node Behavior Profiling

The substantial proliferation of the Internet has made it the most critical infrastructure in today\u27s world. However, it is still vulnerable to various kinds of attacks/malwares and poses a number of great security challenges. Furthermore, we have also witnessed in the past decade that there is always a fast self-evolution of attacks/malwares (e.g. from worms to botnets) against every success in network security. Network security thereby remains a hot topic in both research and industry and requires both continuous and great attention. In this research, we consider two fundamental areas in network security, malware detection and background traffic modeling, from a new view point of node behavior profiling under enterprise network environments. Our main objectives are to extend and enhance the current research in these two areas. In particular, central to our research is the node behavior profiling approach that groups the behaviors of different nodes by jointly considering time and spatial correlations. We also present an extensive study on botnets, which are believed to be the largest threat to the Internet. To better understand the botnet, we propose a botnet framework and predict a new P2P botnet that is much stronger and stealthier than the current ones. We then propose anomaly malware detection approaches based directly on the insights (statistical characteristics) from the node behavior study and apply them on P2P botnet detection. Further, by considering the worst case attack model where the botmaster knows all the parameter values used in detection, we propose a fast and optimized anomaly detection approach by formulating the detection problem as an optimization problem. In addition, we propose a novel traffic modeling structure using behavior profiles for NIDS evaluations. It is efficient and takes into account the node heterogeneity in traffic modeling. It is also compatible with most current modeling schemes and helpful in generating better realistic background traffic. Last but not least, we evaluate the proposed approaches using real user trace from enterprise networks and achieve encouraging results. Our contributions in this research include: 1) a new node behavior profiling approach to study the normal node behavior; 2) a framework for botnets; 3) a new P2P botnet and performance comparisons with other P2P botnets; 4) two anomaly detection approaches based on node behavior profiles; 4) a fast and optimized anomaly detection approach under the worst case attack model; 5) a new traffic modeling structure and 6) simulations and evaluations of the above approaches under real user data from enterprise networks. To the best of our knowledge, we are the first to propose the botnet framework, consider the worst case attack model and propose corresponding fast and optimized solution in botnet related research. We are also the first to propose efficient solutions in traffic modeling without the assumption of node homogeneity

JOINT DETECTION-STATE ESTIMATION AND SECURE SIGNAL PROCESSING

In this dissertation, joint detection-state estimation and secure signal processing are studied. Detection and state estimation are two important research topics in surveillance systems. The detection problems investigated in this dissertation include object detection and fault detection. The goal of object detection is to determine the presence or absence of an object under measurement uncertainty. The aim of fault detection is to determine whether or not the measurements are provided by faulty sensors. State estimation is to estimate the states of moving objects from measurements with random measurement noise or disturbance, which typically consist of their positions and velocities over time. Detection and state estimation are typically implemented separately and state estimation is usually performed after the decision is made. In this two-stage approach, missed detection and false alarms in detection stage decrease accuracy of state estimation. In this dissertation, several joint detection and state estimation algorithms are proposed. Secure signal processing is indispensable in dynamic systems especially when an adversary exists. In this dissertation, the developed joint fault detection and state estimation approach is used to detect attacks launched by an adversary on the system and improve state estimation accuracy. The security problem in satellite communication systems is studied and a minimax anti-jammer is designed in a frequency hopping spread spectrum (FHSS)/quadrature phase-shift keying (QPSK) satellite communication system

Spectrum Sensing and Multiple Access Schemes for Cognitive Radio Networks

Increasing demands on the radio spectrum have driven wireless engineers to rethink approaches by which devices should access this natural, and arguably scarce, re- source. Cognitive Radio (CR) has arisen as a new wireless communication paradigm aimed at solving the spectrum underutilization problem. In this thesis, we explore a novel variety of techniques aimed at spectrum sensing which serves as a fundamental mechanism to find unused portions of the electromagnetic spectrum. We present several spectrum sensing methods based on multiple antennas and evaluate their receiving operating characteristics. We study a cyclostationary feature detection technique by means of multiple cyclic frequencies. We make use of a spec- trum sensing method called sequential analysis that allows us to significantly decrease the time needed for detecting the presence of a licensed user. We extend this scheme allowing each CR user to perform the sequential analysis algorithm and send their local decision to a fusion centre. This enables for an average faster and more accurate detection. We present an original technique for accounting for spatial and temporal cor- relation influence in spectrum sensing. This reflects on the impact of the scattering environment on detection methods using multiple antennas. The approach is based on the scattering geometry and resulting correlation properties of the received signal at each CR device. Finally, the problem of spectrum sharing for CR networks is addressed in or- der to take advantage of the detected unused frequency bands. We proposed a new multiple access scheme based on the Game Theory. We examine the scenario where a random number of CR users (considered as players) compete to access the radio spec- trum. We calculate the optimal probability of transmission which maximizes the CR throughput along with the minimum harm caused to the licensed users’ performance

Byzantine Attack and Defense in Cognitive Radio Networks: A Survey

The Byzantine attack in cooperative spectrum sensing (CSS), also known as the spectrum sensing data falsification (SSDF) attack in the literature, is one of the key adversaries to the success of cognitive radio networks (CRNs). In the past couple of years, the research on the Byzantine attack and defense strategies has gained worldwide increasing attention. In this paper, we provide a comprehensive survey and tutorial on the recent advances in the Byzantine attack and defense for CSS in CRNs. Specifically, we first briefly present the preliminaries of CSS for general readers, including signal detection techniques, hypothesis testing, and data fusion. Second, we analyze the spear and shield relation between Byzantine attack and defense from three aspects: the vulnerability of CSS to attack, the obstacles in CSS to defense, and the games between attack and defense. Then, we propose a taxonomy of the existing Byzantine attack behaviors and elaborate on the corresponding attack parameters, which determine where, who, how, and when to launch attacks. Next, from the perspectives of homogeneous or heterogeneous scenarios, we classify the existing defense algorithms, and provide an in-depth tutorial on the state-of-the-art Byzantine defense schemes, commonly known as robust or secure CSS in the literature. Furthermore, we highlight the unsolved research challenges and depict the future research directions.Comment: Accepted by IEEE Communications Surveys and Tutoiral

Adaptive Estimation and Detection Techniques with Applications

Hybrid systems have been identified as one of the main directions in control theory and attracted increasing attention in recent years due to their huge diversity of engineering applications. Multiplemodel (MM) estimation is the state-of-the-art approach to many hybrid estimation problems. Existing MM methods with fixed structure usually perform well for problems that can be handled by a small set of models. However, their performance is limited when the required number of models to achieve a satisfactory accuracy is large due to time evolution of the true mode over a large continuous space. In this research, variable-structure multiple model (VSMM) estimation was investigated, further developed and evaluated. A fundamental solution for on-line adaptation of model sets was developed as well as several VSMM algorithms. These algorithms have been successfully applied to the fields of fault detection and identification as well as target tracking in this thesis. In particular, an integrated framework to detect, identify and estimate failures is developed based on the VSMM. It can handle sequential failures and multiple failures by sensors or actuators. Fault detection and target maneuver detection can be formulated as change-point detection problems in statistics. It is of great importance to have the quickest detection of such mode changes in a hybrid system. Traditional maneuver detectors based on simplistic models are not optimal and are computationally demanding due to the requirement of batch processing. In this presentation, a general sequential testing procedure is proposed for maneuver detection based on advanced sequential tests. It uses a likelihood marginalization technique to cope with the difficulty that the target accelerations are unknown. The approach essentially utilizes a priori information about the accelerations in typical tracking engagements and thus allows improved detection performance. The proposed approach is applicable to change-point detection problems under similar formulation, such as fault detection

Adaptive Estimation and Detection Techniques with Applications

Hybrid systems have been identified as one of the main directions in control theory and attracted increasing attention in recent years due to their huge diversity of engineering applications. Multiplemodel (MM) estimation is the state-of-the-art approach to many hybrid estimation problems. Existing MM methods with fixed structure usually perform well for problems that can be handled by a small set of models. However, their performance is limited when the required number of models to achieve a satisfactory accuracy is large due to time evolution of the true mode over a large continuous space. In this research, variable-structure multiple model (VSMM) estimation was investigated, further developed and evaluated. A fundamental solution for on-line adaptation of model sets was developed as well as several VSMM algorithms. These algorithms have been successfully applied to the fields of fault detection and identification as well as target tracking in this thesis. In particular, an integrated framework to detect, identify and estimate failures is developed based on the VSMM. It can handle sequential failures and multiple failures by sensors or actuators. Fault detection and target maneuver detection can be formulated as change-point detection problems in statistics. It is of great importance to have the quickest detection of such mode changes in a hybrid system. Traditional maneuver detectors based on simplistic models are not optimal and are computationally demanding due to the requirement of batch processing. In this presentation, a general sequential testing procedure is proposed for maneuver detection based on advanced sequential tests. It uses a likelihood marginalization technique to cope with the difficulty that the target accelerations are unknown. The approach essentially utilizes a priori information about the accelerations in typical tracking engagements and thus allows improved detection performance. The proposed approach is applicable to change-point detection problems under similar formulation, such as fault detection

An Integrated Fuzzy Inference Based Monitoring, Diagnostic, and Prognostic System

To date the majority of the research related to the development and application of monitoring, diagnostic, and prognostic systems has been exclusive in the sense that only one of the three areas is the focus of the work. While previous research progresses each of the respective fields, the end result is a variable grab bag of techniques that address each problem independently. Also, the new field of prognostics is lacking in the sense that few methods have been proposed that produce estimates of the remaining useful life (RUL) of a device or can be realistically applied to real-world systems. This work addresses both problems by developing the nonparametric fuzzy inference system (NFIS) which is adapted for monitoring, diagnosis, and prognosis and then proposing the path classification and estimation (PACE) model that can be used to predict the RUL of a device that does or does not have a well defined failure threshold. To test and evaluate the proposed methods, they were applied to detect, diagnose, and prognose faults and failures in the hydraulic steering system of a deep oil exploration drill. The monitoring system implementing an NFIS predictor and sequential probability ratio test (SPRT) detector produced comparable detection rates to a monitoring system implementing an autoassociative kernel regression (AAKR) predictor and SPRT detector, specifically 80% vs. 85% for the NFIS and AAKR monitor respectively. It was also found that the NFIS monitor produced fewer false alarms. Next, the monitoring system outputs were used to generate symptom patterns for k-nearest neighbor (kNN) and NFIS classifiers that were trained to diagnose different fault classes. The NFIS diagnoser was shown to significantly outperform the kNN diagnoser, with overall accuracies of 96% vs. 89% respectively. Finally, the PACE implementing the NFIS was used to predict the RUL for different failure modes. The errors of the RUL estimates produced by the PACE-NFIS prognosers ranged from 1.2-11.4 hours with 95% confidence intervals (CI) from 0.67-32.02 hours, which are significantly better than the population based prognoser estimates with errors of ~45 hours and 95% CIs of ~162 hours

Efficient Algorithms for Robust Estimation

One of the most commonly encountered tasks in computer vision is the estimation of model parameters from image measurements. This scenario arises in a variety of applications -- for instance, in the estimation of geometric entities, such as camera pose parameters, from feature matches between images. The main challenge in this task is to handle the problem of outliers -- in other words, data points that do not conform to the model being estimated. It is well known that if these outliers are not properly accounted for, even a single outlier in the data can result in arbitrarily bad model estimates. Due to the widespread prevalence of problems of this nature, the field of robust estimation has been well studied over the years, both in the statistics community as well as in computer vision, leading to the development of popular algorithms like Random Sample Consensus (RANSAC). While recent years have seen exciting advances in this area, a number of important issues still remain open. In this dissertation, we aim to address some of these challenges. The main goal of this dissertation is to advance the state of the art in robust estimation techniques by developing algorithms capable of efficiently and accurately delivering model parameter estimates in the face of noise and outliers. To this end, the first contribution of this work is in the development of a coherent framework for the analysis of RANSAC-based robust estimators, which consolidates various improvements made over the years. In turn, this analysis leads naturally to the development of new techniques that combine the strengths of existing methods, and yields high-performance robust estimation algorithms, including for real-time applications. A second contribution of this dissertation is the development of an algorithm that explicitly characterizes the effects of estimation uncertainty in RANSAC. This uncertainty arises from small-scale measurement noise that affects the data points, and consequently, impacts the accuracy of model parameters. We show that knowledge of this measurement noise can be leveraged to develop an inlier classification scheme that is dependent on the model uncertainty, as opposed to a fixed inlier threshold, as in RANSAC. This has the advantage that, given a model with associated uncertainty, we can immediately identify a set of points that support this solution, which in turn leads to an improvement in computational efficiency. Finally, we have also developed an approach to addresses the issue of the inlier threshold, which is a user-supplied parameter that can vary depending on the estimation problem and the data being processed. Our technique is based on the intuition that the residual errors for good models are in some way consistent with each other, while bad models do not exhibit this consistency. In other words, looking at the relationship between \\subsets of models can reveal useful information about the validity of the models themselves. We show that it is possible to efficiently identify this consistent behaviour by exploiting residual ordering information coupled with simple non-parametric statistical tests, which leads to an effective algorithm for threshold-free robust estimation.Doctor of Philosoph