On the Adaptive Real-Time Detection of Fast-Propagating Network Worms

We present two light-weight worm detection algorithms thatoffer significant advantages over fixed-threshold methods.The first algorithm, RBS (rate-based sequential hypothesis testing)aims at the large class of worms that attempts to quickly propagate, thusexhibiting abnormal levels of the rate at which hosts initiateconnections to new destinations. The foundation of RBS derives fromthe theory of sequential hypothesis testing, the use of which fordetecting randomly scanning hosts was first introduced by our previouswork with the TRW (Threshold Random Walk) scan detection algorithm. The sequential hypothesistesting methodology enables engineering the detectors to meet falsepositives and false negatives targets, rather than triggering whenfixed thresholds are crossed. In this sense, the detectors that weintroduce are truly adaptive.We then introduce RBS+TRW, an algorithm that combines fan-out rate (RBS)and probability of failure (TRW) of connections to new destinations.RBS+TRW provides a unified framework that at one end acts as a pure RBSand at the other end as pure TRW, and extends RBS's power in detectingworms that scan randomly selected IP addresses

Real-time detection of malicious network activity using stochastic models

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2006.Includes bibliographical references (p. 115-122).This dissertation develops approaches to rapidly detect malicious network traffic including packets sent by portscanners and network worms. The main hypothesis is that stochastic models capturing a host's particular connection-level behavior provide a good foundation for identifying malicious network activity in real-time. Using the models, the dissertation shows that a detection problem can be formulated as one of observing a particular "trajectory" of arriving packets and inferring from it the most likely classification for the given host's behavior. This stochastic approach enables us not only to estimate an algorithm's performance based on the measurable statistics of a host's traffic but also to balance the goals of promptness and accuracy in detecting malicious network activity. This dissertation presents three detection algorithms based on Wald's mathematical framework of sequential analysis. First, Threshold Random Walk (TRW) rapidly detects remote hosts performing a portscan to a target network. TRW is motivated by the empirically observed disparity between the frequency with which connections to newly visited local addresses are successful for benign hosts vs. for portscanners. Second, it presents a hybrid approach that accurately detects scanning worm infections quickly after the infected local host begins to engage in worm propagation.(cont.) Finally, it presents a targeting worm detection algorithm, Rate-Based Sequential Hypothesis Testing (RBS), that promptly identifies high-fan-out behavior by hosts (e.g., targeting worms) based on the rate at which the hosts initiate connections to new destinations. RBS is built on an empirically-driven probability model that captures benign network characteristics. It then presents RBS+TRW, a unified framework for detecting fast-propagating worms independently of their target discovery strategy. All these schemes have been implemented and evaluated using real packet traces collected from multiple network vantage points.by Jaeyeon Jung.Ph.D

Graph-theoretic Approach To Modeling Propagation And Control Of Network Worms

In today\u27s network-dependent society, cyber attacks with network worms have become the predominant threat to confidentiality, integrity, and availability of network computing resources. Despite ongoing research efforts, there is still no comprehensive network-security solution aimed at controling large-scale worm propagation. The aim of this work is fivefold: (1) Developing an accurate combinatorial model of worm propagation that can facilitate the analysis of worm control strategies, (2) Building an accurate epidemiological model for the propagation of a worm employing local strategies, (3) Devising distributed architecture and algorithms for detection of worm scanning activities, (4) Designing effective control strategies against the worm, and (5) Simulation of the developed models and strategies on large, scale-free graphs representing real-world communication networks. The proposed pair-approximation model uses the information about the network structure--order, size, degree distribution, and transitivity. The empirical study of propagation on large scale-free graphs is in agreement with the theoretical analysis of the proposed pair-approximation model. We, then, describe a natural generalization of the classical cops-and-robbers game--a combinatorial model of worm propagation and control. With the help of this game on graphs, we show that the problem of containing the worm is NP-hard. Six novel near-optimal control strategies are devised: combination of static and dynamic immunization, reactive dynamic and invariant dynamic immunization, soft quarantining, predictive traffic-blocking, and contact-tracing. The analysis of the predictive dynamic traffic-blocking, employing only local information, shows that the worm can be contained so that 40\% of the network nodes are not affected. Finally, we develop the Detection via Distributed Blackholes architecture and algorithm which reflect the propagation strategy used by the worm and the salient properties of the network. Our distributed detection algorithm can detect the worm scanning activity when only 1.5% of the network has been affected by the propagation. The proposed models and algorithms are analyzed with an individual-based simulation of worm propagation on realistic scale-free topologies

Towards secure message systems

Message systems, which transfer information from sender to recipient via communication networks, are indispensable to our modern society. The enormous user base of message systems and their critical role in information delivery make it the top priority to secure message systems. This dissertation focuses on securing the two most representative and dominant messages systems---e-mail and instant messaging (IM)---from two complementary aspects: defending against unwanted messages and ensuring reliable delivery of wanted messages.;To curtail unwanted messages and protect e-mail and instant messaging users, this dissertation proposes two mechanisms DBSpam and HoneyIM, which can effectively thwart e-mail spam laundering and foil malicious instant message spreading, respectively. DBSpam exploits the distinct characteristics of connection correlation and packet symmetry embedded in the behavior of spam laundering and utilizes a simple statistical method, Sequential Probability Ratio Test, to detect and break spam laundering activities inside a customer network in a timely manner. The experimental results demonstrate that DBSpam is effective in quickly and accurately capturing and suppressing e-mail spam laundering activities and is capable of coping with high speed network traffic. HoneyIM leverages the inherent characteristic of spreading of IM malware and applies the honey-pot technology to the detection of malicious instant messages. More specifically, HoneyIM uses decoy accounts in normal users\u27 contact lists as honey-pots to capture malicious messages sent by IM malware and suppresses the spread of malicious instant messages by performing network-wide blocking. The efficacy of HoneyIM has been validated through both simulations and real experiments.;To improve e-mail reliability, that is, prevent losses of wanted e-mail, this dissertation proposes a collaboration-based autonomous e-mail reputation system called CARE. CARE introduces inter-domain collaboration without central authority or third party and enables each e-mail service provider to independently build its reputation database, including frequently contacted and unacquainted sending domains, based on the local e-mail history and the information exchanged with other collaborating domains. The effectiveness of CARE on improving e-mail reliability has been validated through a number of experiments, including a comparison of two large e-mail log traces from two universities, a real experiment of DNS snooping on more than 36,000 domains, and extensive simulation experiments in a large-scale environment

Stochastic propagation modeling and early detection of malicious mobile code

Epidemic models are commonly used to model the propagation of malicious mobile code like a computer virus or a worm. In this dissertation, we introduce stochastic techniques to describe the propagation behavior of malicious mobile code. We propose a stochastic infection-immunization (INIM) model based on the standard Susceptible-Infected-Removed (SIR) epidemic model, and we get an explicit solution of this model using probability generating function (pgf.). Our experiments simulate the propagation of malicious mobile code with immunization. The simulation results match the theoretical results of the model, which indicates that it is reliable to use INIM model to predict the propagation of malicious mobile code at the early infection stage when immunization factor is considered. In this dissertation, we also propose a control system that could automatically detect and mitigate the propagation of malicious mobile programs at the early infection stage. The detection method is based on the observation that a worm always opens as many connections as possible in order to propagate as fast as possible. To develop the detection algorithm, we extend the traditional statistical process control technique by adding a sliding window. We do the experiment to demonstrate the training process and testing process of a control system using both real and simulation data set. The experiment results show that the control system detects the propagation of malicious mobile code with zero false negative rate and less than 6% false positive rate. Moreover, we introduce risk analysis using Sequential Probability Ratio Test (SPRT) to limit the false positive rate. Examples of risk control using SPTR are presented. Furthermore, we analyze the network behavior using the propagation models we developed to evaluate the effect of the control system in a network environment. The theoretical analysis of the model shows that the propagation of malicious program is reduced when hosts in a network applied the control system. To verify the theoretical result, we also develop the experiment to simulate the propagation process in a network. The experiment results match the mathematical results

Stochastic propagation modeling and early detection of malicious mobile code

Epidemic models are commonly used to model the propagation of malicious mobile code like a computer virus or a worm. In this dissertation, we introduce stochastic techniques to describe the propagation behavior of malicious mobile code. We propose a stochastic infection-immunization (INIM) model based on the standard Susceptible-Infected-Removed (SIR) epidemic model, and we get an explicit solution of this model using probability generating function (pgf.). Our experiments simulate the propagation of malicious mobile code with immunization. The simulation results match the theoretical results of the model, which indicates that it is reliable to use INIM model to predict the propagation of malicious mobile code at the early infection stage when immunization factor is considered. In this dissertation, we also propose a control system that could automatically detect and mitigate the propagation of malicious mobile programs at the early infection stage. The detection method is based on the observation that a worm always opens as many connections as possible in order to propagate as fast as possible. To develop the detection algorithm, we extend the traditional statistical process control technique by adding a sliding window. We do the experiment to demonstrate the training process and testing process of a control system using both real and simulation data set. The experiment results show that the control system detects the propagation of malicious mobile code with zero false negative rate and less than 6% false positive rate. Moreover, we introduce risk analysis using Sequential Probability Ratio Test (SPRT) to limit the false positive rate. Examples of risk control using SPTR are presented. Furthermore, we analyze the network behavior using the propagation models we developed to evaluate the effect of the control system in a network environment. The theoretical analysis of the model shows that the propagation of malicious program is reduced when hosts in a network applied the control system. To verify the theoretical result, we also develop the experiment to simulate the propagation process in a network. The experiment results match the mathematical results

Shadow Honeypots

We present Shadow Honeypots, a novel hybrid architecture that combines the best features of honeypots and anomaly detection. At a high level, we use a variety of anomaly detectors to monitor all traffic to a protected network or service. Traffic that is considered anomalous is processed by a "shadow honeypot" to determine the accuracy of the anomaly prediction. The shadow is an instance of the protected software that shares all internal state with a regular ("production") instance of the application, and is instrumented to detect potential attacks. Attacks against the shadow are caught, and any incurred state changes are discarded. Legitimate traffic that was misclassified will be validated by the shadow and will be handled correctly by the system transparently to the end user. The outcome of processing a request by the shadow is used to filter future attack instances and could be used to update the anomaly detector. Our architecture allows system designers to fine-tune systems for performance, since false positives will be filtered by the shadow. We demonstrate the feasibility of our approach in a proof-of-concept implementation of the Shadow Honeypot architecture for the Apache web server and the Mozilla Firefox browser. We show that despite a considerable overhead in the instrumentation of the shadow honeypot (up to 20% for Apache), the overall impact on the system is diminished by the ability to minimize the rate of false-positives

An Interactive Relaxation Approach for Anomaly Detection and Preventive Measures in Computer Networks

It is proposed to develop a framework of detecting and analyzing small and widespread changes in specific dynamic characteristics of several nodes. The characteristics are locally measured at each node in a large network of computers and analyzed using a computational paradigm known as the Relaxation technique. The goal is to be able to detect the onset of a worm or virus as it originates, spreads-out, attacks and disables the entire network. Currently, selective disabling of one or more features across an entire subnet, e.g. firewalls, provides limited security and keeps us from designing high performance net-centric systems. The most desirable response is to surgically disable one or more nodes, or to isolate one or more subnets.The proposed research seeks to model virus/worm propagation as a spatio-temporal process. Such models have been successfully applied in heat-flow and evidence or gestalt driven perception of images among others. In particular, we develop an iterative technique driven by the self-assessed dynamic status of each node in a network. The status of each node will be updated incrementally in concurrence with its connected neighbors to enable timely identification of compromised nodes and subnets. Several key insights used in image analysis of line-diagrams, through an iterative and relaxation-driven node labeling method, are explored to help develop this new framework