An Innovative Signature Detection System for Polymorphic and Monomorphic Internet Worms Detection and Containment

Most current anti-worm systems and intrusion-detection systems use signature-based technology instead of anomaly-based technology. Signature-based technology can only detect known attacks with identified signatures. Existing anti-worm systems cannot detect unknown Internet scanning worms automatically because these systems do not depend upon worm behaviour but upon the worm’s signature. Most detection algorithms used in current detection systems target only monomorphic worm payloads and offer no defence against polymorphic worms, which changes the payload dynamically. Anomaly detection systems can detect unknown worms but usually suffer from a high false alarm rate. Detecting unknown worms is challenging, and the worm defence must be automated because worms spread quickly and can flood the Internet in a short time. This research proposes an accurate, robust and fast technique to detect and contain Internet worms (monomorphic and polymorphic). The detection technique uses specific failure connection statuses on specific protocols such as UDP, TCP, ICMP, TCP slow scanning and stealth scanning as characteristics of the worms. Whereas the containment utilizes flags and labels of the segment header and the source and destination ports to generate the traffic signature of the worms. Experiments using eight different worms (monomorphic and polymorphic) in a testbed environment were conducted to verify the performance of the proposed technique. The experiment results showed that the proposed technique could detect stealth scanning up to 30 times faster than the technique proposed by another researcher and had no false-positive alarms for all scanning detection cases. The experiments showed the proposed technique was capable of containing the worm because of the traffic signature’s uniqueness

Detecting Malicious Software By Dynamicexecution

Traditional way to detect malicious software is based on signature matching. However, signature matching only detects known malicious software. In order to detect unknown malicious software, it is necessary to analyze the software for its impact on the system when the software is executed. In one approach, the software code can be statically analyzed for any malicious patterns. Another approach is to execute the program and determine the nature of the program dynamically. Since the execution of malicious code may have negative impact on the system, the code must be executed in a controlled environment. For that purpose, we have developed a sandbox to protect the system. Potential malicious behavior is intercepted by hooking Win32 system calls. Using the developed sandbox, we detect unknown virus using dynamic instruction sequences mining techniques. By collecting runtime instruction sequences in basic blocks, we extract instruction sequence patterns based on instruction associations. We build classification models with these patterns. By applying this classification model, we predict the nature of an unknown program. We compare our approach with several other approaches such as simple heuristics, NGram and static instruction sequences. We have also developed a method to identify a family of malicious software utilizing the system call trace. We construct a structural system call diagram from captured dynamic system call traces. We generate smart system call signature using profile hidden Markov model (PHMM) based on modularized system call block. Smart system call signature weakly identifies a family of malicious software

Data Mining Methods For Malware Detection

This research investigates the use of data mining methods for malware (malicious programs) detection and proposed a framework as an alternative to the traditional signature detection methods. The traditional approaches using signatures to detect malicious programs fails for the new and unknown malwares case, where signatures are not available. We present a data mining framework to detect malicious programs. We collected, analyzed and processed several thousand malicious and clean programs to find out the best features and build models that can classify a given program into a malware or a clean class. Our research is closely related to information retrieval and classification techniques and borrows a number of ideas from the field. We used a vector space model to represent the programs in our collection. Our data mining framework includes two separate and distinct classes of experiments. The first are the supervised learning experiments that used a dataset, consisting of several thousand malicious and clean program samples to train, validate and test, an array of classifiers. In the second class of experiments, we proposed using sequential association analysis for feature selection and automatic signature extraction. With our experiments, we were able to achieve as high as 98.4% detection rate and as low as 1.9% false positive rate on novel malwares

An examination of the Asus WL-HDD 2.5 as a nepenthes malware collector

The Linksys WRT54g has been used as a host for network forensics tools for instance Snort for a long period of time. Whilst large corporations are already utilising network forensic tools, this paper demonstrates that it is quite feasible for a non-security specialist to track and capture malicious network traffic. This paper introduces the Asus Wireless Hard disk as a replacement for the popular Linksys WRT54g. Firstly, the Linksys router will be introduced detailing some of the research that was undertaken on the device over the years amongst the security community. It then briefly discusses malicious software and the impact this may have for a home user. The paper then outlines the trivial steps in setting up Nepenthes 0.1.7 (a malware collector) for the Asus WL-HDD 2.5 according to the Nepenthes and tests the feasibility of running the malware collector on the selected device. The paper then concludes on discussing the limitations of the device when attempting to execute Nepenthes

MFIRE-2: A Multi Agent System for Flow-based Intrusion Detection Using Stochastic Search

Detecting attacks targeted against military and commercial computer networks is a crucial element in the domain of cyberwarfare. The traditional method of signature-based intrusion detection is a primary mechanism to alert administrators to malicious activity. However, signature-based methods are not capable of detecting new or novel attacks. This research continues the development of a novel simulated, multiagent, flow-based intrusion detection system called MFIRE. Agents in the network are trained to recognize common attacks, and they share data with other agents to improve the overall effectiveness of the system. A Support Vector Machine (SVM) is the primary classifier with which agents determine an attack is occurring. Agents are prompted to move to different locations within the network to find better vantage points, and two methods for achieving this are developed. One uses a centralized reputation-based model, and the other uses a decentralized model optimized with stochastic search. The latter is tested for basic functionality. The reputation model is extensively tested in two configurations and results show that it is significantly superior to a system with non-moving agents. The resulting system, MFIRE-2, demonstrates exciting new network defense capabilities, and should be considered for implementation in future cyberwarfare applications

A Multi Agent System for Flow-Based Intrusion Detection Using Reputation and Evolutionary Computation

The rising sophistication of cyber threats as well as the improvement of physical computer network properties present increasing challenges to contemporary Intrusion Detection (ID) techniques. To respond to these challenges, a multi agent system (MAS) coupled with flow-based ID techniques may effectively complement traditional ID systems. This paper develops: 1) a scalable software architecture for a new, self-organized, multi agent, flow-based ID system; and 2) a network simulation environment suitable for evaluating implementations of this MAS architecture and for other research purposes. Self-organization is achieved via 1) a reputation system that influences agent mobility in the search for effective vantage points in the network; and 2) multi objective evolutionary algorithms that seek effective operational parameter values. This paper illustrates, through quantitative and qualitative evaluation, 1) the conditions for which the reputation system provides a significant benefit; and 2) essential functionality of a complex network simulation environment supporting a broad range of malicious activity scenarios. These results establish an optimistic outlook for further research in flow-based multi agent systems for ID in computer networks

Improving Existing Static and Dynamic Malware Detection Techniques with Instruction-level Behavior

My Ph.D. focuses on detecting malware by leveraging the information obtained at an instruction-level. Instruction-level information is obtained by looking at the instructions or disassembly that make up an executable. My initial work focused on using a dynamic binary instrumentation (DBI) tool. A DBI tool enables the study of instruction-level behavior while the malware is executing, which I show proves to be valuable in detecting malware. To expand on my work with dynamic instruction-level information, I integrated it with machine learning to increase the scalability and robustness of my detection tool. To further increase the scalability of the dynamic detection of malware, I created a two stage static-dynamic malware detection scheme aimed at achieving the accuracy of a fully-dynamic detection scheme without the high computational resources and time required. Lastly, I show the improvement of static analysis-based detection of malware by automatically generated machine learning features based on opcode sequences with the help of convolutional neural networks. The first part of my research focused on obfuscated malware. Obfuscation is the process in which malware tries to hide itself from static analysis and trick disassemblers. I found that by using a DBI tool, I was able to not only detect obfuscation, but detect the differences in how it occurred in malware versus goodware. Through dynamic program-level analysis, I was able to detect specific obfuscations and use the varying methods in which it was used by programs to differentiate malware and goodware. I found that by using the mere presence of obfuscation as a method of detecting malware, I was able to detect previously undetected malware. I then focused on using my knowledge of dynamic program-level features to build a highly accurate machine learning-based malware detection tool. Machine learning is useful in malware detection because it can process a large amount of data to determine meaningful relationships to distinguish malware from benign programs. Through the integration of machine learning, I was able to expand my obfuscation detection schemes to address a broader class of malware, which ultimately led to a malware detection tool that can detect 98.45% of malware with a 1% false positive rate. Understanding the pitfalls of dynamic analysis of malware, I focused on creating a more efficient method of detecting malware. Malware detection comes in three methods: static analysis, dynamic analysis, and hybrids. Static analysis is fast and effective for detecting previously seen malware where as dynamic analysis can be more accurate and robust against zero-day or polymorphic malware, but at the cost of a high computational load. Most modern defenses today use a hybrid approach, which uses both static and dynamic analysis, but are suboptimal. I created a two-phase malware detection tool that approaches the accuracy of the dynamic-only system with only a small fraction of its computational cost, while maintaining a real-time malware detection timeliness similar to a static-only system, thus achieving the best of both approaches. Lastly, my Ph.D. focused on reducing the need for manual feature generation by utilizing Convolutional Neural Networks (CNNs) to automatically generate feature vectors from raw input data. My work shows that using a raw sequence of opcode sequences from static disassembly with a CNN model can automatically produce feature vectors that are useful for detecting malware. Because this process is automated, it presents as a scalable method of consistently producing useful features without human intervention or labor that can be used to detect malware

A Multi Agent System for Flow-Based Intrusion Detection

The detection and elimination of threats to cyber security is essential for system functionality, protection of valuable information, and preventing costly destruction of assets. This thesis presents a Mobile Multi-Agent Flow-Based IDS called MFIREv3 that provides network anomaly detection of intrusions and automated defense. This version of the MFIRE system includes the development and testing of a Multi-Objective Evolutionary Algorithm (MOEA) for feature selection that provides agents with the optimal set of features for classifying the state of the network. Feature selection provides separable data points for the selected attacks: Worm, Distributed Denial of Service, Man-in-the-Middle, Scan, and Trojan. This investigation develops three techniques of self-organization for multiple distributed agents in an intrusion detection system: Reputation, Stochastic, and Maximum Cover. These three movement models are tested for effectiveness in locating good agent vantage points within the network to classify the state of the network. MFIREv3 also introduces the design of defensive measures to limit the effects of network attacks. Defensive measures included in this research are rate-limiting and elimination of infected nodes. The results of this research provide an optimistic outlook for flow-based multi-agent systems for cyber security. The impact of this research illustrates how feature selection in cooperation with movement models for multi agent systems provides excellent attack detection and classification

Intelligent Malware Detection Using File-to-file Relations and Enhancing its Security against Adversarial Attacks

With computing devices and the Internet being indispensable in people\u27s everyday life, malware has posed serious threats to their security, making its detection of utmost concern. To protect legitimate users from the evolving malware attacks, machine learning-based systems have been successfully deployed and offer unparalleled flexibility in automatic malware detection. In most of these systems, resting on the analysis of different content-based features either statically or dynamically extracted from the file samples, various kinds of classifiers are constructed to detect malware. However, besides content-based features, file-to-file relations, such as file co-existence, can provide valuable information in malware detection and make evasion harder. To better understand the properties of file-to-file relations, we construct the file co-existence graph. Resting on the constructed graph, we investigate the semantic relatedness among files, and leverage graph inference, active learning and graph representation learning for malware detection. Comprehensive experimental results on the real sample collections from Comodo Cloud Security Center demonstrate the effectiveness of our proposed learning paradigms. As machine learning-based detection systems become more widely deployed, the incentive for defeating them increases. Therefore, we go further insight into the arms race between adversarial malware attack and defense, and aim to enhance the security of machine learning-based malware detection systems. In particular, we first explore the adversarial attacks under different scenarios (i.e., different levels of knowledge the attackers might have about the targeted learning system), and define a general attack strategy to thoroughly assess the adversarial behaviors. Then, considering different skills and capabilities of the attackers, we propose the corresponding secure-learning paradigms to counter the adversarial attacks and enhance the security of the learning systems while not compromising the detection accuracy. We conduct a series of comprehensive experimental studies based on the real sample collections from Comodo Cloud Security Center and the promising results demonstrate the effectiveness of our proposed secure-learning models, which can be readily applied to other detection tasks

Diversity management in intrusion tolerant systems

Tese de mestrado em Informática, apresentada à Universidade de Lisboa, através da Faculdade de Ciências, 2011Uma aplicação importante dos protocolos de tolerância a faltas arbitrárias (ou Bizantinas) é a construção de sistemas tolerantes a intrusões, que são capazes de funcionar correctamente mesmo que alguns dos seus componentes sejam comprometidos. Estes sistemas são concretizados através da replicação de componentes e da utilização de protocolos capazes de tolerar faltas arbitrárias, quer na rede como num subconjunto de réplicas. Os protocolos garantem um comportamento correcto ainda que exista uma minoria (normalmente menor do que um terço) de componentes controlados por um adversário malicioso. Para cumprir esta condição os componentes do sistema têm de apresentar independência nas falhas. No entanto, quando estamos no contexto da segurança de sistemas, temos de admitir a possibilidade de ocorrerem ataques simultâneos contra várias réplicas. Se os vários componentes tiverem as mesmas vulnerabilidades, então podem ser comprometidos com um só ataque, o que destrói o propósito de se construir sistemas tolerantesa intrusões. Com o objectivo de reduzir a probabilidade de existirem vulnerabilidades comuns pretendemos utilizar diversidade: cada componente usa software distinto que fornece as mesmas funcionalidades, com a expectativa de que as diferenças vão reduzir o número de vulnerabilidades semelhantes. Reconhecemos também que isoladamente a tolerância a faltas arbitrárias tem algumas limitações uma vez que consideramos faltas maliciosas: uma das limitações mais importantes é que dado tempo suficiente o adversário pode comprometer f + 1 réplicas, e então violar a hipótese de que no máximo f componentes podem sofrer uma falta, levando os recursos do sistema à exaustão. Uma forma de lidar com esta limitação consiste em renovar periodicamente as réplicas, uma técnica denominada por Recuperação Proactiva. O propósito das recuperações é limpar o estado do sistema, reiniciando a replica com código disponível em armazenamento apenas com permissões de leitura (ex: CD-ROM) e validar/obter o estado de outro componente (que seja correcto). Num sistema tolerante a intrusões com recuperação proactiva o intervalo temporal que o adversário tem para comprometer f + 1 réplicas passa a ser uma pequena janela de vulnerabilidade, que compreende o tempo de recuperação do sistema todo. Apesar dos benefícios que as recuperações periódicas oferecem em termos de fiabilidade persiste a seguinte dificuldade: as vulnerabilidades exploradas nas execuções anteriores da replica podem ainda ser exploradas depois da recuperação. Esta limitação permite facilmente a um atacante criar um script que automaticamente compromete novamente a replica logo a seguir à sua recuperação pois as vulnerabilidades não são apagadas mas sim a falta (i.e., o estado incorrecto no componente devido à intrusão). Com o objectivo de melhorar o sistema introduzimos diversidade nas recuperações, mais precisamente em componentes off-the-shelf (OTS). Hoje em dia praticamente todo o software desenvolvido é baseado neste tipo de componentes, como por exemplo sistemas operativos (SO) e gestores de bases de dados. Isto deve-se principalmente à complexidade do desenvolvimento destes componentes em conjugação com os benefícios relacionados com o baixo custo, a instalação rápida e a variedade de opções disponíveis. No entanto, a maior parte dos componentes OTS não foram desenhados com segurança como prioridade, o que significa que em todos eles existem vulnerabilidades que podem ser maliciosamente exploradas. Por vezes, sistemas supostamente seguros são comprometidos através de uma componente critica na sua infraestrutura. Por outro lado, dada a quantidade de oferta das componentes OTS, utilizar diversidade nestes componentes é menos complexo e tem um menor custo do que desenvolver várias componentes de software diferentes. Um bom exemplo disto é o caso dos SO: as organizações na verdade preferem um sistema operativo OTS do que construir o seu próprio SO. Dada a variedade de sistemas operativos disponíveis e a criticidade do papel desempenhado por estes em qualquer computador, a diversidade ao nível dos SO pode ser uma forma razoável de garantir segurança contra vulnerabilidades comuns com um baixo custo adicional. O foco nas vulnerabilidades comuns é um aspecto importante deste trabalho. Visto que a tolerância intrusões ´e aplicada em sistemas críticos, ´e seguro afirmar que no sistema operativo vai ser assegurada a máxima segurança, aplicando todos os patches disponíveis. No entanto, mesmo com sistemas actualizados, o sistema pode ser comprometido através de vulnerabilidades que ainda não foram descobertas pelos programadores (vulnerabilidades de dia zero), visto que os patches aparecem normalmente depois da vulnerabilidade ser anunciada. Se uma vulnerabilidade de dia zero afectar o sistema existe uma janela de oportunidade para o atacante causar uma intrusão. A questão principal que tratamos na primeira parte desta tese é: Quais são os ganhos de se aplicar diversidade de SO num sistema tolerante a intrusões replicado? Para responder a esta questão, recolhemos e selecionámos dados sobre vulnerabilidades do NIST National Vulnerability Database (NVD) entre 1994 e 2010 para 11 sistemas operativos. Os dados do NVD relativamente aos SO são consideráveis, o que nos permite tirar algumas conclusões. Cada vulnerabilidade presente no NVD contém (entre outras coisas) informação sobre que produtos são afectados pela vulnerabilidade. Recolhemos estes dados e verificámos quantas vulnerabilidades afectam mais do que um sistema operativo. Constatámos que este número é relativamente pequeno para a maior parte de de sistemas operativos. Este estudo foi depois estendido a um número maior de SO, com conclusões semelhantes para esses conjuntos. Estes resultados sugerem que existem ganhos de segurança que podem ser alcançados recorrendo à utilização de sistemas operativos diferentes num sistema replicado. Como nota de cautela, não pretendemos afirmar que estes resultados são uma prova final sobre a ausência de vulnerabilidades comuns (embora sejam bastante promissores). Um dos principais problemas encontrados é que os relatórios focam-se nas vulnerabilidades e não em quantas intrusões ou exploits ocorreram para cada vulnerabilidade; isto faz com que a avaliação, em termos de segurança, seja mais difícil. A segunda parte da tese propõe uma arquitectura que explora a diversidade disponível nos sistemas operativos juntamente com mecanismos de recuperação proactiva. O objectivo principal é mudar a configuração das réplicas de forma a alterar o conjunto de vulnerabilidades após uma recuperação. Desenvolvemos também um algoritmo que seleciona entre os candidatos o melhor sistema operativo para ser usado numa recuperação, assegurando o maior nível de diversidade possível entre as réplicas que se encontram em execução.One of the key benefits of using intrusion-tolerant systems is the possibility of ensuring correct behavior in the presence of attacks and intrusions. These security gains are directly dependent on the components exhibiting failure diversity. To what extent failure diversity is observed in practical deployment depends on how diverse are the components that constitute the system. In this thesis we present a study with operating systems (OS) vulnerability reports from the NIST National Vulnerability Database. We have analyzed the vulnerabilities of 11 different OS over a period of roughly 15 years, to check how many of these vulnerabilities occur in more than one OS. We found this number to be low for several combinations of OS. Hence, our analysis provides a strong indication that building a system with diverse OS may be a useful technique to improve its intrusion tolerance capabilities. However, even with diversity the attacker eventually will find vulnerabilities in all OS replicas. To mitigate/eliminate this problem we introduce diverse proactive recovery on the replicas. Proactive recovery is a technique that periodically rejuvenates the components of a replicated system. When used in the context of intrusiontolerant systems, in which faulty replicas may be under control of some malicious user, it allows the removal of intrusions from the compromised replicas. We propose that after each recovery a replica starts to run a different software. The selection of the new replica configuration is a non-trivial problem, as we will explain, since we would like to maximize the diversity of the system under the constraint of the available configurations