Intrusion detection based on behavioral rules for the bytes of the headers of the data units in IP networks

Nowadays, communications through computer networks are of utmost importance for the normal functioning of organizations, worldwide transactions and content delivery. These networks are threatened by all kinds of attacks, leading to traffic anomalies that will eventually disrupt the normal behaviour of the networks, exploring specific breaches on a system component or exhausting network resources. Automatic detection of these network anomalies comprises one of the most important resources for network administration, and Intrusion Detection Systems(IDSs) are amongst the systems responsible for this automatic detection. This dissertation starts from the assumption that it is possible to use machine learning to, consistently and automatically, produce rules for an intrusion detector based on statistics for the first 64 bytes of the headers of Internet Protocol (IP) packets. The survey on the state of the art on related works and currently available IDSs shows that the specific approach taken here is worth to be explored. The decision tree learning algorithm known as C4.5 is identified as a suitable means to produce the aforementioned rules, due to the similarity between their syntax and the tree structure. Several rules are then devised using the ML approach for several attacks. The attacks were the same used in a previous work, in which the rules were devised manually. Both rule sets are then compared to show that, in fact, it is possible to construct rules using the approach taken herein, and that the rules created resorting to the C4.5 algorithm are superior to the ones devised after thorough human analysis of several statistics calculated for the bytes of the headers of the packets. To compare them, each rule set was used to detect intrusions in third party traces containing attacks and in live traffic during simulation of attacks. Most of the attacks producing noticeable impact on the headers were detected by both rule sets, but the results for the third party traces were better in the case of the ML devised rules, providing a clear evidence for the aforementioned assumptions.Hoje em dia, as comunicações através de redes informáticas são da maior importância para o normal funcionamento das organizações, transações mundiais e entrega de conteúdos. Essas redes são ameaçadas por todo o tipo de ataques, levando a anomalias no tráfego, que eventualmente vão corromper o normal funcionamento da rede, explorando falhas específicas num componente de um sistema, ou esgotando os recursos de rede. A deteção automática dessas anomalias de rede é um dos recursos mais importantes para os administradores de rede, e os Sistemas de Deteção de Intrusões estão entre os sistemas responsáveis por essa deteção. Esta dissertação tem como ponto de partida, a assunção que é possível usar mecanismos de aprendizagem automática para produzir, de modo consistente e automático, regras para a deteção de intrusões, baseadas em estatísticas dos primeiros 64 bytes dos cabeçalhos dos pacotes IP. O estudo sobre o estado da arte em trabalhos da área, e em sistemas de deteção atualmente disponíveis, mostrou que o método usado nesta dissertação merece ser estudado. O algoritmo de árvores de decisão C4.5 foi identificado como um meio apropriado para produzir as regras já referidas, devido à semelhança entre a sintaxe das mesmas e a estrutura em árvore deste algoritmo. Várias regras foram depois produzidas para vários tipos de ataque, usando a abordagem por aprendizagem automática. Os ataques tomados em consideração foram os mesmos que foram utilizados num trabalho anterior, em que a regras foram concebidas manualmente. Ambos os conjuntos de regras são depois comparados, para mostrar que, de facto, é possível construir regras através da abordagem utilizada nesta dissertação, e que as regras criadas através do algoritmo C4.5 são superiores às que foram criadas através de análise humana das várias estatísticas calculadas para os bytes dos cabeçalhos dos pacotes. Para as comparar, cada conjunto de regras foi utilizado para detetar intrusões em registos de tráfego disponíveis na Internet contendo ataques e em tráfego em tempo real, durante a simulação de ataques. A maioria dos ataques que produz um forte impacto nos cabeçalhos dos pacotes foi detetado por ambos os conjuntos, mas os resultados com os registos retirados da Internet foram melhores para as regras produzidas por aprendizagem automática, dando uma prova clara para o que foi previamente assumido

Security risk assessment in cloud computing domains

Cyber security is one of the primary concerns persistent across any computing platform. While addressing the apprehensions about security risks, an infinite amount of resources cannot be invested in mitigation measures since organizations operate under budgetary constraints. Therefore the task of performing security risk assessment is imperative to designing optimal mitigation measures, as it provides insight about the strengths and weaknesses of different assets affiliated to a computing platform. The objective of the research presented in this dissertation is to improve upon existing risk assessment frameworks and guidelines associated to different key assets of Cloud computing domains - infrastructure, applications, and users. The dissertation presents various informal approaches of performing security risk assessment which will help to identify the security risks confronted by the aforementioned assets, and utilize the results to carry out the required cost-benefit tradeoff analyses. This will be beneficial to organizations by aiding them in better comprehending the security risks their assets are exposed to and thereafter secure them by designing cost-optimal mitigation measures --Abstract, page iv

A monitoring and threat detection system using stream processing as a virtual function for big data

The late detection of security threats causes a significant increase in the risk of irreparable damages, disabling any defense attempt. As a consequence, fast realtime threat detection is mandatory for security guarantees. In addition, Network Function Virtualization (NFV) provides new opportunities for efficient and low-cost security solutions. We propose a fast and efficient threat detection system based on stream processing and machine learning algorithms. The main contributions of this work are i) a novel monitoring threat detection system based on stream processing; ii) two datasets, first a dataset of synthetic security data containing both legitimate and malicious traffic, and the second, a week of real traffic of a telecommunications operator in Rio de Janeiro, Brazil; iii) a data pre-processing algorithm, a normalizing algorithm and an algorithm for fast feature selection based on the correlation between variables; iv) a virtualized network function in an open-source platform for providing a real-time threat detection service; v) near-optimal placement of sensors through a proposed heuristic for strategically positioning sensors in the network infrastructure, with a minimum number of sensors; and, finally, vi) a greedy algorithm that allocates on demand a sequence of virtual network functions.A detecção tardia de ameaças de segurança causa um significante aumento no risco de danos irreparáveis, impossibilitando qualquer tentativa de defesa. Como consequência, a detecção rápida de ameaças em tempo real é essencial para a administração de segurança. Além disso, A tecnologia de virtualização de funções de rede (Network Function Virtualization - NFV) oferece novas oportunidades para soluções de segurança eficazes e de baixo custo. Propomos um sistema de detecção de ameaças rápido e eficiente, baseado em algoritmos de processamento de fluxo e de aprendizado de máquina. As principais contribuições deste trabalho são: i) um novo sistema de monitoramento e detecção de ameaças baseado no processamento de fluxo; ii) dois conjuntos de dados, o primeiro ´e um conjunto de dados sintético de segurança contendo tráfego suspeito e malicioso, e o segundo corresponde a uma semana de tráfego real de um operador de telecomunicações no Rio de Janeiro, Brasil; iii) um algoritmo de pré-processamento de dados composto por um algoritmo de normalização e um algoritmo para seleção rápida de características com base na correlação entre variáveis; iv) uma função de rede virtualizada em uma plataforma de código aberto para fornecer um serviço de detecção de ameaças em tempo real; v) posicionamento quase perfeito de sensores através de uma heurística proposta para posicionamento estratégico de sensores na infraestrutura de rede, com um número mínimo de sensores; e, finalmente, vi) um algoritmo guloso que aloca sob demanda uma sequencia de funções de rede virtual

Study of stochastic and machine learning tecniques for anomaly-based Web atack detection

Mención Internacional en el título de doctorWeb applications are exposed to different threats and it is necessary to protect them. Intrusion Detection Systems (IDSs) are a solution external to the web application that do not require the modification of the application’s code in order to protect it. These systems are located in the network, monitoring events and searching for signs of anomalies or threats that can compromise the security of the information systems. IDSs have been applied to traffic analysis of different protocols, such as TCP, FTP or HTTP. Web Application Firewalls (WAFs) are special cases of IDSs that are specialized in analyzing HTTP traffic with the aim of safeguarding web applications. The increase in the amount of data traveling through the Internet and the growing sophistication of the attacks, make necessary protection mechanisms that are both effective and efficient. This thesis proposes three anomaly-based WAFs with the characteristics of being high-speed, reaching high detection results and having a simple design. The anomaly-based approach defines the normal behavior of web application. Actions that deviate from it are considered anomalous. The proposed WAFs work at the application layer analyzing the payload of HTTP requests. These systems are designed with different detection algorithms in order to compare their results and performance. Two of the systems proposed are based on stochastic techniques: one of them is based on statistical techniques and the other one in Markov chains. The third WAF presented in this thesis is ML-based. Machine Learning (ML) deals with constructing computer programs that automatically learn with experience and can be very helpful in dealing with big amounts of data. Concretely, this third WAF is based on decision trees given their proved effectiveness in intrusion detection. In particular, four algorithms are employed: C4.5, CART, Random Tree and Random Forest. Typically, two phases are distinguished in IDSs: preprocessing and processing. In the case of stochastic systems, preprocessing includes feature extraction. The processing phase consists in training the system in order to learn the normal behavior and later testing how well it classifies the incoming requests as either normal or anomalous. The detection models of the systems are implemented either with statistical techniques or with Markov chains, depending on the system considered. For the system based on decision trees, the preprocessing phase comprises feature extraction as well as feature selection. These two phases are optimized. On the one hand, new feature extraction methods are proposed. They combine features extracted by means of expert knowledge and n-grams, and have the capacity of improving the detection results of both techniques separately. For feature selection, the Generic Feature Selection GeFS measure has been used, which has been proven to be very effective in reducing the number of redundant and irrelevant features. Additionally, for the three systems, a study for establishing the minimum number of requests required to train them in order to achieve a certain detection result has been performed. Reducing the number of training requests can greatly help in the optimization of the resource consumption of WAFs as well as on the data gathering process. Besides designing and implementing the systems, evaluating them is an essential step. For that purpose, a dataset is necessary. Unfortunately, finding labeled and adequate datasets is not an easy task. In fact, the study of the most popular datasets in the intrusion detection field reveals that most of them do not satisfy the requirements for evaluating WAFs. In order to tackle this situation, this thesis proposes the new CSIC dataset, that satisfies the necessary conditions to satisfactorily evaluate WAFs. The proposed systems have been experimentally evaluated. For that, the proposed CSIC dataset and the existing ECML/PKDD dataset have been used. The three presented systems have been compared in terms of their detection results, processing time and number of training requests used. For this comparison, the CSIC dataset has been used. In summary, this thesis proposes three WAFs based on stochastic and ML techniques. Additionally, the systems are compared, what allows to determine which system is the most appropriate for each scenario.Las aplicaciones web están expuestas a diferentes amenazas y es necesario protegerlas. Los sistemas de detección de intrusiones (IDSs del inglés Intrusion Detection Systems) son una solución externa a la aplicación web que no requiere la modificación del código de la aplicación para protegerla. Estos sistemas se sitúan en la red, monitorizando los eventos y buscando señales de anomalías o amenazas que puedan comprometer la seguridad de los sistemas de información. Los IDSs se han aplicado al análisis de tráfico de varios protocolos, tales como TCP, FTP o HTTP. Los Cortafuegos de Aplicaciones Web (WAFs del inglés Web Application Firewall) son un caso especial de los IDSs que están especializados en analizar tráfico HTTP con el objetivo de salvaguardar las aplicaciones web. El incremento en la cantidad de datos circulando por Internet y la creciente sofisticación de los ataques hace necesario contar con mecanismos de protección que sean efectivos y eficientes. Esta tesis propone tres WAFs basados en anomalías que tienen las características de ser de alta velocidad, alcanzar altos resultados de detección y contar con un diseño sencillo. El enfoque basado en anomalías define el comportamiento normal de la aplicación, de modo que las acciones que se desvían del mismo se consideran anómalas. Los WAFs diseñados trabajan en la capa de aplicación y analizan el contenido de las peticiones HTTP. Estos sistemas están diseñados con diferentes algoritmos de detección para comparar sus resultados y rendimiento. Dos de los sistemas propuestos están basados en técnicas estocásticas: una de ellas está basada en técnicas estadísticas y la otra en cadenas de Markov. El tercer WAF presentado en esta tesis está basado en aprendizaje automático. El aprendizaje automático (ML del inglés Machine Learning) se ocupa de cómo construir programas informáticos que aprenden automáticamente con la experiencia y puede ser muy útil cuando se trabaja con grandes cantidades de datos. En concreto, este tercer WAF está basado en árboles de decisión, dada su probada efectividad en la detección de intrusiones. En particular, se han empleado cuatro algoritmos: C4.5, CART, Random Tree y Random Forest. Típicamente se distinguen dos fases en los IDSs: preprocesamiento y procesamiento. En el caso de los sistemas estocásticos, en la fase de preprocesamiento se realiza la extracción de características. El procesamiento consiste en el entrenamiento del sistema para que aprenda el comportamiento normal y más tarde se comprueba cuán bien el sistema es capaz de clasificar las peticiones entrantes como normales o anómalas. Los modelos de detección de los sistemas están implementados bien con técnicas estadísticas o bien con cadenas de Markov, dependiendo del sistema considerado. Para el sistema basado en árboles de decisión la fase de preprocesamiento comprende tanto la extracción de características como la selección de características. Estas dos fases se han optimizado. Por un lado, se proponen nuevos métodos de extracción de características. Éstos combinan características extraídas por medio de conocimiento experto y n-gramas y tienen la capacidad de mejorar los resultados de detección de ambas técnicas por separado. Para la selección de características, se ha utilizado la medida GeFS (del inglés Generic Feature Selection), la cual ha probado ser muy efectiva en la reducción del número de características redundantes e irrelevantes. Además, para los tres sistemas, se ha realizado un estudio para establecer el mínimo número de peticiones necesarias para entrenarlos y obtener un cierto resultado. Reducir el número de peticiones de entrenamiento puede ayudar en gran medida a la optimización del consumo de recursos de los WAFs así como en el proceso de adquisición de datos. Además de diseñar e implementar los sistemas, la tarea de evaluarlos es esencial. Para este propósito es necesario un conjunto de datos. Desafortunadamente, encontrar conjuntos de datos etiquetados y adecuados no es una tarea fácil. De hecho, el estudio de los conjuntos de datos más utilizados en el campo de la detección de intrusiones revela que la mayoría de ellos no cumple los requisitos para evaluar WAFs. Para enfrentar esta situación, esta tesis presenta un nuevo conjunto de datos llamado CSIC, que satisface las condiciones necesarias para evaluar WAFs satisfactoriamente. Los sistemas propuestos se han evaluado experimentalmente. Para ello, se ha utilizado el conjunto de datos propuesto (CSIC) y otro existente llamado ECML/PKDD. Los tres sistemas presentados se han comparado con respecto a sus resultados de detección, tiempo de procesamiento y número de peticiones de entrenamiento utilizadas. Para esta comparación se ha utilizado el conjunto de datos CSIC. En resumen, esta tesis propone tres WAFs basados en técnicas estocásticas y de ML. Además, se han comparado estos sistemas entre sí, lo que permite determinar qué sistema es el más adecuado para cada escenario.Este trabajo ha sido realizado en el marco de las becas predoctorales de la Junta de Amplicación de Estudios (JAE) de la Agencia Estatal Consejo Superior de Investigaciones Científicas (CSIC).Programa Oficial de Doctorado en Ciencia y Tecnología InformáticaPresidente: Luis Hernández Encinas.- Secretario: Juan Manuel Estévez Tapiador.- Vocal: Georg Carl

HUC-HISF: A Hybrid Intelligent Security Framework for Human-centric Ubiquitous Computing

制度:新 ; 報告番号:乙2336号 ; 学位の種類:博士(人間科学) ; 授与年月日:2012/1/18 ; 早大学位記番号:新584

Attacks against intrusion detection networks: evasion, reverse engineering and optimal countermeasures

Intrusion Detection Networks (IDNs) constitute a primary element in current cyberdefense systems. IDNs are composed of different nodes distributed among a network infrastructure, performing functions such as local detection --mostly by Intrusion Detection Systems (IDS) --, information sharing with other nodes in the IDN, and aggregation and correlation of data from different sources. Overall, they are able to detect distributed attacks taking place at large scale or in different parts of the network simultaneously. IDNs have become themselves target of advanced cyberattacks aimed at bypassing the security barrier they offer and thus gaining control of the protected system. In order to guarantee the security and privacy of the systems being protected and the IDN itself, it is required to design resilient architectures for IDNs capable of maintaining a minimum level of functionality even when certain IDN nodes are bypassed, compromised, or rendered unusable. Research in this field has traditionally focused on designing robust detection algorithms for IDS. However, almost no attention has been paid to analyzing the security of the overall IDN and designing robust architectures for them. This Thesis provides various contributions in the research of resilient IDNs grouped into two main blocks. The first two contributions analyze the security of current proposals for IDS nodes against specific attacks, while the third and fourth contributions provide mechanisms to design IDN architectures that remain resilient in the presence of adversaries. In the first contribution, we propose evasion and reverse engineering attacks to anomaly detectors that use classification algorithms at the core of the detection engine. These algorithms have been widely studied in the anomaly detection field, as they generally are claimed to be both effective and efficient. However, such anomaly detectors do not consider potential behaviors incurred by adversaries to decrease the effectiveness and efficiency of the detection process. We demonstrate that using well-known classification algorithms for intrusion detection is vulnerable to reverse engineering and evasion attacks, which makes these algorithms inappropriate for real systems. The second contribution discusses the security of randomization as a countermeasure to evasion attacks against anomaly detectors. Recent works have proposed the use of secret (random) information to hide the detection surface, thus making evasion harder for an adversary. We propose a reverse engineering attack using a query-response analysis showing that randomization does not provide such security. We demonstrate our attack on Anagram, a popular application-layer anomaly detector based on randomized n-gram analysis. We show how an adversary can _rst discover the secret information used by the detector by querying it with carefully constructed payloads and then use this information to evade the detector. The difficulties found to properly address the security of nodes in an IDN motivate our research to protect cyberdefense systems globally, assuming the possibility of attacks against some nodes and devising ways of allocating countermeasures optimally. In order to do so, it is essential to model both IDN nodes and adversarial capabilities. In the third contribution of this Thesis, we provide a conceptual model for IDNs viewed as a network of nodes whose connections and internal components determine the architecture and functionality of the global defense network. Such a model is based on the analysis and abstraction of a number of existing proposals for IDNs. Furthermore, we also develop an adversarial model for IDNs that builds on classical attack capabilities for communication networks and allow to specify complex attacks against IDN nodes. Finally, the fourth contribution of this Thesis presents DEFIDNET, a framework to assess the vulnerabilities of IDNs, the threats to which they are exposed, and optimal countermeasures to minimize risk considering possible economic and operational constraints. The framework uses the system and adversarial models developed earlier in this Thesis, together with a risk rating procedure that evaluates the propagation of attacks against particular nodes throughout the entire IDN and estimates the impacts of such actions according to different attack strategies. This assessment is then used to search for countermeasures that are both optimal in terms of involved cost and amount of mitigated risk. This is done using multi-objective optimization algorithms, thus offering the analyst sets of solutions that could be applied in different operational scenarios. -------------------------------------------------------------Las Redes de Detección de Intrusiones (IDNs, por sus siglas en inglés) constituyen un elemento primordial de los actuales sistemas de ciberdefensa. Una IDN está compuesta por diferentes nodos distribuidos a lo largo de una infraestructura de red que realizan funciones de detección de ataques --fundamentalmente a través de Sistemas de Detección de Intrusiones, o IDS--, intercambio de información con otros nodos de la IDN, y agregación y correlación de eventos procedentes de distintas fuentes. En conjunto, una IDN es capaz de detectar ataques distribuidos y de gran escala que se manifiestan en diferentes partes de la red simultáneamente. Las IDNs se han convertido en objeto de ataques avanzados cuyo fin es evadir las funciones de seguridad que ofrecen y ganar así control sobre los sistemas protegidos. Con objeto de garantizar la seguridad y privacidad de la infraestructura de red y de la IDN, es necesario diseñar arquitecturas resilientes para IDNs que sean capaces de mantener un nivel mínimo de funcionalidad incluso cuando ciertos nodos son evadidos, comprometidos o inutilizados. La investigación en este campo se ha centrado tradicionalmente en el diseño de algoritmos de detección robustos para IDS. Sin embargo, la seguridad global de la IDN ha recibido considerablemente menos atención, lo que ha resultado en una carencia de principios de diseño para arquitecturas de IDN resilientes. Esta Tesis Doctoral proporciona varias contribuciones en la investigación de IDN resilientes. La investigación aquí presentada se agrupa en dos grandes bloques. Por un lado, las dos primeras contribuciones proporcionan técnicas de análisis de la seguridad de nodos IDS contra ataques deliberados. Por otro lado, las contribuciones tres y cuatro presentan mecanismos de diseño de arquitecturas IDS robustas frente a adversarios. En la primera contribución se proponen ataques de evasión e ingeniería inversa sobre detectores de anomalíaas que utilizan algoritmos de clasificación en el motor de detección. Estos algoritmos han sido ampliamente estudiados en el campo de la detección de anomalías y son generalmente considerados efectivos y eficientes. A pesar de esto, los detectores de anomalías no consideran el papel que un adversario puede desempeñar si persigue activamente decrementar la efectividad o la eficiencia del proceso de detección. En esta Tesis se demuestra que el uso de algoritmos de clasificación simples para la detección de anomalías es, en general, vulnerable a ataques de ingeniería inversa y evasión, lo que convierte a estos algoritmos en inapropiados para sistemas reales. La segunda contribución analiza la seguridad de la aleatorización como contramedida frente a los ataques de evasión contra detectores de anomalías. Esta contramedida ha sido propuesta recientemente como mecanismo de ocultación de la superficie de decisión, lo que supuestamente dificulta la tarea del adversario. En esta Tesis se propone un ataque de ingeniería inversa basado en un análisis consulta-respuesta que demuestra que, en general, la aleatorización no proporciona un nivel de seguridad sustancialmente superior. El ataque se demuestra contra Anagram, un detector de anomalías muy popular basado en el análisis de n-gramas que opera en la capa de aplicación. El ataque permite a un adversario descubrir la información secreta utilizada durante la aleatorización mediante la construcción de paquetes cuidadosamente diseñados. Tras la finalización de este proceso, el adversario se encuentra en disposición de lanzar un ataque de evasión. Los trabajos descritos anteriormente motivan la investigación de técnicas que permitan proteger sistemas de ciberdefensa tales como una IDN incluso cuando la seguridad de algunos de sus nodos se ve comprometida, así como soluciones para la asignación óptima de contramedidas. Para ello, resulta esencial disponer de modelos tanto de los nodos de una IDN como de las capacidades del adversario. En la tercera contribución de esta Tesis se proporcionan modelos conceptuales para ambos elementos. El modelo de sistema permite representar una IDN como una red de nodos cuyas conexiones y componentes internos determinan la arquitectura y funcionalidad de la red global de defensa. Este modelo se basa en el análisis y abstracción de diferentes arquitecturas para IDNs propuestas en los últimos años. Asimismo, se desarrolla un modelo de adversario para IDNs basado en las capacidades clásicas de un atacante en redes de comunicaciones que permite especificar ataques complejos contra nodos de una IDN. Finalmente, la cuarta y última contribución de esta Tesis Doctoral describe DEFIDNET, un marco que permite evaluar las vulnerabilidades de una IDN, las amenazas a las que están expuestas y las contramedidas que permiten minimizar el riesgo de manera óptima considerando restricciones de naturaleza económica u operacional. DEFIDNET se basa en los modelos de sistema y adversario desarrollados anteriormente en esta Tesis, junto con un procedimiento de evaluación de riesgos que permite calcular la propagación a lo largo de la IDN de ataques contra nodos individuales y estimar el impacto de acuerdo a diversas estrategias de ataque. El resultado del análisis de riesgos es utilizado para determinar contramedidas óptimas tanto en términos de coste involucrado como de cantidad de riesgo mitigado. Este proceso hace uso de algoritmos de optimización multiobjetivo y ofrece al analista varios conjuntos de soluciones que podrían aplicarse en distintos escenarios operacionales.Programa en Ciencia y Tecnología InformáticaPresidente: Andrés Marín López; Vocal: Sevil Sen; Secretario: David Camacho Fernánde

Security in Distributed, Grid, Mobile, and Pervasive Computing

This book addresses the increasing demand to guarantee privacy, integrity, and availability of resources in networks and distributed systems. It first reviews security issues and challenges in content distribution networks, describes key agreement protocols based on the Diffie-Hellman key exchange and key management protocols for complex distributed systems like the Internet, and discusses securing design patterns for distributed systems. The next section focuses on security in mobile computing and wireless networks. After a section on grid computing security, the book presents an overview of security solutions for pervasive healthcare systems and surveys wireless sensor network security