TOWARDS A HOLISTIC EFFICIENT STACKING ENSEMBLE INTRUSION DETECTION SYSTEM USING NEWLY GENERATED HETEROGENEOUS DATASETS

With the exponential growth of network-based applications globally, there has been a transformation in organizations\u27 business models. Furthermore, cost reduction of both computational devices and the internet have led people to become more technology dependent. Consequently, due to inordinate use of computer networks, new risks have emerged. Therefore, the process of improving the speed and accuracy of security mechanisms has become crucial.Although abundant new security tools have been developed, the rapid-growth of malicious activities continues to be a pressing issue, as their ever-evolving attacks continue to create severe threats to network security. Classical security techniquesfor instance, firewallsare used as a first line of defense against security problems but remain unable to detect internal intrusions or adequately provide security countermeasures. Thus, network administrators tend to rely predominantly on Intrusion Detection Systems to detect such network intrusive activities. Machine Learning is one of the practical approaches to intrusion detection that learns from data to differentiate between normal and malicious traffic. Although Machine Learning approaches are used frequently, an in-depth analysis of Machine Learning algorithms in the context of intrusion detection has received less attention in the literature.Moreover, adequate datasets are necessary to train and evaluate anomaly-based network intrusion detection systems. There exist a number of such datasetsas DARPA, KDDCUP, and NSL-KDDthat have been widely adopted by researchers to train and evaluate the performance of their proposed intrusion detection approaches. Based on several studies, many such datasets are outworn and unreliable to use. Furthermore, some of these datasets suffer from a lack of traffic diversity and volumes, do not cover the variety of attacks, have anonymized packet information and payload that cannot reflect the current trends, or lack feature set and metadata.This thesis provides a comprehensive analysis of some of the existing Machine Learning approaches for identifying network intrusions. Specifically, it analyzes the algorithms along various dimensionsnamely, feature selection, sensitivity to the hyper-parameter selection, and class imbalance problemsthat are inherent to intrusion detection. It also produces a new reliable dataset labeled Game Theory and Cyber Security (GTCS) that matches real-world criteria, contains normal and different classes of attacks, and reflects the current network traffic trends. The GTCS dataset is used to evaluate the performance of the different approaches, and a detailed experimental evaluation to summarize the effectiveness of each approach is presented. Finally, the thesis proposes an ensemble classifier model composed of multiple classifiers with different learning paradigms to address the issue of detection accuracy and false alarm rate in intrusion detection systems

Evaluating the Proliferation and Pervasiveness of Leaking Sensitive Data in the Secure Shell Protocol and in Internet Protocol Camera Frameworks

In George Orwell’s `nineteen eighty-four: A novel', there is fear regarding what “Big Brother”, knows due to the fact that even thoughts could be “heard”. Though we are not quite to this point, it should concern us all in what data we are transferring, both intentionally and unintentionally, and whether or not that data is being “leaked”. In this work, we consider the evolving landscape of IoT devices and the threat posed by the pervasive botnets that have been forming over the last several years. We look at two specific cases in this work. One being the practical application of a botnet system actively executing a Man in the Middle Attack against SSH, and the other leveraging the same paradigm as a case of eavesdropping on Internet Protocol (IP) cameras. For the latter case, we construct a web portal for interrogating IP cameras directly for information that they may be exposing

A systematic literature review

Bahaa, A., Abdelaziz, A., Sayed, A., Elfangary, L., & Fahmy, H. (2021). Monitoring real time security attacks for iot systems using devsecops: A systematic literature review. Information (Switzerland), 12(4), 1-23. [154]. https://doi.org/10.3390/info12040154In many enterprises and the private sector, the Internet of Things (IoT) has spread globally. The growing number of different devices connected to the IoT and their various protocols have contributed to the increasing number of attacks, such as denial-of-service (DoS) and remote-to-local (R2L) ones. There are several approaches and techniques that can be used to construct attack detection models, such as machine learning, data mining, and statistical analysis. Nowadays, this technique is commonly used because it can provide precise analysis and results. Therefore, we decided to study the previous literature on the detection of IoT attacks and machine learning in order to understand the process of creating detection models. We also evaluated various datasets used for the models, IoT attack types, independent variables used for the models, evaluation metrics for assessment of models, and monitoring infrastructure using DevSecOps pipelines. We found 49 primary studies, and the detection models were developed using seven different types of machine learning techniques. Most primary studies used IoT device testbed datasets, and others used public datasets such as NSL-KDD and UNSW-NB15. When it comes to measuring the efficiency of models, both numerical and graphical measures are commonly used. Most IoT attacks occur at the network layer according to the literature. If the detection models applied DevSecOps pipelines in development processes for IoT devices, they were more secure. From the results of this paper, we found that machine learning techniques can detect IoT attacks, but there are a few issues in the design of detection models. We also recommend the continued use of hybrid frameworks for the improved detection of IoT attacks, advanced monitoring infrastructure configurations using methods based on software pipelines, and the use of machine learning techniques for advanced supervision and monitoring.publishersversionpublishe

MULTI-DIMENSIONAL PROFILING OF CYBER THREATS FOR LARGE-SCALE NETWORKS

Current multi-domain command and control computer networks require significant oversight to ensure acceptable levels of security. Firewalls are the proactive security management tool at the network’s edge to determine malicious and benign traffic classes. This work aims to develop machine learning algorithms through deep learning and semi-supervised clustering, to enable the profiling of potential threats through network traffic analysis within large-scale networks. This research accomplishes these objectives by analyzing enterprise network data at the packet level using deep learning to classify traffic patterns. In addition, this work examines the efficacy of several machine learning model types and multiple imbalanced data handling techniques. This work also incorporates packet streams for identifying and classifying user behaviors. Tests of the packet classification models demonstrated that deep learning is sensitive to malicious traffic but underperforms in identifying allowed traffic compared to traditional algorithms. However, imbalanced data handling techniques provide performance benefits to some deep learning models. Conversely, semi-supervised clustering accurately identified and classified multiple user behaviors. These models provide an automated tool to learn and predict future traffic patterns. Applying these techniques within large-scale networks detect abnormalities faster and gives network operators greater awareness of user traffic.Outstanding ThesisCaptain, United States Marine CorpsApproved for public release. Distribution is unlimited

Detecting obfuscated and malicious network traffic using open source software

Hämäännytetyllä tietoliikenteellä tarkoitetaan tietoliikenneprotokollan toiminnan tarkoituksenmukaista monimutkaistamista. Tavoitteena on hämäännyttää liikennettä tutkiva verkkosensori, jonka tarkoituksena on tunnistaa liikennöivä sovellus, eli suorittaa liikenteen luokitus. Verkkosensorit perustuvat tavallisesti pakettien syvätarkastukseen ja liikenteen tilastolliseen analyysiin. Syvätarkastuksen avulla tutkitaan IP-paketin hyötykuormaa. Tilastollinen analyysi pyrkii tutkimaan liikenteelle ominaisia piirteitä, kuten pakettikokoa tai pakettien välistä saapumisaikaa. Tässä diplomityössä tutkitaan hämäännytetyn ja haitallisen verkkoliikenteen tunnistamista. Tämän diplomityön keskeisimmät tutkimuskysymykset ovat: Kykenevätkö avoimen lähdekoodin pakettien syvätarkastusohjelmistot luokittelemaan hämäännytettyä liikennettä? Kuinka hämäännytettyä liikennettä voidaan havaita? Kuinka muodostetaan liikennettä, jota on vaikea havaita? Tutkimus suoritettiin sitä varten kehitetyssä suljetussa testausympäristössä, jossa generoitiin hämäännytettyä liikennettä sekä avoimen lähdekoodin hämäännyttämisohjelmistojen avulla, että haitallisten takaporttiohjelmistojen avulla. Haitallisista takaporttiohjelmistoista laadittiin liikenneanalyysi eri ilmaisohjelmistojen avulla. Testausympäristössä otettiin käyttöön kolme eri avoimen lähdekoodin pakettien syvätarkastusohjelmistoa, joiden avulla testattiin generoidun liikenteen luokittelua. Hämäännyttämis- ja takaporttiohjelmistojen muodostamasta liikenteestä laadittiin myös tilastollinen analyysi. Tutkimuksen keskeisimpinä tuloksina havaittiin, että oletusasetuksilla DPI-kirjastot kykenevät luokittelemaan hämäännytettyä liikennettä. Kuitenkin pääosa hämäännytetystä liikenteestä luokiteltiin tuntematon-luokkaan, kuten alkuperäinen oletus olikin. Vääriä positiivisia luokituksia syntyi eniten säännöllisiin lausekkeisiin perustuvassa DPI-kirjastossa. Työn tuloksissa esitetään, kuinka osalle tuntemattomaksi luokitellusta liikenteestä voidaan kehittää omat protokolladekooderinsa. Selkeimmät tapaukset ovat ne, joiden hyötykuormasta voidaan lukea selväkielisiä merkkijonoja tai yhteydenmuodostuskättely on tunnistettavissa. Osaan esitetyistä hämäännytysmenetelmistä ei voida soveltaa pakettien syvätarkastusta, koska liikenne on salakirjoitettua ja yhteydenmuodostuskättelystä on vaikea havaita tunnistettavia piirteitä. Näihin menetelmiin voidaan hyödyntää tilastollista analyysiä. Esimerkiksi jaetun salaisuuden avulla salatun takaporttiohjelmiston liikenne on mahdollista tunnistaa pakettikokojakauman avulla. Tilastollisen analyysin tuloksia ei voida kuitenkaan yleistää kaikkiin menetelmiin, sillä hämäännytettävällä sovelluksella on mahdollisesti vaikutusta edellä esitettyihin eri jakaumiin

Deteção de propagação de ameaças e exfiltração de dados em redes empresariais

Modern corporations face nowadays multiple threats within their networks. In an era where companies are tightly dependent on information, these threats can seriously compromise the safety and integrity of sensitive data. Unauthorized access and illicit programs comprise a way of penetrating the corporate networks, able to traversing and propagating to other terminals across the private network, in search of confidential data and business secrets. The efficiency of traditional security defenses are being questioned with the number of data breaches occurred nowadays, being essential the development of new active monitoring systems with artificial intelligence capable to achieve almost perfect detection in very short time frames. However, network monitoring and storage of network activity records are restricted and limited by legal laws and privacy strategies, like encryption, aiming to protect the confidentiality of private parties. This dissertation proposes methodologies to infer behavior patterns and disclose anomalies from network traffic analysis, detecting slight variations compared with the normal profile. Bounded by network OSI layers 1 to 4, raw data are modeled in features, representing network observations, and posteriorly, processed by machine learning algorithms to classify network activity. Assuming the inevitability of a network terminal to be compromised, this work comprises two scenarios: a self-spreading force that propagates over internal network and a data exfiltration charge which dispatch confidential info to the public network. Although features and modeling processes have been tested for these two cases, it is a generic operation that can be used in more complex scenarios as well as in different domains. The last chapter describes the proof of concept scenario and how data was generated, along with some evaluation metrics to perceive the model’s performance. The tests manifested promising results, ranging from 96% to 99% for the propagation case and 86% to 97% regarding data exfiltration.Nos dias de hoje, várias organizações enfrentam múltiplas ameaças no interior da sua rede. Numa época onde as empresas dependem cada vez mais da informação, estas ameaças podem compremeter seriamente a segurança e a integridade de dados confidenciais. O acesso não autorizado e o uso de programas ilícitos constituem uma forma de penetrar e ultrapassar as barreiras organizacionais, sendo capazes de propagarem-se para outros terminais presentes no interior da rede privada com o intuito de atingir dados confidenciais e segredos comerciais. A eficiência da segurança oferecida pelos sistemas de defesa tradicionais está a ser posta em causa devido ao elevado número de ataques de divulgação de dados sofridos pelas empresas. Desta forma, o desenvolvimento de novos sistemas de monitorização ativos usando inteligência artificial é crucial na medida de atingir uma deteção mais precisa em curtos períodos de tempo. No entanto, a monitorização e o armazenamento dos registos da atividade da rede são restritos e limitados por questões legais e estratégias de privacidade, como a cifra dos dados, visando proteger a confidencialidade das entidades. Esta dissertação propõe metodologias para inferir padrões de comportamento e revelar anomalias através da análise de tráfego que passa na rede, detetando pequenas variações em comparação com o perfil normal de atividade. Delimitado pelas camadas de rede OSI 1 a 4, os dados em bruto são modelados em features, representando observações de rede e, posteriormente, processados por algoritmos de machine learning para classificar a atividade de rede. Assumindo a inevitabilidade de um terminal ser comprometido, este trabalho compreende dois cenários: um ataque que se auto-propaga sobre a rede interna e uma tentativa de exfiltração de dados que envia informações para a rede pública. Embora os processos de criação de features e de modelação tenham sido testados para estes dois casos, é uma operação genérica que pode ser utilizada em cenários mais complexos, bem como em domínios diferentes. O último capítulo inclui uma prova de conceito e descreve o método de criação dos dados, com a utilização de algumas métricas de avaliação de forma a espelhar a performance do modelo. Os testes mostraram resultados promissores, variando entre 96% e 99% para o caso da propagação e entre 86% e 97% relativamente ao roubo de dados.Mestrado em Engenharia de Computadores e Telemátic