MACHINE LEARNING STATISTICAL DETECTION OF ANOMALIES USING NETFLOW RECORDS

NetFlow is a network protocol system that is used to represent an overall summary of computer network conversations. A NetFlow record can convert previously captured packet captures or obtain NetFlow session data in real time. This research examines the use of machine-learning techniques to identify anomalies in NetFlow records and classify malware behavior for further investigation. The intent is to identify low-cost solutions leveraging open-source software capable of deployment on computer hardware of currently in-use data networks. This work seeks to determine whether expert selection of features can improve machine-learning detection algorithm performance and evaluate the trade-offs associated with eliminating redundant or excessive numbers of features. We identify the Random Forest algorithm as the strongest single algorithm across three of four metrics, with our chosen NetFlow features cutting the testing and training times in half while incurring minor reductions in two metrics. The experiment demonstrates that the chosen NetFlow features are sufficiently discriminative to detect attacks with a success rate higher than 94%.NCWDGLieutenant, United States NavyApproved for public release. Distribution is unlimited

Detection of Network Attacks Based on NetFlow Data

V současné době stále pokračuje dlouhodobý trend nárůstu kyberkriminality takřka po celém světě. Tato práce se zabývá stále sílící problematikou bezpečnosti síťového provozu, konkrétně detekcí útoků. V rámci práce je navržen program pro detekci anomálií na síti na základě NetFlow dat, za účelem důkladnější ochrany běžných uživatelů. Program je realizován metodou TCM-KNN využívající statistických odlišností útoků, čímž umožňuje zaznamenat i jejich nové, dříve neviděné instanceWith rising popularity of the internet there is also rising number of people misusing it. This thesis analyzes the problem of network attack detection based on NetFlow data. A program is designed to point out anomalous behaviour by analyzing the flow records using data mining techniques. The method of TCM-KNN utilizing the fact that attacks statistically deviate is implemented. Thus even new types of attacks are detected

Discovery of Web Attacks by Inspecting HTTPS Network Traffic with Machine Learning and Similarity Search

Tese de mestrado, Segurança Informática, Universidade de Lisboa, Faculdade de Ciências, 2022Web applications are the building blocks of many services, from social networks to banks. Network security threats have remained a permanent concern since the advent of data communication. Not withstanding, security breaches are still a serious problem since web applications incorporate both company information and private client data. Traditional Intrusion Detection Systems (IDS) inspect the payload of the packets looking for known intrusion signatures or deviations from nor mal behavior. However, this Deep Packet Inspection (DPI) approach cannot inspect encrypted network traffic of Hypertext Transfer Protocol Secure (HTTPS), a protocol that has been widely adopted nowadays to protect data communication. We are interested in web application attacks, and to accurately detect them, we must access the payload. Network flows are able to aggregate flows of traffic with common properties, so they can be employed for inspecting large amounts of traffic. The main objective of this thesis is to develop a system to discover anomalous HTTPS traffic and confirm that the payloads included in it contains web applications attacks. We propose a new reliable method and system to identify traffic that may include web application attacks by analysing HTTPS network flows (netflows) and discovering payload content similarities. We resort to unsupervised machine learning algorithms to cluster netflows and identify anomalous traffic and to Locality Sensitive Hashing (LSH) algorithms to create a Similarity Search Engine (SSE) capable of correctly identifying the presence of known web applications attacks over this traffic. We involve the system in a continuous improvement process to keep a reliable detection as new web applications attacks are discovered. We evaluated the system, which showed that it could detect anomalous traffic, the SSE was able to confirm the presence of web attacks into that anomalous traffic, and the continuous improvement process was able to increase the accuracy of the SSE

Federated Agentless Detection of Endpoints Using Behavioral and Characteristic Modeling

During the past two decades computer networks and security have evolved that, even though we use the same TCP/IP stack, network traffic behaviors and security needs have significantly changed. To secure modern computer networks, complete and accurate data must be gathered in a structured manner pertaining to the network and endpoint behavior. Security operations teams struggle to keep up with the ever-increasing number of devices and network attacks daily. Often the security aspect of networks gets managed reactively instead of providing proactive protection. Data collected at the backbone are becoming inadequate during security incidents. Incident response teams require data that is reliably attributed to each individual endpoint over time. With the current state of dissociated data collected from networks using different tools it is challenging to correlate the necessary data to find origin and propagation of attacks within the network. Critical indicators of compromise may go undetected due to the drawbacks of current data collection systems leaving endpoints vulnerable to attacks. Proliferation of distributed organizations demand distributed federated security solutions. Without robust data collection systems that are capable of transcending architectural and computational challenges, it is becoming increasingly difficult to provide endpoint protection at scale. This research focuses on reliable agentless endpoint detection and traffic attribution in federated networks using behavioral and characteristic modeling for incident response

Detection of HTTPS brute-force attacks in high-speed computer networks

Tato práce představuje přehled metod pro detekci síťových hrozeb se zaměřením na útoky hrubou silou proti webovým aplikacím, jako jsou WordPress a Joomla. Byl vytvořen nový dataset, který se skládá z provozu zachyceného na páteřní síti a útoků generovaných pomocí open-source nástrojů. Práce přináší novou metodu pro detekci útoku hrubou silou, která je založena na charakteristikách jednotlivých paketů a používá moderní metody strojového učení. Metoda funguje s šifrovanou HTTPS komunikací, a to bez nutnosti dešifrování jednotlivých paketů. Stále více webových aplikací používá HTTPS pro zabezpečení komunikace, a proto je nezbytné aktualizovat detekční metody, aby byla zachována základní viditelnost do síťového provozu.This thesis presents a review of flow-based network threat detection, with the focus on brute-force attacks against popular web applications, such as WordPress and Joomla. A new dataset was created that consists of benign backbone network traffic and brute-force attacks generated with open-source attack tools. The thesis proposes a method for brute-force attack detection that is based on packet-level characteristics and uses modern machine-learning models. Also, it works with encrypted HTTPS traffic, even without decrypting the payload. More and more network traffic is being encrypted, and it is crucial to update our intrusion detection methods to maintain at least some level of network visibility

The Dark Menace: Characterizing Network-based Attacks in the Cloud

ABSTRACT As the cloud computing market continues to grow, the cloud platform is becoming an attractive target for attackers to disrupt services and steal data, and to compromise resources to launch attacks. In this paper, using three months of NetFlow data in 2013 from a large cloud provider, we present the first large-scale characterization of inbound attacks towards the cloud and outbound attacks from the cloud. We investigate nine types of attacks ranging from network-level attacks such as DDoS to application-level attacks such as SQL injection and spam. Our analysis covers the complexity, intensity, duration, and distribution of these attacks, highlighting the key challenges in defending against attacks in the cloud. By characterizing the diversity of cloud attacks, we aim to motivate the research community towards developing future security solutions for cloud systems

Towards Scalable Network Traffic Measurement With Sketches

Driven by the ever-increasing data volume through the Internet, the per-port speed of network devices reached 400 Gbps, and high-end switches are capable of processing 25.6 Tbps of network traffic. To improve the efficiency and security of the network, network traffic measurement becomes more important than ever. For fast and accurate traffic measurement, managing an accurate working set of active flows (WSAF) at line rates is a key challenge. WSAF is usually located in high-speed but expensive memories, such as TCAM or SRAM, and thus their capacity is quite limited. To scale up the per-flow measurement, we pursue three thrusts. In the first thrust, we propose to use In-DRAM WSAF and put a compact data structure (i.e., sketch) called FlowRegulator before WSAF to compensate for DRAM\u27s slow access time. Per our results, FlowRegulator can substantially reduce massive influxes to WSAF without compromising measurement accuracy. In the second thrust, we integrate our sketch into a network system and propose an SDN-based WLAN monitoring and management framework called RFlow+, which can overcome the limitations of existing traffic measurement solutions (e.g., OpenFlow and sFlow), such as a limited view, incomplete flow statistics, and poor trade-off between measurement accuracy and CPU/network overheads. In the third thrust, we introduce a novel sampling scheme to deal with the poor trade-off that is provided by the standard simple random sampling (SRS). Even though SRS has been widely used in practice because of its simplicity, it provides non-uniform sampling rates for different flows, because it samples packets over an aggregated data flow. Starting with a simple idea that independent per-flow packet sampling provides the most accurate estimation of each flow, we introduce a new concept of per-flow systematic sampling, aiming to provide the same sampling rate across all flows. In addition, we provide a concrete sampling method called SketchFlow, which approximates the idea of the per-flow systematic sampling using a sketch saturation event

A survey of defense mechanisms against distributed denial of service (DDOS) flooding attacks

Distributed Denial of Service (DDoS) flooding attacks are one of the biggest concerns for security professionals. DDoS flooding attacks are typically explicit attempts to disrupt legitimate users' access to services. Attackers usually gain access to a large number of computers by exploiting their vulnerabilities to set up attack armies (i.e., Botnets). Once an attack army has been set up, an attacker can invoke a coordinated, large-scale attack against one or more targets. Developing a comprehensive defense mechanism against identified and anticipated DDoS flooding attacks is a desired goal of the intrusion detection and prevention research community. However, the development of such a mechanism requires a comprehensive understanding of the problem and the techniques that have been used thus far in preventing, detecting, and responding to various DDoS flooding attacks. In this paper, we explore the scope of the DDoS flooding attack problem and attempts to combat it. We categorize the DDoS flooding attacks and classify existing countermeasures based on where and when they prevent, detect, and respond to the DDoS flooding attacks. Moreover, we highlight the need for a comprehensive distributed and collaborative defense approach. Our primary intention for this work is to stimulate the research community into developing creative, effective, efficient, and comprehensive prevention, detection, and response mechanisms that address the DDoS flooding problem before, during and after an actual attack. © 1998-2012 IEEE

A Deep Learning-based Approach to Identifying and Mitigating Network Attacks Within SDN Environments Using Non-standard Data Sources

Modern society is increasingly dependent on computer networks, which are essential to delivering an increasing number of key services. With this increasing dependence, comes a corresponding increase in global traffic and users. One of the tools administrators are using to deal with this growth is Software Defined Networking (SDN). SDN changes the traditional distributed networking design to a more programmable centralised solution, based around the SDN controller. This allows administrators to respond more quickly to changing network conditions. However, this change in paradigm, along with the growing use of encryption can cause other issues. For many years, security administrators have used techniques such as deep packet inspection and signature analysis to detect malicious activity. These methods are becoming less common as artificial intelligence (AI) and deep learning technologies mature. AI and deep learning have advantages in being able to cope with 0-day attacks and being able to detect malicious activity despite the use of encryption and obfuscation techniques. However, SDN reduces the volume of data that is available for analysis with these machine learning techniques. Rather than packet information, SDN relies on flows, which are abstract representations of network activity. Security researchers have been slow to move to this new method of networking, in part because of this reduction in data, however doing so could have advantages in responding quickly to malicious activity. This research project seeks to provide a way to reconcile the contradiction apparent, by building a deep learning model that can achieve comparable results to other state-of-the-art models, while using 70% fewer features. This is achieved through the creation of new data from logs, as well as creation of a new risk-based sampling method to prioritise suspect flows for analysis, which can successfully prioritise over 90% of malicious flows from leading datasets. Additionally, provided is a mitigation method that can work with a SDN solution to automatically mitigate attacks after they are found, showcasing the advantages of closer integration with SDN