Flow Monitoring Explained: From Packet Capture to Data Analysis With NetFlow and IPFIX

Flow monitoring has become a prevalent method for monitoring traffic in high-speed networks. By focusing on the analysis of flows, rather than individual packets, it is often said to be more scalable than traditional packet-based traffic analysis. Flow monitoring embraces the complete chain of packet observation, flow export using protocols such as NetFlow and IPFIX, data collection, and data analysis. In contrast to what is often assumed, all stages of flow monitoring are closely intertwined. Each of these stages therefore has to be thoroughly understood, before being able to perform sound flow measurements. Otherwise, flow data artifacts and data loss can be the consequence, potentially without being observed. This paper is the first of its kind to provide an integrated tutorial on all stages of a flow monitoring setup. As shown throughout this paper, flow monitoring has evolved from the early 1990s into a powerful tool, and additional functionality will certainly be added in the future. We show, for example, how the previously opposing approaches of deep packet inspection and flow monitoring have been united into novel monitoring approaches

Enhanced IPFIX flow monitoring for VXLAN based cloud overlay networks

The demands for cloud computing services is rapidly growing due to its fast adoption and the migration of workloads from private data centers to cloud data centers. Many companies, small and large, prefer switching their data to the enterprise cloud environment rather than expanding their own data centers. As a result, the network traffic in cloud data centers is increasing rapidly. However, due to the dynamic resource provisioning and high-speed virtualized cloud networks, the traditional flow-monitoring systems is unable to provide detail visibility and information of traffic traversing the cloud overlay network environment. Hence, it does not fulfill the monitoring requirement of cloud overlay traffic. As the growth of cloud network traffic causes difficulties for the service providers and end-users to manage the traffic efficiently, an enhanced IPFIX flow monitoring mechanism for cloud overlay networks was proposed to address this problem. The monitoring mechanism provided detail visibility and information of overlay network traffic that traversed the cloud environment, which is not available in the current network monitoring systems. The experimental results showed that the proposed monitoring system able to capture overlay network traffic and segregated the tenant traffic based on virtual machines as compare to the standard monitoring system

A flow-based intrusion detection framework for internet of things networks

The application of the Internet of Things concept in domains such as industrial control, building automation, human health, and environmental monitoring, introduces new privacy and security challenges. Consequently, traditional implementation of monitoring and security mechanisms cannot always be presently feasible and adequate due to the number of IoT devices, their heterogeneity and the typical limitations of their technical specifications. In this paper, we propose an IP flow-based Intrusion Detection System (IDS) framework to monitor and protect IoT networks from external and internal threats in real-time. The proposed framework collects IP flows from an IoT network and analyses them in order to monitor and detect attacks, intrusions, and other types of anomalies at different IoT architecture layers based on some flow features instead of using packet headers fields and their payload. The proposed framework was designed to consider both the IoT network architecture and other IoT contextual characteristics such as scalability, heterogeneity, interoperability, and the minimization of the use of IoT networks resources. The proposed IDS framework is network-based and relies on a hybrid architecture, as it involves both centralized analysis and distributed data collection components. In terms of detection method, the framework uses a specification-based approach drawn on normal traffic specifications. The experimental results show that this framework can achieve & 100% success and 0% of false positives in detection of intrusions and anomalies. In terms of performance and scalability in the operation of the IDS components, we study and compare it with three different conventional IDS (Snort, Suricata, and Zeek) and the results demonstrate that the proposed solution can consume fewer computational resources (CPU, RAM, and persistent memory) when compared to those conventional IDS.This work was supported by Portuguese national funds through the FCT—Foundation for Science and Technology, I.P., under the project UID/CEC/04524/2019info:eu-repo/semantics/publishedVersio

Engineering the application of machine learning in an IDS based on IoT traffic flow

Internet of Things (IoT) devices are now widely used, enabling intelligent services that, in association with new communication technologies like the 5G and broadband internet, boost smart-city environments. Despite their limited resources, IoT devices collect and share large amounts of data and are connected to the internet, becoming an attractive target for malicious actors. This work uses machine learning combined with an Intrusion Detection System (IDS) to detect possible attacks. Due to the limitations of IoT devices and low latency services, the IDS must have a specialized architecture. Furthermore, although machine learning-based solutions have high potential, there are still challenges related to training and generalization, which may impose constraints on the architecture. Our proposal is an IDS with a distributed architecture that relies on Fog computing to run specialized modules and use deep neural networks to identify malicious traffic inside IoT data flows. We compare our IoT-Flow IDS with three other architectures. We assess model generalization using test data from different datasets and evaluate their performance in terms of Recall, Precision, and F1-Score. Results confirm the feasibility of flowbased anomaly detection and the importance of network traffic segmentation and specialized models in the AI-based IDS for IoT.info:eu-repo/semantics/publishedVersio

Research methods of collecting traffic statistics in ip data networks

Настоящая статья посвящена исследованию нескольких различных методов сбора статистических данных, применительно к IP-сетям начиная с самых примитивных и заканчивая наиболее фукциональными – коммандная строка, SNMP, прокси-сервер, NetFlow, DPI. Все эти методы так или иначе применяются во всех современных сетях передачи данных. Были рассмотрены различия между методами и применимость каждого из них на практике.This article is devoted to the study of several different statistical techniques, applied to IP-based networks – from the most primitive and ending with the most functionality - the command line, SNMP, proxy server, NetFlow, DPI. All these methods are used anyway in all modern data networks. Have been considered the differences between the methods and applicability of each of them practically

Towards real-time intrusion detection for NetFlow and IPFIX

DDoS attacks bring serious economic and technical damage to networks and enterprises. Timely detection and mitigation are therefore of great importance. However, when flow monitoring systems are used for intrusion detection, as it is often the case in campus, enterprise and backbone networks, timely data analysis is constrained by the architecture of NetFlow and IPFIX. In their current architecture, the analysis is performed after certain timeouts, which generally delays the intrusion detection for several minutes. This paper presents a functional extension for both NetFlow and IPFIX flow exporters, to allow for timely intrusion detection and mitigation of large flooding attacks. The contribution of this paper is threefold. First, we integrate a lightweight intrusion detection module into a flow exporter, which moves detection closer to the traffic observation point. Second, our approach mitigates attacks in near real-time by instructing firewalls to filter malicious traffic. Third, we filter flow data of malicious traffic to prevent flow collectors from overload. We validate our approach by means of a prototype that has been deployed on a backbone link of the Czech national research and education network CESNET

Unveiling flat traffic on the internet: An SSH attack case study

Many types of brute-force attacks are known to exhibit a characteristic ‘flat’ behavior at the network-level, meaning that connections belonging to an attack feature a similar number of packets and bytes, and duration. Flat traffic usually results from repeating similar application-layer actions, such as login attempts in a brute-force attack. For typical attacks, hundreds of attempts span over multiple connections, with each connection containing the same, small number of attempts. The characteristic flat behavior is used by many Intrusion Detection Systems (IDSes), both for identifying the presence of attacks and — once detected — for observing deviations, pointing out potential compromises, for example. However, flatness of network traffic may become indistinct when TCP retransmissions and control information come into play. These TCP phenomena affect not only intrusion detection, but also other forms of network traffic analysis. The contribution of this work is twofold. First, we analyze the impact of retransmissions and control information on network traffic based on traffic measurements. To do so, we have developed a flow exporter extension that was deployed in both a campus and a backbone network. Second, we show that intrusion detection results improve dramatically by up to 16 percentage points once IDSes are able to ‘flatten’ network traffic again, which we have validated by means of analyzing log files of almost 60 hosts over a period of one month

Monitoring multicast traffic in heterogeneous networks

Estágio realizado no INESC - Porto e orientado pelo Prof. Doutor Ricardo MorlaTese de mestrado integrado. Engenharia Electrotécnica e de Computadores - Major Telecomunicações. Faculdade de Engenharia. Universidade do Porto. 200

Extending IP Flow-Based Network Monitoring with Location Information

Internet Draft - IETFIP Flow-based monitoring lacks a mechanism to associate measured IP Flow information with the geographic location of the device where theIP Flows have been observed. This document defines a set of guidelines and best practices to extend IP Flow monitoring protocols with location information of the device (both fixed and mobile) that acts as an IP Flow metering process