Flow Monitoring Explained: From Packet Capture to Data Analysis With NetFlow and IPFIX

Flow monitoring has become a prevalent method for monitoring traffic in high-speed networks. By focusing on the analysis of flows, rather than individual packets, it is often said to be more scalable than traditional packet-based traffic analysis. Flow monitoring embraces the complete chain of packet observation, flow export using protocols such as NetFlow and IPFIX, data collection, and data analysis. In contrast to what is often assumed, all stages of flow monitoring are closely intertwined. Each of these stages therefore has to be thoroughly understood, before being able to perform sound flow measurements. Otherwise, flow data artifacts and data loss can be the consequence, potentially without being observed. This paper is the first of its kind to provide an integrated tutorial on all stages of a flow monitoring setup. As shown throughout this paper, flow monitoring has evolved from the early 1990s into a powerful tool, and additional functionality will certainly be added in the future. We show, for example, how the previously opposing approaches of deep packet inspection and flow monitoring have been united into novel monitoring approaches

Tietoverkkojen valvonnan yhdenmukaistaminen

As the modern society is increasingly dependant on computer networks especially as the Internet of Things gaining popularity, a need to monitor computer networks along with associated devices increases. Additionally, the amount of cyber attacks is increasing and certain malware such as Mirai target especially network devices. In order to effectively monitor computer networks and devices, effective solutions are required for collecting and storing the information. This thesis designs and implements a novel network monitoring system. The presented system is capable of utilizing state-of-the-art network monitoring protocols and harmonizing the collected information using a common data model. This design allows effective queries and further processing on the collected information. The presented system is evaluated by comparing the system against the requirements imposed on the system, by assessing the amount of harmonized information using several protocols and by assessing the suitability of the chosen data model. Additionally, the protocol overheads of the used network monitoring protocols are evaluated. The presented system was found to fulfil the imposed requirements. Approximately 21% of the information provided by the chosen network monitoring protocols could be harmonized into the chosen data model format. The result is sufficient for effective querying and combining the information, as well as for processing the information further. The result can be improved by extending the data model and improving the information processing. Additionally, the chosen data model was shown to be suitable for the use case presented in this thesis.Yhteiskunnan ollessa jatkuvasti verkottuneempi erityisesti Esineiden Internetin kasvattaessa suosiotaan, tarve seurata sekä verkon että siihen liitettyjen laitteiden tilaa ja mahdollisia poikkeustilanteita kasvaa. Lisäksi tietoverkkohyökkäysten määrä on kasvamassa ja erinäiset haittaohjelmat kuten Mirai, ovat suunnattu erityisesti verkkolaitteita kohtaan. Jotta verkkoa ja sen laitteiden tilaa voidaan seurata, tarvitaan tehokkaita ratkaisuja tiedon keräämiseen sekä säilöntään. Tässä diplomityössä suunnitellaan ja toteutetaan verkonvalvontajärjestelmä, joka mahdollistaa moninaisten verkonvalvontaprotokollien hyödyntämisen tiedonkeräykseen. Lisäksi järjestelmä säilöö kerätyn tiedon käyttäen yhtenäistä tietomallia. Yhtenäisen tietomallin käyttö mahdollistaa tiedon tehokkaan jatkojalostamisen sekä haut tietosisältöihin. Diplomityössä esiteltävän järjestelmän ominaisuuksia arvioidaan tarkastelemalla, minkälaisia osuuksia eri verkonvalvontaprotokollien tarjoamasta informaatiosta voidaan yhdenmukaistaa tietomalliin, onko valittu tietomalli soveltuva verkonvalvontaan sekä varmistetaan esiteltävän järjestelmän täyttävän sille asetetut vaatimukset. Lisäksi työssä arvioidaan käytettävien verkonvalvontaprotokollien siirtämisen kiinteitä kustannuksia kuten otsakkeita. Työssä esitellyn järjestelmän todettiin täyttävän sille asetetut vaatimukset. Eri verkonvalvontaprotokollien tarjoamasta informaatiosta keskimäärin 21% voitiin harmonisoida tietomalliin. Saavutettu osuus on riittävä, jotta eri laitteista saatavaa informaatiota voidaan yhdistellä ja hakea tehokkaasti. Lukemaa voidaan jatkossa parantaa laajentamalla tietomallia sekä kehittämällä kerätyn informaation prosessointia. Lisäksi valittu tietomalli todettiin soveltuvaksi tämän diplomityön käyttötarkoitukseen

Monitoring multicast traffic in heterogeneous networks

Estágio realizado no INESC - Porto e orientado pelo Prof. Doutor Ricardo MorlaTese de mestrado integrado. Engenharia Electrotécnica e de Computadores - Major Telecomunicações. Faculdade de Engenharia. Universidade do Porto. 200

A modular traffic sampling architecture for flexible network measurements

Dissertação de Mestrado (Programa Doutoral em Informática)The massive traffic volumes and the heterogeneity of services in today’s networks urge for flexible, yet simple measurement solutions to assist network management tasks, without impairing network performance. To turn treatable tasks requiring traffic analysis, sampling the traffic has become mandatory, triggering substantial research in the area. In fact, multiple sampling techniques have been proposed to assist network engineering tasks, each one targeting specific measurement goals and traffic scenarios. Despite that, there is still a lack of an encompassing solution able to support the flexible deployment of these techniques in production networks. In this context, this research work proposes a modular traffic sampling architecture able to foster the flexible design and deployment of efficient measurement strategies. The architecture is composed of three layers i.e., management plane, control plane and data plane covering key components to achieve versatile and lightweight measurements in diverse traffic scenarios and measurement activities. The flexibility and modularity in deploying different sampling strategies relies upon a novel taxonomy of sampling techniques, in which, current and emerging techniques are identified regarding their inner characteristics - granularity, selection trigger and selection scheme. Following the proposed taxonomy, a sampling framework prototype has been developed and used as an experimental implementation of the proposed architecture, providing a fair environment to assess and compare sampling techniques under distinct measurement scenarios. Supported by the sampling framework, distinct techniques have been evaluated regarding their performance in balancing the computational burden and the accuracy in supporting traffic workload estimation and flow analysis. The results have demonstrated the relevance and applicability of the proposed architecture, revealing that a modular and configurable approach to sampling is a step forward for improving sampling scope and efficiency.Os grandes volumes de tráfego e a heterogeneidade de serviços nas redes atuais requerem soluções de medição que sejam flexíveis e simples de modo a sustentar as tarefas de gestão de redes sem afetar o desempenho das mesmas. Para tornar tratável as tarefas que exigem análise de tráfego, tornou-se obrigatório recorrer a amostragem do tráfego, motivando uma investigação substancial na área. Como consequência, várias técnicas de amostragem foram propostas para auxiliar as tarefas de engenharia de redes, cada uma orientada a satisfazer objetivos de medição e cenários de tráfego específicos. Apesar disso, ainda não existe uma solução abrangente capaz de suportar a implantação flexível destas técnicas em redes de produção. Neste contexto, este trabalho propõe uma arquitetura modular de amostragem de tráfego capaz de fomentar a concepção flexível e a implementação de estratégias efi- cientes de medição de tráfego. A arquitetura é composta por três camadas, nomeadamente, camada de gestão, camada de controle e camada de dados, cobrindo os principais componentes para alcançar versatilidade e baixo custo computacional em variados cenários de tráfego e atividades de medição. A flexibilidade e modularidade na implementação de diferentes técnicas de amostragem baseia-se numa nova taxonomia, na qual técnicas atuais e emergentes são identificadas de acordo com suas características internas - granularidade, trigger de seleção e esquema de seleção. Seguindo a taxonomia proposta, um protótipo estruturando e agregando as diferentes técnicas de amostragem foi desenvolvido e utilizado na implementação experimental da arquitetura, permitindo avaliar e comparar as técnicas de amostragem em diversos cenários de medição. Suportado pelo protótipo desenvolvido, distintas técnicas foram avaliadas quanto ao seu desempenho em equilibrar a carga computacional e a acurácia na estimação do volume de tráfego e na análise de fluxos. Os resultados demonstraram a relevância e aplicabilidade da arquitetura de amostragem proposta, revelando que uma abordagem modular e configurável constitui um avanço no sentido de melhorar a eficiência na amostragem de tráfego

NetFlow, RMON and Cisco-NAM deployment

In this report, we present the deployment of NetFlow, RMON and the Cisco Network Analysis Module, Cisco-NAM, on the team testbed. First, we present the different technologies, and then we describe their deployment and how they were integrated in the team testbed

Network traffic management for the next generation Internet

Measurement-based performance evaluation of network traffic is a fundamental prerequisite for the provisioning of managed and controlled services in short timescales, as well as for enabling the accountability of network resources. The steady introduction and deployment of the Internet Protocol Next Generation (IPNG-IPv6) promises a network address space that can accommodate any device capable of generating a digital heart-beat. Under such a ubiquitous communication environment, Internet traffic measurement becomes of particular importance, especially for the assured provisioning of differentiated levels of service quality to the different application flows. The non-identical response of flows to the different types of network-imposed performance degradation and the foreseeable expansion of networked devices raise the need for ubiquitous measurement mechanisms that can be equally applicable to different applications and transports. This thesis introduces a new measurement technique that exploits native features of IPv6 to become an integral part of the Internet's operation, and to provide intrinsic support for performance measurements at the universally-present network layer. IPv6 Extension Headers have been used to carry both the triggers that invoke the measurement activity and the instantaneous measurement indicators in-line with the payload data itself, providing a high level of confidence that the behaviour of the real user traffic flows is observed. The in-line measurements mechanism has been critically compared and contrasted to existing measurement techniques, and its design and a software-based prototype implementation have been documented. The developed system has been used to provisionally evaluate numerous performance properties of a diverse set of application flows, over different-capacity IPv6 experimental configurations. Through experimentation and theoretical argumentation, it has been shown that IPv6-based, in-line measurements can form the basis for accurate and low-overhead performance assessment of network traffic flows in short time-scales, by being dynamically deployed where and when required in a multi-service Internet environment.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Disseny, desplegament i configuració d'un testbed de xarxa amb eines de monitorització

El monitoratge de les xarxes de dades és necessari per tal d’avaluar el seu rendiment i el grau de servei que ofereix. Aquest ajuda a conèixer l’estat dels dispositius (elements de commutació o encaminament) per entendre el seu funcionament, detectar anomalies, diagnosticar problemes i determinar-ne les causes. Tot i això, ofereix poc detall sobre el tràfic cursat per els equips. Per aquest motiu, també es important el monitoratge del tràfic, ja que indica l’ús que se li està donant a la infraestructura, quin tipus de tràfic s’hi cursa, qui el genera i a on va, i si els recursos actuals són suficients per a la demanda del tràfic. Tot i que existeixen eines per al monitoratge de la xarxa i del tràfic, actualment s’estan desenvolupant nous mecanismes per solucionar les deficiències de les eines existents quant a fiabilitat, càrrega computacional als equips o escalabilitat. L’objectiu d’aquest TFC és el disseny i desplegament d’un testbed de xarxa IP per poder avaluar eines de monitoratge de xarxa i mesures de tràfic. Per poder assolir aquest objectiu, es plantegen els requisits que hauria de complir l’escenari per a poder reproduir un entorn xarxa el més realista possible quant a topologia, nombre d’equips, protocols i encaminament. Es proposa un escenari que satisfà els requisits identificats i és, a la vegada, compatible amb l’equipament disponible. Un cop dissenyat, muntat i verificat el funcionament de la xarxa, s’enriqueix l’escenari amb eines de gestió, mesura i monitoratge, i es verifica el seu funcionament. Per finalitzar, es configura els equips de la xarxa per poder accedir-hi remotament i no tenir la necessitat d’estar connectats físicament a l’escenari