Big Data and Analysis of Data Transfers for International Research Networks Using NetSage

Modern science is increasingly data-driven and collaborative in nature. Many scientific disciplines, including genomics, high-energy physics, astronomy, and atmospheric science, produce petabytes of data that must be shared with collaborators all over the world. The National Science Foundation-supported International Research Network Connection (IRNC) links have been essential to enabling this collaboration, but as data sharing has increased, so has the amount of information being collected to understand network performance. New capabilities to measure and analyze the performance of international wide-area networks are essential to ensure end-users are able to take full advantage of such infrastructure for their big data applications. NetSage is a project to develop a unified, open, privacy-aware network measurement, and visualization service to address the needs of monitoring today's high-speed international research networks. NetSage collects data on both backbone links and exchange points, which can be as much as 1Tb per month. This puts a significant strain on hardware, not only in terms storage needs to hold multi-year historical data, but also in terms of processor and memory needs to analyze the data to understand network behaviors. This paper addresses the basic NetSage architecture, its current data collection and archiving approach, and details the constraints of dealing with this big data problem of handling vast amounts of monitoring data, while providing useful, extensible visualization to end users

Neural visualization of network traffic data for intrusion detection

This study introduces and describes a novel intrusion detection system (IDS) called MOVCIDS (mobile visualization connectionist IDS). This system applies neural projection architectures to detect anomalous situations taking place in a computer network. By its advanced visualization facilities, the proposed IDS allows providing an overview of the network traffic as well as identifying anomalous situations tackled by computer networks, responding to the challenges presented by volume, dynamics and diversity of the traffic, including novel (0-day) attacks. MOVCIDS provides a novel point of view in the field of IDSs by enabling the most interesting projections (based on the fourth order statistics; the kurtosis index) of a massive traffic dataset to be extracted. These projections are then depicted through a functional and mobile visualization interface, providing visual information of the internal structure of the traffic data. The interface makes MOVCIDS accessible from any mobile device to give more accessibility to network administrators, enabling continuous visualization, monitoring and supervision of computer networks. Additionally, a novel testing technique has been developed to evaluate MOVCIDS and other IDSs employing numerical datasets. To show the performance and validate the proposed IDS, it has been tested in different real domains containing several attacks and anomalous situations. In addition, the importance of the temporal dimension on intrusion detection, and the ability of this IDS to process it, are emphasized in this workJunta de Castilla and Leon project BU006A08, Business intelligence for production within the framework of the Instituto Tecnologico de Cas-tilla y Leon (ITCL) and the Agencia de Desarrollo Empresarial (ADE), and the Spanish Ministry of Education and Innovation project CIT-020000-2008-2. The authors would also like to thank the vehicle interior manufacturer, Grupo Antolin Ingenieria S. A., within the framework of the project MAGNO2008-1028-CENIT Project funded by the Spanish Government

RT-MOVICAB-IDS: Addressing real-time intrusion detection

This study presents a novel Hybrid Intelligent Intrusion Detection System (IDS) known as RT-MOVICAB-IDS that incorporates temporal control. One of its main goals is to facilitate real-time Intrusion Detection, as accurate and swift responses are crucial in this field, especially if automatic abortion mechanisms are running. The formulation of this hybrid IDS combines Artificial Neural Networks (ANN) and Case-Based Reasoning (CBR) within a Multi-Agent System (MAS) to detect intrusions in dynamic computer networks. Temporal restrictions are imposed on this IDS, in order to perform real/execution time processing and assure system response predictability. Therefore, a dynamic real-time multi-agent architecture for IDS is proposed in this study, allowing the addition of predictable agents (both reactive and deliberative). In particular, two of the deliberative agents deployed in this system incorporate temporal-bounded CBR. This upgraded CBR is based on an anytime approximation, which allows the adaptation of this Artificial Intelligence paradigm to real-time requirements. Experimental results using real data sets are presented which validate the performance of this novel hybrid IDSMinisterio de Economía y Competitividad (TIN2010-21272-C02-01, TIN2009-13839-C03-01), Ministerio de Ciencia e Innovación (CIT-020000-2008-2, CIT-020000-2009-12

Deliverable JRA1.1: Evaluation of current network control and management planes for multi-domain network infrastructure

This deliverable includes a compilation and evaluation of available control and management architectures and protocols applicable to a multilayer infrastructure in a multi-domain Virtual Network environment.The scope of this deliverable is mainly focused on the virtualisation of the resources within a network and at processing nodes. The virtualization of the FEDERICA infrastructure allows the provisioning of its available resources to users by means of FEDERICA slices. A slice is seen by the user as a real physical network under his/her domain, however it maps to a logical partition (a virtual instance) of the physical FEDERICA resources. A slice is built to exhibit to the highest degree all the principles applicable to a physical network (isolation, reproducibility, manageability, ...). Currently, there are no standard definitions available for network virtualization or its associated architectures. Therefore, this deliverable proposes the Virtual Network layer architecture and evaluates a set of Management- and Control Planes that can be used for the partitioning and virtualization of the FEDERICA network resources. This evaluation has been performed taking into account an initial set of FEDERICA requirements; a possible extension of the selected tools will be evaluated in future deliverables. The studies described in this deliverable define the virtual architecture of the FEDERICA infrastructure. During this activity, the need has been recognised to establish a new set of basic definitions (taxonomy) for the building blocks that compose the so-called slice, i.e. the virtual network instantiation (which is virtual with regard to the abstracted view made of the building blocks of the FEDERICA infrastructure) and its architectural plane representation. These definitions will be established as a common nomenclature for the FEDERICA project. Other important aspects when defining a new architecture are the user requirements. It is crucial that the resulting architecture fits the demands that users may have. Since this deliverable has been produced at the same time as the contact process with users, made by the project activities related to the Use Case definitions, JRA1 has proposed a set of basic Use Cases to be considered as starting point for its internal studies. When researchers want to experiment with their developments, they need not only network resources on their slices, but also a slice of the processing resources. These processing slice resources are understood as virtual machine instances that users can use to make them behave as software routers or end nodes, on which to download the software protocols or applications they have produced and want to assess in a realistic environment. Hence, this deliverable also studies the APIs of several virtual machine management software products in order to identify which best suits FEDERICA’s needs.Postprint (published version

Visualising Network Traffic Data From AirTraffic Control Radio Systems

In recent years the aviation industry has begun to embrace digital technology forAir Traffic Control (ATC) radio systems. This change has created challenges not onlyfor the industry but also for personnel. However, this implementation offers manyimprovements over older systems; more precise control, straightforward integrationwith other ATC systems and a more efficient way to provide software updates. Thechallenge for personnel is to develop a new skillset enabling a learning transitionfrom analogue to digital systems, with a specific emphasis on computer networkingskills.This project was undertaken in collaboration between the University of Lincoln(UoL) and Park Air Systems (PAS), an industry-leading provider of Air-Space com-munication solutions. A system has been developed to find a mechanism to monitorand visualise network traffic. The use of graphs provides a direct interface for theend-users, enabling a mechanism for identifying performance issues to meet thetransitional challenges from analogue to digital. An easy to use interface has beendesigned, which will enable non-technical users to interact effectively with the sys-tem.Considerable testing was undertaken to investigate the system usability concern-ing the practical application for users with limited networking experience. A surveyprovided a range of quantitative and qualitative data which was further analysed toscrutinize user perspectives on system usability. This involved engineers from PASand postgraduate students from UoL to compare results between direct industrypersonnel and unaffiliated participants

Interpretable Learning in Multivariate Big Data Analysis for Network Monitoring

There is an increasing interest in the development of new data-driven models useful to assess the performance of communication networks. For many applications, like network monitoring and troubleshooting, a data model is of little use if it cannot be interpreted by a human operator. In this paper, we present an extension of the Multivariate Big Data Analysis (MBDA) methodology, a recently proposed interpretable data analysis tool. In this extension, we propose a solution to the automatic derivation of features, a cornerstone step for the application of MBDA when the amount of data is massive. The resulting network monitoring approach allows us to detect and diagnose disparate network anomalies, with a data-analysis workflow that combines the advantages of interpretable and interactive models with the power of parallel processing. We apply the extended MBDA to two case studies: UGR\u2716, a benchmark flow-based real-traffic dataset for anomaly detection, and Dartmouth\u2718, the longest and largest Wi-Fi trace known to date

A Unified Monitoring Framework for Energy Consumption and Network Traffic

International audienceProviding experimenters with deep insight about the effects of theirexperiments is a central feature of testbeds. In this paper, wedescribe Kwapi, a framework designed in the context of the Grid'5000testbed, that unifies measurements for both energy consumption andnetwork traffic. Because all measurements are taken at theinfrastructure level (using sensors in power and network equipment),using this framework has no dependencies on the experiments themselves.Initially designed for OpenStack infrastructures, the Kwapi framework allowsmonitoring and reporting of energy consumption of distributed platforms. Inthis article, we present the extension of Kwapi to network monitoring, andoutline how we overcame several challenges: scaling to a testbed the size ofGrid'5000 while still providing high-frequency measurements; providing long-termloss-less storage of measurements; handling operational issues when deployingsuch a tool on a real infrastructure

Network anomalies detection via event analysis and correlation by a smart system

The multidisciplinary of contemporary societies compel us to look at Information Technology (IT) systems as one of the most significant grants that we can remember. However, its increase implies a mandatory security force for users, a force in the form of effective and robust tools to combat cybercrime to which users, individual or collective, are ex-posed almost daily. Monitoring and detection of this kind of problem must be ensured in real-time, allowing companies to intervene fruitfully, quickly and in unison. The proposed framework is based on an organic symbiosis between credible, affordable, and effective open-source tools for data analysis, relying on Security Information and Event Management (SIEM), Big Data and Machine Learning (ML) techniques commonly applied for the development of real-time monitoring systems. Dissecting this framework, it is composed of a system based on SIEM methodology that provides monitoring of data in real-time and simultaneously saves the information, to assist forensic investigation teams. Secondly, the application of the Big Data concept is effective in manipulating and organising the flow of data. Lastly, the use of ML techniques that help create mechanisms to detect possible attacks or anomalies on the network. This framework is intended to provide a real-time analysis application in the institution ISCTE – Instituto Universitário de Lisboa (Iscte), offering a more complete, efficient, and secure monitoring of the data from the different devices comprising the network.A multidisciplinaridade das sociedades contemporâneas obriga-nos a perspetivar os sistemas informáticos como uma das maiores dádivas de que há memória. Todavia o seu incremento implica uma mandatária força de segurança para utilizadores, força essa em forma de ferramentas eficazes e robustas no combate ao cibercrime a que os utilizadores, individuais ou coletivos, são sujeitos quase diariamente. A monitorização e deteção deste tipo de problemas tem de ser assegurada em tempo real, permitindo assim, às empresas intervenções frutuosas, rápidas e em uníssono. A framework proposta é alicerçada numa simbiose orgânica entre ferramentas open source credíveis, acessíveis pecuniariamente e eficazes na monitorização de dados, recorrendo a um sistema baseado em técnicas de Security Information and Event Management (SIEM), Big Data e Machine Learning (ML) comumente aplicadas para a criação de sistemas de monitorização em tempo real. Dissecando esta framework, é composta pela metodologia SIEM que possibilita a monitorização de dados em tempo real e em simultâneo guardar a informação, com o objetivo de auxiliar as equipas de investigação forense. Em segundo lugar, a aplicação do conceito Big Data eficaz na manipulação e organização do fluxo dos dados. Por último, o uso de técnicas de ML que ajudam a criação de mecanismos de deteção de possíveis ataques ou anomalias na rede. Esta framework tem como objetivo uma aplicação de análise em tempo real na instituição ISCTE – Instituto Universitário de Lisboa (Iscte), apresentando uma monitorização mais completa, eficiente e segura dos dados dos diversos dispositivos presentes na mesma

Distributed control of reconfigurable mobile network agents for resource coordination

Includes abstract.Includes bibliographical references.Considering the tremendous growth of internet applications and network resource federation proposed towards future open access network (FOAN), the need to analyze the robustness of the classical signalling mechanisms across multiple network operators cannot be over-emphasized. It is envisaged, there will be additional challenges in meeting the bandwidth requirements and network management...The first objective of this project is to describe the networking environment based on the support for heterogeneity of network components..