Big Data and Analysis of Data Transfers for International Research Networks Using NetSage

Modern science is increasingly data-driven and collaborative in nature. Many scientific disciplines, including genomics, high-energy physics, astronomy, and atmospheric science, produce petabytes of data that must be shared with collaborators all over the world. The National Science Foundation-supported International Research Network Connection (IRNC) links have been essential to enabling this collaboration, but as data sharing has increased, so has the amount of information being collected to understand network performance. New capabilities to measure and analyze the performance of international wide-area networks are essential to ensure end-users are able to take full advantage of such infrastructure for their big data applications. NetSage is a project to develop a unified, open, privacy-aware network measurement, and visualization service to address the needs of monitoring today's high-speed international research networks. NetSage collects data on both backbone links and exchange points, which can be as much as 1Tb per month. This puts a significant strain on hardware, not only in terms storage needs to hold multi-year historical data, but also in terms of processor and memory needs to analyze the data to understand network behaviors. This paper addresses the basic NetSage architecture, its current data collection and archiving approach, and details the constraints of dealing with this big data problem of handling vast amounts of monitoring data, while providing useful, extensible visualization to end users

Bridging the gap of network management and anomaly detection through interactive visualization

Large-scale networks have become increasingly challenging to manage. It is vital for a system administrator or network manager to be able to analyze the vast amount of log data in order to detect suspicious behaviors or patterns, possibly due to malicious users/applications or faulty devices. While an intrusion detection system (IDS) log can provide a large number of warnings, exactly which alarms are true while the others are false, and more importantly what are the underlying causes are still difficult to know. To bridge the gap between network log and anomaly discovery, we design and implement a visualization tool that combines multiple commodity visualizations with minimum learning curve. While each individual view is well understood, the effects of such views in analyzing network anomalies are not well studied. Since each visualization technique has advantages as well as limitations in addressing a particular task, we show that these views, when combined and linked together, may provide an effective and lightweight network anomaly analysis tool. The web-based open platform may simplify network administration as well as promote collaborative analysis among researchers. © 2014 IEEE.Large-scale networks have become increasingly challenging to manage. It is vital for a system administrator or network manager to be able to analyze the vast amount of log data in order to detect suspicious behaviors or patterns, possibly due to malicious users/applications or faulty devices. While an intrusion detection system (IDS) log can provide a large number of warnings, exactly which alarms are true while the others are false, and more importantly what are the underlying causes are still difficult to know. To bridge the gap between network log and anomaly discovery, we design and implement a visualization tool that combines multiple commodity visualizations with minimum learning curve. While each individual view is well understood, the effects of such views in analyzing network anomalies are not well studied. Since each visualization technique has advantages as well as limitations in addressing a particular task, we show that these views, when combined and linked together, may provide an effective and lightweight network anomaly analysis tool. The web-based open platform may simplify network administration as well as promote collaborative analysis among researchers. © 2014 IEEE