Matrix profile data mining for BGP anomaly detection

The Border Gateway Protocol (BGP), acting as the communication protocol that binds the Internet, remains vulnerable despite Internet security advancements. This is not surprising, as the Internet was not designed to be resilient to cyber-attacks, therefore the detection of anomalous activity was not of prime importance to the Internet creators. Detection of BGP anomalies can potentially provide network operators with an early warning system to focus on protecting networks, systems, and infrastructure from significant impact, improve security posture and resilience, while ultimately contributing to a secure global Internet environment. In this paper, we present a novel technique for the detection of BGP anomalies in different events. This research uses publicly available datasets of BGP messages collected from the repositories, Route Views and Réseaux IP Européens (RIPE). Our contribution is the application of a time series data mining approach, Matrix Profile (MP), to detect BGP anomalies in all categories of BGP events. Advantages of the MP detection technique compared to extant approaches include that it is domain agnostic, is assumption-free, requires few parameters, does not require training data, and is scalable and storage efficient. The single hyper-parameter analyzed in MP shows it is robust to change. Our results indicate the MP detection scheme is competitive against existing detection schemes. A novel BGP anomaly detection scheme is also proposed for further research and validation

A Survey, Taxonomy, and Analysis of Network Security Visualization Techniques

Network security visualization is a relatively new field and is quickly gaining momentum. Network security visualization allows the display and projection of the network or system data, in hope to efficiently monitor and protect the system from any intrusions or possible attacks. Intrusions and attacks are constantly continuing to increase in number, size, and complexity. Textually reading through log files or other textual sources is currently insufficient to secure a network or system. Using graphical visualization, security information is presented visually, and not only by text. Without network security visualization, reading through log files or other textual sources is an endless and aggravating task for network security analysts. Visualization provides a method of displaying large volume of information in a relatively small space. It also makes patterns easier to detect, recognize, and analyze. This can help security experts to detect problems that may otherwise be missed in reading text based log files. Network security visualization has become an active research field in the past six years and a large number of visualization techniques have been proposed. A comprehensive analysis of the existing techniques is needed to help network security designers make informed decisions about the appropriate visualization techniques under various circumstances. Moreover, a taxonomy of the existing visualization techniques is needed to classify the existing network security visualization techniques and present a high level overview of the field. In this thesis, the author surveyed the field of network security visualization. Specifically, the author analyzed the network security visualization techniques from the perspective of data model, visual primitives, security analysis tasks, user interaction, and other design issues. Various statistics were generated from the literatures. Based on this analysis, the author has attempted to generate useful guidelines and principles for designing effective network security visualization techniques. The author also proposed a taxonomy for the security visualization techniques. To the author’s knowledge, this is the first attempt to generate a taxonomy for network security visualization. Finally, the author evaluated the existing network security visualization techniques and discussed their characteristics and limitations. For future research, the author also discussed some open research problems in this field. This research is a step towards a thorough analysis of the problem space and the solution space in network security visualization

Cybersecurity knowledge graphs

Cybersecurity knowledge graphs, which represent cyber-knowledge with a graph-based data model, provide holistic approaches for processing massive volumes of complex cybersecurity data derived from diverse sources. They can assist security analysts to obtain cyberthreat intelligence, achieve a high level of cyber-situational awareness, discover new cyber-knowledge, visualize networks, data flow, and attack paths, and understand data correlations by aggregating and fusing data. This paper reviews the most prominent graph-based data models used in this domain, along with knowledge organization systems that define concepts and properties utilized in formal cyber-knowledge representation for both background knowledge and specific expert knowledge about an actual system or attack. It is also discussed how cybersecurity knowledge graphs enable machine learning and facilitate automated reasoning over cyber-knowledge

VISUALIZATION OF NETWORK TRAFFIC

Network security in today’s world is critical. System administrators must quickly perceive the security state of their networks, but they often have only text-based tools to work with. These tools often provide no overview that would help users grasp the big-picture.Network traffic visualization tools have successfully enabled security analysts to understand the nature of traffic present in a network. However these tools rely mainly on human expertise to discover anomalies in traffic and attack patterns.The visualization capability provided by Visualization tool allows an operator to assess the state of a large and complex network given an overall view of the entire network and filter/drill-down features with a friendly user interface that allows users to request more detailed information of interest such as specific protocol traffic flows.Visualization tool allows operators to detect and investigate anomalous internal and external network traffic. Visualization tool shows network events graphically. We model the network as a graph with hosts being nodes and traffic being flows on edges. We present a detailed description of Visualization tool functionality and demonstrate its application to traffic dynamics in order to monitor, discover, and investigate security-relevant events

Interactive visualization of event logs for cybersecurity

Hidden cyber threats revealed with new visualization software Eventpa

Machine Learning and Big Data Methodologies for Network Traffic Monitoring

Over the past 20 years, the Internet saw an exponential grown of traffic, users, services and applications. Currently, it is estimated that the Internet is used everyday by more than 3.6 billions users, who generate 20 TB of traffic per second. Such a huge amount of data challenge network managers and analysts to understand how the network is performing, how users are accessing resources, how to properly control and manage the infrastructure, and how to detect possible threats. Along with mathematical, statistical, and set theory methodologies machine learning and big data approaches have emerged to build systems that aim at automatically extracting information from the raw data that the network monitoring infrastructures offer. In this thesis I will address different network monitoring solutions, evaluating several methodologies and scenarios. I will show how following a common workflow, it is possible to exploit mathematical, statistical, set theory, and machine learning methodologies to extract meaningful information from the raw data. Particular attention will be given to machine learning and big data methodologies such as DBSCAN, and the Apache Spark big data framework. The results show that despite being able to take advantage of mathematical, statistical, and set theory tools to characterize a problem, machine learning methodologies are very useful to discover hidden information about the raw data. Using DBSCAN clustering algorithm, I will show how to use YouLighter, an unsupervised methodology to group caches serving YouTube traffic into edge-nodes, and latter by using the notion of Pattern Dissimilarity, how to identify changes in their usage over time. By using YouLighter over 10-month long races, I will pinpoint sudden changes in the YouTube edge-nodes usage, changes that also impair the end users’ Quality of Experience. I will also apply DBSCAN in the deployment of SeLINA, a self-tuning tool implemented in the Apache Spark big data framework to autonomously extract knowledge from network traffic measurements. By using SeLINA, I will show how to automatically detect the changes of the YouTube CDN previously highlighted by YouLighter. Along with these machine learning studies, I will show how to use mathematical and set theory methodologies to investigate the browsing habits of Internauts. By using a two weeks dataset, I will show how over this period, the Internauts continue discovering new websites. Moreover, I will show that by using only DNS information to build a profile, it is hard to build a reliable profiler. Instead, by exploiting mathematical and statistical tools, I will show how to characterize Anycast-enabled CDNs (A-CDNs). I will show that A-CDNs are widely used either for stateless and stateful services. That A-CDNs are quite popular, as, more than 50% of web users contact an A-CDN every day. And that, stateful services, can benefit of A-CDNs, since their paths are very stable over time, as demonstrated by the presence of only a few anomalies in their Round Trip Time. Finally, I will conclude by showing how I used BGPStream an open-source software framework for the analysis of both historical and real-time Border Gateway Protocol (BGP) measurement data. By using BGPStream in real-time mode I will show how I detected a Multiple Origin AS (MOAS) event, and how I studies the black-holing community propagation, showing the effect of this community in the network. Then, by using BGPStream in historical mode, and the Apache Spark big data framework over 16 years of data, I will show different results such as the continuous growth of IPv4 prefixes, and the growth of MOAS events over time. All these studies have the aim of showing how monitoring is a fundamental task in different scenarios. In particular, highlighting the importance of machine learning and of big data methodologies

A Survey on Big Data for Network Traffic Monitoring and Analysis

Network Traffic Monitoring and Analysis (NTMA) represents a key component for network management, especially to guarantee the correct operation of large-scale networks such as the Internet. As the complexity of Internet services and the volume of traffic continue to increase, it becomes difficult to design scalable NTMA applications. Applications such as traffic classification and policing require real-time and scalable approaches. Anomaly detection and security mechanisms require to quickly identify and react to unpredictable events while processing millions of heterogeneous events. At last, the system has to collect, store, and process massive sets of historical data for post-mortem analysis. Those are precisely the challenges faced by general big data approaches: Volume, Velocity, Variety, and Veracity. This survey brings together NTMA and big data. We catalog previous work on NTMA that adopt big data approaches to understand to what extent the potential of big data is being explored in NTMA. This survey mainly focuses on approaches and technologies to manage the big NTMA data, additionally briefly discussing big data analytics (e.g., machine learning) for the sake of NTMA. Finally, we provide guidelines for future work, discussing lessons learned, and research directions

MDFRCNN: Malware Detection using Faster Region Proposals Convolution Neural Network

Technological advancement of smart devices has opened up a new trend: Internet of Everything (IoE), where all devices are connected to the web. Large scale networking benefits the community by increasing connectivity and giving control of physical devices. On the other hand, there exists an increased ‘Threat’ of an ‘Attack’. Attackers are targeting these devices, as it may provide an easier ‘backdoor entry to the users’ network’.MALicious softWARE (MalWare) is a major threat to user security. Fast and accurate detection of malware attacks are the sine qua non of IoE, where large scale networking is involved. The paper proposes use of a visualization technique where the disassembled malware code is converted into gray images, as well as use of Image Similarity based Statistical Parameters (ISSP) such as Normalized Cross correlation (NCC), Average difference (AD), Maximum difference (MaxD), Singular Structural Similarity Index Module (SSIM), Laplacian Mean Square Error (LMSE), MSE and PSNR. A vector consisting of gray image with statistical parameters is trained using a Faster Region proposals Convolution Neural Network (F-RCNN) classifier. The experiment results are promising as the proposed method includes ISSP with F-RCNN training. Overall training time of learning the semantics of higher-level malicious behaviors is less. Identification of malware (testing phase) is also performed in less time. The fusion of image and statistical parameter enhances system performance with greater accuracy. The benchmark database from Microsoft Malware Classification challenge has been used to analyze system performance, which is available on the Kaggle website. An overall average classification accuracy of 98.12% is achieved by the proposed method