11 research outputs found
Hypersparse Neural Network Analysis of Large-Scale Internet Traffic
The Internet is transforming our society, necessitating a quantitative
understanding of Internet traffic. Our team collects and curates the largest
publicly available Internet traffic data containing 50 billion packets.
Utilizing a novel hypersparse neural network analysis of "video" streams of
this traffic using 10,000 processors in the MIT SuperCloud reveals a new
phenomena: the importance of otherwise unseen leaf nodes and isolated links in
Internet traffic. Our neural network approach further shows that a
two-parameter modified Zipf-Mandelbrot distribution accurately describes a wide
variety of source/destination statistics on moving sample windows ranging from
100,000 to 100,000,000 packets over collections that span years and continents.
The inferred model parameters distinguish different network streams and the
model leaf parameter strongly correlates with the fraction of the traffic in
different underlying network topologies. The hypersparse neural network
pipeline is highly adaptable and different network statistics and training
models can be incorporated with simple changes to the image filter functions.Comment: 11 pages, 10 figures, 3 tables, 60 citations; to appear in IEEE High
Performance Extreme Computing (HPEC) 201
Deployment of Real-Time Network Traffic Analysis using GraphBLAS Hypersparse Matrices and D4M Associative Arrays
Matrix/array analysis of networks can provide significant insight into their
behavior and aid in their operation and protection. Prior work has demonstrated
the analytic, performance, and compression capabilities of GraphBLAS
(graphblas.org) hypersparse matrices and D4M (d4m.mit.edu) associative arrays
(a mathematical superset of matrices). Obtaining the benefits of these
capabilities requires integrating them into operational systems, which comes
with its own unique challenges. This paper describes two examples of real-time
operational implementations. First, is an operational GraphBLAS implementation
that constructs anonymized hypersparse matrices on a high-bandwidth network
tap. Second, is an operational D4M implementation that analyzes daily cloud
gateway logs. The architectures of these implementations are presented.
Detailed measurements of the resources and the performance are collected and
analyzed. The implementations are capable of meeting their operational
requirements using modest computational resources (a couple of processing
cores). GraphBLAS is well-suited for low-level analysis of high-bandwidth
connections with relatively structured network data. D4M is well-suited for
higher-level analysis of more unstructured data. This work demonstrates that
these technologies can be implemented in operational settings.Comment: Accepted to IEEE HPEC, 8 pages, 8 figures, 1 table, 69 references.
arXiv admin note: text overlap with arXiv:2203.13934. text overlap with
arXiv:2309.0180
Focusing and Calibration of Large Scale Network Sensors using GraphBLAS Anonymized Hypersparse Matrices
Defending community-owned cyber space requires community-based efforts.
Large-scale network observations that uphold the highest regard for privacy are
key to protecting our shared cyberspace. Deployment of the necessary network
sensors requires careful sensor placement, focusing, and calibration with
significant volumes of network observations. This paper demonstrates novel
focusing and calibration procedures on a multi-billion packet dataset using
high-performance GraphBLAS anonymized hypersparse matrices. The run-time
performance on a real-world data set confirms previously observed real-time
processing rates for high-bandwidth links while achieving significant data
compression. The output of the analysis demonstrates the effectiveness of these
procedures at focusing the traffic matrix and revealing the underlying stable
heavy-tail statistical distributions that are necessary for anomaly detection.
A simple model of the corresponding probability of detection () and
probability of false alarm () for these distributions highlights
the criticality of network sensor focusing and calibration. Once a sensor is
properly focused and calibrated it is then in a position to carry out two of
the central tenets of good cybersecurity: (1) continuous observation of the
network and (2) minimizing unbrokered network connections.Comment: Accepted to IEEE HPEC, 9 pages, 12 figures, 1 table, 63 references, 2
appendice
Streaming 1.9 Billion Hypersparse Network Updates per Second with D4M
The Dynamic Distributed Dimensional Data Model (D4M) library implements
associative arrays in a variety of languages (Python, Julia, and Matlab/Octave)
and provides a lightweight in-memory database implementation of hypersparse
arrays that are ideal for analyzing many types of network data. D4M relies on
associative arrays which combine properties of spreadsheets, databases,
matrices, graphs, and networks, while providing rigorous mathematical
guarantees, such as linearity. Streaming updates of D4M associative arrays put
enormous pressure on the memory hierarchy. This work describes the design and
performance optimization of an implementation of hierarchical associative
arrays that reduces memory pressure and dramatically increases the update rate
into an associative array. The parameters of hierarchical associative arrays
rely on controlling the number of entries in each level in the hierarchy before
an update is cascaded. The parameters are easily tunable to achieve optimal
performance for a variety of applications. Hierarchical arrays achieve over
40,000 updates per second in a single instance. Scaling to 34,000 instances of
hierarchical D4M associative arrays on 1,100 server nodes on the MIT SuperCloud
achieved a sustained update rate of 1,900,000,000 updates per second. This
capability allows the MIT SuperCloud to analyze extremely large streaming
network data sets.Comment: 6 pages; 6 figures; accepted to IEEE High Performance Extreme
Computing (HPEC) Conference 2019. arXiv admin note: text overlap with
arXiv:1807.05308, arXiv:1902.0084
Large Scale Enrichment and Statistical Cyber Characterization of Network Traffic
Modern network sensors continuously produce enormous quantities of raw data
that are beyond the capacity of human analysts. Cross-correlation of network
sensors increases this challenge by enriching every network event with
additional metadata. These large volumes of enriched network data present
opportunities to statistically characterize network traffic and quickly answer
a key question: "What are the primary cyber characteristics of my network
data?" The Python GraphBLAS and PyD4M analysis frameworks enable anonymized
statistical analysis to be performed quickly and efficiently on very large
network data sets. This approach is tested using billions of anonymized network
data samples from the largest Internet observatory (CAIDA Telescope) and tens
of millions of anonymized records from the largest commercially available
background enrichment capability (GreyNoise). The analysis confirms that most
of the enriched variables follow expected heavy-tail distributions and that a
large fraction of the network traffic is due to a small number of cyber
activities. This information can simplify the cyber analysts' task by enabling
prioritization of cyber activities based on statistical prevalence.Comment: 8 pages, 8 figures, HPE
GNN4IFA: Interest Flooding Attack Detection With Graph Neural Networks
In the context of Information-Centric Networking, Interest Flooding Attacks (IFAs) represent a new and dangerous sort of distributed denial of service. Since existing proposals targeting IFAs mainly focus on local information, in this paper we propose GNN4IFA as the first mechanism exploiting complex non-local knowledge for IFA detection by leveraging Graph Neural Networks (GNNs) handling the overall network topology. In order to test GNN4IFA, we collect SPOTIFAI, a novel dataset filling the current lack of available IFA datasets by covering a variety of IFA setups, including ?40 heterogeneous scenarios over three network topologies. We show that GNN4IFA performs well on all tested topologies and setups, reaching over 99% detection rate along with a negligible false positive rate and small computational costs. Overall, GNN4IFA overcomes state-of-the-art detection mechanisms both in terms of raw detection and flexibility, and – unlike all previous solutions in the literature – also enables the transfer of its detection on network topologies different from the one used in its design phase
Optimizing Sparse Linear Algebra on Reconfigurable Architecture
The rise of cloud computing and deep machine learning in recent years have led to a tremendous growth of workloads that are not only large, but also have highly sparse representations. A large fraction of machine learning problems are formulated as sparse linear algebra problems in which the entries in the matrices are mostly zeros. Not surprisingly, optimizing linear algebra algorithms to take advantage of this sparseness is critical for efficient computation on these large datasets.
This thesis presents a detailed analysis of several approaches to sparse matrix-matrix multiplication, a core computation of linear algebra kernels. While the arithmetic count of operations for the nonzero elements of the matrices are the same regardless of the algorithm used to perform matrix-matrix multiplication, there is significant variation in the overhead of navigating the sparse data structures to match the nonzero elements with the correct indices. This work explores approaches to minimize these overheads as well as the number of memory accesses for sparse matrices. To provide concrete numbers, the thesis examines inner product, outer product and row-wise algorithms on Transmuter, a flexible accelerator that can reconfigure its cache and crossbars at runtime to meet the demands of the program. This thesis shows how the reconfigurability of Transmuter can improve complex workloads that contain multiple kernels with varying compute and memory requirements, such as the Graphsage deep neural network and the Sinkhorn algorithm for optimal transport distance. Finally, we examine a novel Transmuter feature: register-to-register queues for efficient communication between its processing elements, enabling systolic array style computation for signal processing algorithms.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/169877/1/dohypark_1.pd