30 research outputs found
Large Scale Enrichment and Statistical Cyber Characterization of Network Traffic
Modern network sensors continuously produce enormous quantities of raw data
that are beyond the capacity of human analysts. Cross-correlation of network
sensors increases this challenge by enriching every network event with
additional metadata. These large volumes of enriched network data present
opportunities to statistically characterize network traffic and quickly answer
a key question: "What are the primary cyber characteristics of my network
data?" The Python GraphBLAS and PyD4M analysis frameworks enable anonymized
statistical analysis to be performed quickly and efficiently on very large
network data sets. This approach is tested using billions of anonymized network
data samples from the largest Internet observatory (CAIDA Telescope) and tens
of millions of anonymized records from the largest commercially available
background enrichment capability (GreyNoise). The analysis confirms that most
of the enriched variables follow expected heavy-tail distributions and that a
large fraction of the network traffic is due to a small number of cyber
activities. This information can simplify the cyber analysts' task by enabling
prioritization of cyber activities based on statistical prevalence.Comment: 8 pages, 8 figures, HPE
Zero Botnets: An Observe-Pursue-Counter Approach
Adversarial Internet robots (botnets) represent a growing threat to the safe
use and stability of the Internet. Botnets can play a role in launching
adversary reconnaissance (scanning and phishing), influence operations
(upvoting), and financing operations (ransomware, market manipulation, denial
of service, spamming, and ad click fraud) while obfuscating tailored tactical
operations. Reducing the presence of botnets on the Internet, with the
aspirational target of zero, is a powerful vision for galvanizing policy
action. Setting a global goal, encouraging international cooperation, creating
incentives for improving networks, and supporting entities for botnet takedowns
are among several policies that could advance this goal. These policies raise
significant questions regarding proper authorities/access that cannot be
answered in the abstract. Systems analysis has been widely used in other
domains to achieve sufficient detail to enable these questions to be dealt with
in concrete terms. Defeating botnets using an observe-pursue-counter
architecture is analyzed, the technical feasibility is affirmed, and the
authorities/access questions are significantly narrowed. Recommended next steps
include: supporting the international botnet takedown community, expanding
network observatories, enhancing the underlying network science at scale,
conducting detailed systems analysis, and developing appropriate policy
frameworks.Comment: 26 pages, 13 figures, 2 tables, 72 references, submitted to PlosOn
Multi-Temporal Analysis and Scaling Relations of 100,000,000,000 Network Packets
Our society has never been more dependent on computer networks. Effective
utilization of networks requires a detailed understanding of the normal
background behaviors of network traffic. Large-scale measurements of networks
are computationally challenging. Building on prior work in interactive
supercomputing and GraphBLAS hypersparse hierarchical traffic matrices, we have
developed an efficient method for computing a wide variety of streaming network
quantities on diverse time scales. Applying these methods to 100,000,000,000
anonymized source-destination pairs collected at a network gateway reveals many
previously unobserved scaling relationships. These observations provide new
insights into normal network background traffic that could be used for anomaly
detection, AI feature engineering, and testing theoretical models of streaming
networks.Comment: 6 pages, 6 figures,3 tables, 49 references, accepted to IEEE HPEC
202
Mixed Criteria Packet Scheduling
Abstract. Packet scheduling in networks with quality of service constraints has been extensively studied as a single criterion scheduling problem. The assumption underlying single criterion packet scheduling is that the value of all packets can be normalized to a single scale, even in cases when packets have different requirements. We demonstrate that this approach can lead to inefficient utilization of network resources. To improve network efficiency, we model packet scheduling as a mixed criteria scheduling problem where there are two distinct sets of jobs: deadline jobs which represent real-time packets in a network and flow jobs which represent other packets in the network. As the names imply, the jobs in these two sets differ by the criteria associated with them. For this problem, the flow jobs are scheduled to minimize the sum of their flow times, and the deadline jobs are scheduled to maximize the value of jobs that complete by their deadlines. We demonstrate that even when there is only a single deadline job, this mixed criteria scheduling problem is NP-Complete. We give a polynomial time optimal algorithm Slacker for the variant where all jobs have unit size and the value of deadline jobs processed by the deadline must be maximized. Given this constraint Slacker minimizes the total flow time. Furthermore, we show that online Slacker is optimal for flow time while being 2-competitive with respect to the deadline jobs when compared to an optimal algorithm like Slacker that maximizes the value of deadline jobs.
Firewall compressor: An algorithm for minimizing firewall policies
Abstract—A firewall is a security guard placed between a private network and the outside Internet that monitors all incoming and outgoing packets. The function of a firewall is to examine every packet and decide whether to accept or discard it based upon the firewall’s policy. This policy is specified as a sequence of (possibly conflicting) rules. When a packet comes to a firewall, the firewall searches for the first rule that the packet matches, and executes the decision of that rule. With the explosive growth of Internet-based applications and malicious attacks, the number of rules in firewalls have been increasing rapidly, which consequently degrades network performance and throughput. In this paper, we propose Firewall Compressor, a framework that can significantly reduce the number of rules in a firewall while keeping the semantics of the firewall unchanged. We make three major contributions in this paper. First, we propose an optimal solution using dynamic programming techniques for compressing one-dimensional firewalls. Second, we present a systematic approach to compressing multi-dimensional firewalls. Last, we conducted extensive experiments to evaluate Firewall Compressor. In terms of effectiveness, Firewall Compressor achieves an average compression ratio of 52.3 % on reallife rule sets. In terms of efficiency, Firewall Compressor runs in seconds even for a large firewall with thousands of rules. Moreover, the algorithms and techniques proposed in this paper are not limited to firewalls. Rather, they can be applied to other rule-based systems such as packet filters on Internet routers. I
TCAM Razor: A Systematic Approach Towards Minimizing Packet Classifiers in TCAMs
Abstract — Packet classification is the core mechanism that enables many networking services on the Internet such as firewall packet filtering and traffic accounting. Using Ternary Content Addressable Memories (TCAMs) to perform high-speed packet classification has become the de facto standard in industry. TCAMs classify packets in constant time by comparing a packet with all classification rules of ternary encoding in parallel. Despite their high speed, TCAMs suffer from the well-known range expansion problem. As packet classification rules usually have fields specified as ranges, converting such rules to TCAMcompatible rules may result in an explosive increase in the number of rules. This is not a problem if TCAMs have large capacities. Unfortunately, TCAMs have very limited capacity, and more rules means more power consumption and more heat generation for TCAMs. Even worse, the number of rules in packet classifiers have been increasing rapidly with the growing number of services deployed on the internet. To address the range expansion problem of TCAMs, we consider the following problem: given a packet classifier, how can we generate another semantically equivalent packet classifier that requires the least number of TCAM entries? In this paper, we propose a systematic approach, the TCAM Razor, that is effective, efficient, and practical. In terms of effectiveness, our TCAM Razor prototype achieves a total compression ratio of 3.9%, which is significantly better than the previously published best result of 54%. In terms of efficiency, our TCAM Razor prototype runs in seconds, even for large packet classifiers. Finally, in terms of practicality, our TCAM Razor approach can be easily deployed as it does not require any modification to existing packet classification systems, unlike many previous range expansion solutions. I