Sharing Computer Network Logs for Security and Privacy: A Motivation for New Methodologies of Anonymization

Logs are one of the most fundamental resources to any security professional. It is widely recognized by the government and industry that it is both beneficial and desirable to share logs for the purpose of security research. However, the sharing is not happening or not to the degree or magnitude that is desired. Organizations are reluctant to share logs because of the risk of exposing sensitive information to potential attackers. We believe this reluctance remains high because current anonymization techniques are weak and one-size-fits-all--or better put, one size tries to fit all. We must develop standards and make anonymization available at varying levels, striking a balance between privacy and utility. Organizations have different needs and trust other organizations to different degrees. They must be able to map multiple anonymization levels with defined risks to the trust levels they share with (would-be) receivers. It is not until there are industry standards for multiple levels of anonymization that we will be able to move forward and achieve the goal of widespread sharing of logs for security researchers.Comment: 17 pages, 1 figur

Cloud Computing Security, An Intrusion Detection System for Cloud Computing Systems

Cloud computing is widely considered as an attractive service model because it minimizes investment since its costs are in direct relation to usage and demand. However, the distributed nature of cloud computing environments, their massive resource aggregation, wide user access and efficient and automated sharing of resources enable intruders to exploit clouds for their advantage. To combat intruders, several security solutions for cloud environments adopt Intrusion Detection Systems. However, most IDS solutions are not suitable for cloud environments, because of problems such as single point of failure, centralized load, high false positive alarms, insufficient coverage for attacks, and inflexible design. The thesis defines a framework for a cloud based IDS to face the deficiencies of current IDS technology. This framework deals with threats that exploit vulnerabilities to attack the various service models of a cloud system. The framework integrates behaviour based and knowledge based techniques to detect masquerade, host, and network attacks and provides efficient deployments to detect DDoS attacks. This thesis has three main contributions. The first is a Cloud Intrusion Detection Dataset (CIDD) to train and test an IDS. The second is the Data-Driven Semi-Global Alignment, DDSGA, approach and three behavior based strategies to detect masquerades in cloud systems. The third and final contribution is signature based detection. We introduce two deployments, a distributed and a centralized one to detect host, network, and DDoS attacks. Furthermore, we discuss the integration and correlation of alerts from any component to build a summarized attack report. The thesis describes in details and experimentally evaluates the proposed IDS and alternative deployments. Acknowledgment: =============== • This PH.D. is achieved through an international joint program with a collaboration between University of Pisa in Italy (Department of Computer Science, Galileo Galilei PH.D. School) and University of Arizona in USA (College of Electrical and Computer Engineering). • The PHD topic is categorized in both Computer Engineering and Information Engineering topics. • The thesis author is also known as "Hisham A. Kholidy"

Towards Scalable Network Traffic Measurement With Sketches

Driven by the ever-increasing data volume through the Internet, the per-port speed of network devices reached 400 Gbps, and high-end switches are capable of processing 25.6 Tbps of network traffic. To improve the efficiency and security of the network, network traffic measurement becomes more important than ever. For fast and accurate traffic measurement, managing an accurate working set of active flows (WSAF) at line rates is a key challenge. WSAF is usually located in high-speed but expensive memories, such as TCAM or SRAM, and thus their capacity is quite limited. To scale up the per-flow measurement, we pursue three thrusts. In the first thrust, we propose to use In-DRAM WSAF and put a compact data structure (i.e., sketch) called FlowRegulator before WSAF to compensate for DRAM\u27s slow access time. Per our results, FlowRegulator can substantially reduce massive influxes to WSAF without compromising measurement accuracy. In the second thrust, we integrate our sketch into a network system and propose an SDN-based WLAN monitoring and management framework called RFlow+, which can overcome the limitations of existing traffic measurement solutions (e.g., OpenFlow and sFlow), such as a limited view, incomplete flow statistics, and poor trade-off between measurement accuracy and CPU/network overheads. In the third thrust, we introduce a novel sampling scheme to deal with the poor trade-off that is provided by the standard simple random sampling (SRS). Even though SRS has been widely used in practice because of its simplicity, it provides non-uniform sampling rates for different flows, because it samples packets over an aggregated data flow. Starting with a simple idea that independent per-flow packet sampling provides the most accurate estimation of each flow, we introduce a new concept of per-flow systematic sampling, aiming to provide the same sampling rate across all flows. In addition, we provide a concrete sampling method called SketchFlow, which approximates the idea of the per-flow systematic sampling using a sketch saturation event

Arhitektura sistema za prepoznavanje nepravilnosti u mrežnom saobraćaju zasnovano na analizi entropije

With the steady increase in reliance on computer networks in all aspects of life, computers and other connected devices have become more vulnerable to attacks, which exposes them to many major threats, especially in recent years. There are different systems to protect networks from these threats such as firewalls, antivirus programs, and data encryption, but it is still hard to provide complete protection for networks and their systems from the attacks, which are increasingly sophisticated with time. That is why it is required to use intrusion detection systems (IDS) on a large scale to be the second line of defense for computer and network systems along with other network security techniques. The main objective of intrusion detection systems is used to monitor network traffic and detect internal and external attacks. Intrusion detection systems represent an important focus of studies today, because most protection systems, no matter how good they are, can fail due to the emergence of new (unknown/predefined) types of intrusions. Most of the existing techniques detect network intrusions by collecting information about known types of attacks, so-called signature-based IDS, using them to recognize any attempt of attack on data or resources. The major problem of this approach is its inability to detect previously unknown attacks, even if these attacks are derived slightly from the known ones (the so-called zero-day attack). Also, it is powerless to detect encryption-related attacks. On the other hand, detecting abnormalities concerning conventional behavior (anomaly-based IDS) exceeds the abovementioned limitations. Many scientific studies have tended to build modern and smart systems to detect both known and unknown intrusions. In this research, an architecture that applies a new technique for IDS using an anomaly-based detection method based on entropy is introduced. Network behavior analysis relies on the profiling of legitimate network behavior in order to efficiently detect anomalous traffic deviations that indicate security threats. Entropy-based detection techniques are attractive due to their simplicity and applicability in real-time network traffic, with no need to train the system with labelled data. Besides the fact that the NetFlow protocol provides only a basic set of information about network communications, it is very beneficial for identifying zero-day attacks and suspicious behavior in traffic structure. Nevertheless, the challenge associated with limited NetFlow information combined with the simplicity of the entropy-based approach is providing an efficient and sensitive mechanism to detect a wide range of anomalies, including those of small intensity. However, a recent study found of generic entropy-based anomaly detection reports its vulnerability to deceit by introducing spoofed data to mask the abnormality. Furthermore, the majority of approaches for further classification of anomalies rely on machine learning, which brings additional complexity. Previously highlighted shortcomings and limitations of these approaches open up a space for the exploration of new techniques and methodologies for the detection of anomalies in network traffic in order to isolate security threats, which will be the main subject of the research in this thesis. Abstract An architrvture for network traffic anomaly detection system based on entropy analysis Page vii This research addresses all these issues by providing a systematic methodology with the main novelty in anomaly detection and classification based on the entropy of flow count and behavior features extracted from the basic data obtained by the NetFlow protocol. Two new approaches are proposed to solve these concerns. Firstly, an effective protection mechanism against entropy deception derived from the study of changes in several entropy types, such as Shannon, Rényi, and Tsallis entropies, as well as the measurement of the number of distinct elements in a feature distribution as a new detection metric. The suggested method improves the reliability of entropy approaches. Secondly, an anomaly classification technique was introduced to the existing entropy-based anomaly detection system. Entropy-based anomaly classification methods were presented and effectively confirmed by tests based on a multivariate analysis of the entropy changes of several features as well as aggregation by complicated feature combinations. Through an analysis of the most prominent security attacks, generalized network traffic behavior models were developed to describe various communication patterns. Based on a multivariate analysis of the entropy changes by anomalies in each of the modelled classes, anomaly classification rules were proposed and verified through the experiments. The concept of the behavior features is generalized, while the proposed data partitioning provides greater efficiency in real-time anomaly detection. The practicality of the proposed architecture for the implementation of effective anomaly detection and classification system in a general real-world network environment is demonstrated using experimental data

On modeling and mitigating new breed of dos attacks

Denial of Service (DoS) attacks pose serious threats to the Internet, exerting in tremendous impact on our daily lives that are heavily dependent on the good health of the Internet. This dissertation aims to achieve two objectives:1) to model new possibilities of the low rate DoS attacks; 2) to develop effective mitigation mechanisms to counter the threat from low rate DoS attacks. A new stealthy DDoS attack model referred to as the quiet attack is proposed in this dissertation. The attack traffic consists of TCP traffic only. Widely used botnets in today\u27s various attacks and newly introduced network feedback control are integral part of the quiet attack model. The quiet attack shows that short-lived TCP flows used as attack flows can be intentionally misused. This dissertation proposes another attack model referred to as the perfect storm which uses a combination of UDP and TCP. Better CAPTCHAs are highlighted as current defense against botnets to mitigate the quiet attack and the perfect storm. A novel time domain technique is proposed that relies on the time difference between subsequent packets of each flow to detect periodicity of the low rate DoS attack flow. An attacker can easily use different IP address spoofing techniques or botnets to launch a low rate DoS attack and fool the detection system. To mitigate such a threat, this dissertation proposes a second detection algorithm that detects the sudden increase in the traffic load of all the expired flows within a short period. In a network rate DoS attacks, it is shown that the traffic load of all the expired flows is less than certain thresholds, which are derived from real Internet traffic analysis. A novel filtering scheme is proposed to drop the low rate DoS attack packets. The simulation results confirm attack mitigation by using proposed technique. Future research directions will be briefly discussed

Fine-grained, Content-agnostic Network Traffic Analysis for Malicious Activity Detection

The rapid evolution of malicious activities in network environments necessitates the development of more effective and efficient detection and mitigation techniques. Traditional traffic analysis (TA) approaches have demonstrated limited efficacy and performance in detecting various malicious activities, resulting in a pressing need for more advanced solutions. To fill the gap, this dissertation proposes several new fine-grained network traffic analysis (FGTA) approaches. These approaches focus on (1) detecting previously hard-to-detect malicious activities by deducing fine-grained, detailed application-layer information in privacy-preserving manners, (2) enhancing usability by providing more explainable results and better adaptability to different network environments, and (3) combining network traffic data with endpoint information to provide users with more comprehensive and accurate protections. We begin by conducting a comprehensive survey of existing FGTA approaches. We then propose CJ-Sniffer, a privacy-aware cryptojacking detection system that efficiently detects cryptojacking traffic. CJ-Sniffer is the first approach to distinguishing cryptojacking traffic from user-initiated cryptocurrency mining traffic, allowing for fine-grained traffic discrimination. This level of fine-grained traffic discrimination has proven challenging to accomplish through traditional TA methodologies. Next, we introduce BotFlowMon, a learning-based, content-agnostic approach for detecting online social network (OSN) bot traffic, which has posed a significant challenge for detection using traditional TA strategies. BotFlowMon is an FGTA approach that relies only on content-agnostic flow-level data as input and utilizes novel algorithms and techniques to classify social bot traffic from real OSN user traffic. To enhance the usability of FGTA-based attack detection, we propose a learning-based DDoS detection approach that emphasizes both explainability and adaptability. This approach provides network administrators with insightful explanatory information and adaptable models for new network environments. Finally, we present a reinforcement learning-based defense approach against L7 DDoS attacks, which combines network traffic data with endpoint information to operate. The proposed approach actively monitors and analyzes the victim server and applies different strategies under different conditions to protect the server while minimizing collateral damage to legitimate requests. Our evaluation results demonstrate that the proposed approaches achieve high accuracy and efficiency in detecting and mitigating various malicious activities, while maintaining privacy-preserving features, providing explainable and adaptable results, or providing comprehensive application-layer situational awareness. This dissertation significantly advances the fields of FGTA and malicious activity detection. This dissertation includes published and unpublished co-authored materials

Large-scale Wireless Local-area Network Measurement and Privacy Analysis

The edge of the Internet is increasingly becoming wireless. Understanding the wireless edge is therefore important for understanding the performance and security aspects of the Internet experience. This need is especially necessary for enterprise-wide wireless local-area networks (WLANs) as organizations increasingly depend on WLANs for mission- critical tasks. To study a live production WLAN, especially a large-scale network, is a difficult undertaking. Two fundamental difficulties involved are (1) building a scalable network measurement infrastructure to collect traces from a large-scale production WLAN, and (2) preserving user privacy while sharing these collected traces to the network research community. In this dissertation, we present our experience in designing and implementing one of the largest distributed WLAN measurement systems in the United States, the Dartmouth Internet Security Testbed (DIST), with a particular focus on our solutions to the challenges of efficiency, scalability, and security. We also present an extensive evaluation of the DIST system. To understand the severity of some potential trace-sharing risks for an enterprise-wide large-scale wireless network, we conduct privacy analysis on one kind of wireless network traces, a user-association log, collected from a large-scale WLAN. We introduce a machine-learning based approach that can extract and quantify sensitive information from a user-association log, even though it is sanitized. Finally, we present a case study that evaluates the tradeoff between utility and privacy on WLAN trace sanitization

AI-based algorithm for intrusion detection on a real Dataset

[Abstract]: In this Project, Novel Machine Learning proposals are given to produce a Network Intrusion Detection System (NIDS). For this, a state of the art Dataset for Cyclo Stationary NIDS has been used, together with a previously proposed standard methodology to compare the results of different models over the same Dataset. An extensive research has been done for this Project about the different Datasets available for NIDS, as has been done to expose the evolution and functioning of IDSs. Finally, experiments have been made with Outlier Detectors, Ensemble Methods, Deep Learning and Conventional Classifiers to compare with previously published results over the same Dataset and with the same methodology. The findings reveal that the Ensemble Methods have been capable to improve the results from prior research being the best approach the Extreme Gradient Boosting method.[Resumen]: En este Proyecto, se presentan novedosas propuestas de Aprendizaje Automático para producir un Sistema de Detección de Intrusos en Red (NIDS). Para ello, se ha utilizado un Dataset de última generación para NIDS Cicloestacionarios, junto con una metodología estándar previamente propuesta para comparar los resultados de diferentes modelos sobre el mismo Dataset. Para este Proyecto se ha realizado una extensa investigación sobre los diferentes conjuntos de datos disponibles para NIDS, así como se ha expuesto la evolución y funcionamiento de los IDSs. Por último, se han realizado experimentos con Detectores de Anomalias, Métodos de Conjunto, Aprendizaje Profundo y Clasificadores Convencionales para comparar con resultados previamente publicados sobre el mismo Dataset y con la misma metodología. Los resultados revelan que los Métodos de Conjunto han sido capaces de mejorar los resultados de investigaciones previas siendo el mejor enfoque el método de Extreme Gradient Boosting.Traballo fin de grao (UDC.FIC). Enxeñaría Informática. Curso 2022/202