SYSTEM FOR EXPERT-ASSISTED CAUSAL INFERENCE FOR RANKING EVENTS OF INTEREST IN NETWORKS

Networks have increased in size and complexity such that the number of events occurring each day has grown drastically. Techniques of this proposal provide for the ability to infer candidates for causal relationships—in some cases, with confidence. In particular, a novel machine learning (ML) based system is described that provides for the ability to narrow-down candidate temporal patterns that may potentially explain an event of interest (e.g., a network outage). The system is trainable with a human in the loop and is highly effective even with minimal amount of prior training

A Survey on Big Data for Network Traffic Monitoring and Analysis

Network Traffic Monitoring and Analysis (NTMA) represents a key component for network management, especially to guarantee the correct operation of large-scale networks such as the Internet. As the complexity of Internet services and the volume of traffic continue to increase, it becomes difficult to design scalable NTMA applications. Applications such as traffic classification and policing require real-time and scalable approaches. Anomaly detection and security mechanisms require to quickly identify and react to unpredictable events while processing millions of heterogeneous events. At last, the system has to collect, store, and process massive sets of historical data for post-mortem analysis. Those are precisely the challenges faced by general big data approaches: Volume, Velocity, Variety, and Veracity. This survey brings together NTMA and big data. We catalog previous work on NTMA that adopt big data approaches to understand to what extent the potential of big data is being explored in NTMA. This survey mainly focuses on approaches and technologies to manage the big NTMA data, additionally briefly discussing big data analytics (e.g., machine learning) for the sake of NTMA. Finally, we provide guidelines for future work, discussing lessons learned, and research directions

Hybrid SDN Evolution: A Comprehensive Survey of the State-of-the-Art

Software-Defined Networking (SDN) is an evolutionary networking paradigm which has been adopted by large network and cloud providers, among which are Tech Giants. However, embracing a new and futuristic paradigm as an alternative to well-established and mature legacy networking paradigm requires a lot of time along with considerable financial resources and technical expertise. Consequently, many enterprises can not afford it. A compromise solution then is a hybrid networking environment (a.k.a. Hybrid SDN (hSDN)) in which SDN functionalities are leveraged while existing traditional network infrastructures are acknowledged. Recently, hSDN has been seen as a viable networking solution for a diverse range of businesses and organizations. Accordingly, the body of literature on hSDN research has improved remarkably. On this account, we present this paper as a comprehensive state-of-the-art survey which expands upon hSDN from many different perspectives

Online learning on the programmable dataplane

This thesis makes the case for managing computer networks with datadriven methods automated statistical inference and control based on measurement data and runtime observations—and argues for their tight integration with programmable dataplane hardware to make management decisions faster and from more precise data. Optimisation, defence, and measurement of networked infrastructure are each challenging tasks in their own right, which are currently dominated by the use of hand-crafted heuristic methods. These become harder to reason about and deploy as networks scale in rates and number of forwarding elements, but their design requires expert knowledge and care around unexpected protocol interactions. This makes tailored, per-deployment or -workload solutions infeasible to develop. Recent advances in machine learning offer capable function approximation and closed-loop control which suit many of these tasks. New, programmable dataplane hardware enables more agility in the network— runtime reprogrammability, precise traffic measurement, and low latency on-path processing. The synthesis of these two developments allows complex decisions to be made on previously unusable state, and made quicker by offloading inference to the network. To justify this argument, I advance the state of the art in data-driven defence of networks, novel dataplane-friendly online reinforcement learning algorithms, and in-network data reduction to allow classification of switchscale data. Each requires co-design aware of the network, and of the failure modes of systems and carried traffic. To make online learning possible in the dataplane, I use fixed-point arithmetic and modify classical (non-neural) approaches to take advantage of the SmartNIC compute model and make use of rich device local state. I show that data-driven solutions still require great care to correctly design, but with the right domain expertise they can improve on pathological cases in DDoS defence, such as protecting legitimate UDP traffic. In-network aggregation to histograms is shown to enable accurate classification from fine temporal effects, and allows hosts to scale such classification to far larger flow counts and traffic volume. Moving reinforcement learning to the dataplane is shown to offer substantial benefits to stateaction latency and online learning throughput versus host machines; allowing policies to react faster to fine-grained network events. The dataplane environment is key in making reactive online learning feasible—to port further algorithms and learnt functions, I collate and analyse the strengths of current and future hardware designs, as well as individual algorithms

A Cognitive Routing framework for Self-Organised Knowledge Defined Networks

This study investigates the applicability of machine learning methods to the routing protocols for achieving rapid convergence in self-organized knowledge-defined networks. The research explores the constituents of the Self-Organized Networking (SON) paradigm for 5G and beyond, aiming to design a routing protocol that complies with the SON requirements. Further, it also exploits a contemporary discipline called Knowledge-Defined Networking (KDN) to extend the routing capability by calculating the “Most Reliable” path than the shortest one. The research identifies the potential key areas and possible techniques to meet the objectives by surveying the state-of-the-art of the relevant fields, such as QoS aware routing, Hybrid SDN architectures, intelligent routing models, and service migration techniques. The design phase focuses primarily on the mathematical modelling of the routing problem and approaches the solution by optimizing at the structural level. The work contributes Stochastic Temporal Edge Normalization (STEN) technique which fuses link and node utilization for cost calculation; MRoute, a hybrid routing algorithm for SDN that leverages STEN to provide constant-time convergence; Most Reliable Route First (MRRF) that uses a Recurrent Neural Network (RNN) to approximate route-reliability as the metric of MRRF. Additionally, the research outcomes include a cross-platform SDN Integration framework (SDN-SIM) and a secure migration technique for containerized services in a Multi-access Edge Computing environment using Distributed Ledger Technology. The research work now eyes the development of 6G standards and its compliance with Industry-5.0 for enhancing the abilities of the present outcomes in the light of Deep Reinforcement Learning and Quantum Computing

Secure Diagnostics And Forensics With Network Provenance

In large-scale networks, many things can go wrong: routers can be misconfigured, programs can be buggy, and computers can be compromised by an attacker. As a result, there is a constant need to perform network diagnostics and forensics. In this dissertation, we leverage the concept of provenance to build better support for diagnostic and forensic tasks. At a high level, provenance tracks causality between network states and events, and produces a detailed explanation of any event of interest, which makes it a good starting point for investigating network problems. However, in order to use provenance for network diagnostics and forensics, several challenges need to be addressed. First, existing provenance systems cannot provide security properties on high-speed network traffic, because the cryptographic operations would cause enormous overhead when the data rates are high. To address this challenge, we design secure packet provenance, a system that comes with a novel lightweight security protocol, to maintain secure provenance with low overhead. Second, in large-scale distributed systems, the provenance of a network event can be quite complex, so it is still challenging to identify the problem root cause from the complex provenance. To address this challenge, we design differential provenance, which can identify a symptom event’s root cause by reasoning about the differences between its provenance and the provenance of a similar “reference” event. Third, provenance can only explain why a current network state came into existence, but by itself, it does not reason about changes to the network state to fix a problem. To provide operators with more diagnostic support, we design causal networks – a generalization of network provenance – to reason about network repairs that can avoid undesirable side effects in the network. Causal networks can encode multiple diagnostic goals in the same data structure, and, therefore, generate repairs that satisfy multiple constraints simultaneously. We have applied these techniques to Software-Defined Networks, Hadoop MapReduce, as well as the Internet’s data plane. Our evaluation with real-world traffic traces and network topologies shows that our systems can run with reasonable overhead, and that they can accurately identify root causes of practical problems and generate repairs without causing collateral damage

Enhancing programmability for adaptive resource management in next generation data centre networks

Recently, Data Centre (DC) infrastructures have been growing rapidly to support a wide range of emerging services, and provide the underlying connectivity and compute resources that facilitate the "*-as-a-Service" model. This has led to the deployment of a multitude of services multiplexed over few, very large-scale centralised infrastructures. In order to cope with the ebb and flow of users, services and traffic, infrastructures have been provisioned for peak-demand resulting in the average utilisation of resources to be low. This overprovisionning has been further motivated by the complexity in predicting traffic demands over diverse timescales and the stringent economic impact of outages. At the same time, the emergence of Software Defined Networking (SDN), is offering new means to monitor and manage the network infrastructure to address this underutilisation. This dissertation aims to show how measurement-based resource management can improve performance and resource utilisation by adaptively tuning the infrastructure to the changing operating conditions. To achieve this dynamicity, the infrastructure must be able to centrally monitor, notify and react based on the current operating state, from per-packet dynamics to longstanding traffic trends and topological changes. However, the management and orchestration abilities of current SDN realisations is too limiting and must evolve for next generation networks. The current focus has been on logically centralising the routing and forwarding decisions. However, in order to achieve the necessary fine-grained insight, the data plane of the individual device must be programmable to collect and disseminate the metrics of interest. The results of this work demonstrates that a logically centralised controller can dynamically collect and measure network operating metrics to subsequently compute and disseminate fine-tuned environment-specific settings. They show how this approach can prevent TCP throughput incast collapse and improve TCP performance by an order of magnitude for partition-aggregate traffic patterns. Futhermore, the paradigm is generalised to show the benefits for other services widely used in DCs such as, e.g, routing, telemetry, and security

Automating Cyber Analytics

Model based security metrics are a growing area of cyber security research concerned with measuring the risk exposure of an information system. These metrics are typically studied in isolation, with the formulation of the test itself being the primary finding in publications. As a result, there is a flood of metric specifications available in the literature but a corresponding dearth of analyses verifying results for a given metric calculation under different conditions or comparing the efficacy of one measurement technique over another. The motivation of this thesis is to create a systematic methodology for model based security metric development, analysis, integration, and validation. In doing so we hope to fill a critical gap in the way we view and improve a system’s security. In order to understand the security posture of a system before it is rolled out and as it evolves, we present in this dissertation an end to end solution for the automated measurement of security metrics needed to identify risk early and accurately. To our knowledge this is a novel capability in design time security analysis which provides the foundation for ongoing research into predictive cyber security analytics. Modern development environments contain a wealth of information in infrastructure-as-code repositories, continuous build systems, and container descriptions that could inform security models, but risk evaluation based on these sources is ad-hoc at best, and often simply left until deployment. Our goal in this work is to lay the groundwork for security measurement to be a practical part of the system design, development, and integration lifecycle. In this thesis we provide a framework for the systematic validation of the existing security metrics body of knowledge. In doing so we endeavour not only to survey the current state of the art, but to create a common platform for future research in the area to be conducted. We then demonstrate the utility of our framework through the evaluation of leading security metrics against a reference set of system models we have created. We investigate how to calibrate security metrics for different use cases and establish a new methodology for security metric benchmarking. We further explore the research avenues unlocked by automation through our concept of an API driven S-MaaS (Security Metrics-as-a-Service) offering. We review our design considerations in packaging security metrics for programmatic access, and discuss how various client access-patterns are anticipated in our implementation strategy. Using existing metric processing pipelines as reference, we show how the simple, modular interfaces in S-MaaS support dynamic composition and orchestration. Next we review aspects of our framework which can benefit from optimization and further automation through machine learning. First we create a dataset of network models labeled with the corresponding security metrics. By training classifiers to predict security values based only on network inputs, we can avoid the computationally expensive attack graph generation steps. We use our findings from this simple experiment to motivate our current lines of research into supervised and unsupervised techniques such as network embeddings, interaction rule synthesis, and reinforcement learning environments. Finally, we examine the results of our case studies. We summarize our security analysis of a large scale network migration, and list the friction points along the way which are remediated by this work. We relate how our research for a large-scale performance benchmarking project has influenced our vision for the future of security metrics collection and analysis through dev-ops automation. We then describe how we applied our framework to measure the incremental security impact of running a distributed stream processing system inside a hardware trusted execution environment

Context-based security function orchestration for the network edge

Over the last few years the number of interconnected devices has increased dramatically, generating zettabytes of traffic each year. In order to cater to the requirements of end-users, operators have deployed network services to enhance their infrastructure. Nowadays, telecommunications service providers are making use of virtualised, flexible, and cost-effective network-wide services, under what is known as Network Function Virtualisation (NFV). Future network and application requirements necessitate services to be delivered at the edge of the network, in close proximity to end-users, which has the potential to reduce end-to-end latency and minimise the utilisation of the core infrastructure while providing flexible allocation of resources. One class of functionality that NFV facilitates is the rapid deployment of network security services. However, the urgency for assuring connectivity to an ever increasing number of devices as well as their resource-constrained nature, has led to neglecting security principles and best practices. These low-cost devices are often exploited for malicious purposes in targeting the network infrastructure, with recent volumetric Distributed Denial of Service (DDoS) attacks often surpassing 1 terabyte per second of network traffic. The work presented in this thesis aims to identify the unique requirements of security modules implemented as Virtual Network Functions (VNFs), and the associated challenges in providing management and orchestration of complex chains consisting of multiple VNFs The work presented here focuses on deployment, placement, and lifecycle management of microservice-based security VNFs in resource-constrained environments using contextual information on device behaviour. Furthermore, the thesis presents a formulation of the latency-optimal placement of service chains at the network edge, provides an optimal solution using Integer Linear Programming, and an associated near-optimal heuristic solution that is able to solve larger-size problems in reduced time, which can be used in conjunction with context-based security paradigms. The results of this work demonstrate that lightweight security VNFs can be tailored for, and hosted on, a variety of devices, including commodity resource-constrained systems found in edge networks. Furthermore, using a context-based implementation of the management and orchestration of lightweight services enables the deployment of real-world complex security service chains tailored towards the user’s performance demands from the network. Finally, the results of this work show that on-path placement of service chains reduces the end-to-end latency and minimise the number of service-level agreement violations, therefore enabling secure use of latency-critical networks