Data analytics in SDN and NFV: Techniques and Challenges

Software defined networking and network function virtualization are drawing huge attention from researchers both in industry and academia. NFV reduces the capital and opera- tional expenditure of the organization by decoupling the network functions from physical hardware on which they run, which poses new challenges in the perspective of network management such as data management, resource management and performance analysis. Consequentially, novel techniques and strategies are required to address these challenges in efficient way. This paper discusses the most widely used data analytics techniques like machine learning and time series data analysis. Further it describes the review of data mining tools and frameworks. Machine learning helps to overcome the challenges of network management by providing intelligence in network. Hence, in this paper we describe an overview of high level architecture of machine learning analysis framework, the challenges of applying machine learning algorithms in virtual environment and also some of the interesting problems of network management which can be solved by using machine learning

A pattern-based framework for the design of secure and dependable SDN/NFV-enabled networks

As the world becomes an interconnected network where objects and humans interact, cyber and physical networks appear to play an important role in smart ecosystems due to their increasing use on critical infrastructure and smart cities. Software Defined Networking (SDN) and Network Function Virtualisation (NFV) are a promising combination for programmable connectivity, rapid service provisioning and service chaining as they offer the necessary end-to-end optimisations. However, with the actual exponential growth of connected devices, future networks, such as SDN and NFV, require open architectures, facilitated by standards and a strong ecosystem.In this thesis, a model-based approach is proposed to support the design and verification of secure and dependable SDN/NFV-enabled networks. The model is based on the development of a pattern-based approach to design executable patterns as solutions for reusable designs and interactions of objects, encoded in a rule based reasoning system, able to guarantee security and dependability (S&D) properties in SDN/NFV enabled networks. To execute S&D patterns, a pattern based framework is implemented for the insertion of patterns at design and at runtime level. The developed pattern framework highlights also the benefit of leveraging the flexibility of SDN/NFV-enabled networks to deploy enhanced reactive security mechanisms for the protection of the industrial network via the use of service function chaining (SFC). To prove the importance of this approach and the functionality of the pattern framework, different pattern instances are implemented to guarantee S&D in network infrastructures. The developed design patterns are able to design network topologies, guarantee network properties and offer security service provisioning and chaining. Finally, in order to evaluate the developed patterns in the pattern framework, three different use cases are described, where a number of usage scenarios are deployed and evaluated experimentally

Automated Security Analysis of Virtualized Infrastructures

Virtualization enables the increasing efficiency and elasticity of modern IT infrastructures, including Infrastructure as a Service. However, the operational complexity of virtualized infrastructures is high, due to their dynamics, multi-tenancy, and size. Misconfigurations and insider attacks carry significant operational and security risks, such as breaches in tenant isolation, which put both the infrastructure provider and tenants at risk. In this thesis we study the question if it is possible to model and analyze complex, scalable, and dynamic virtualized infrastructures with regard to user-defined security and operational policies in an automated way. We establish a new practical and automated security analysis framework for virtualized infrastructures. First, we propose a novel tool that automatically extracts the configuration of heterogeneous environments and builds up a unified graph model of the configuration and topology. The tool is further extended with a monitoring component and a set of algorithms that translates system changes to graph model changes. The benefits of maintaining such a dynamic model are time reduction for model population and closing the gap for transient security violations. Our analysis is the first that lifts static information flow analysis to the entire virtualized infrastructure, in order to detect isolation failures between tenants on all resources. The analysis is configurable using customized rules to reflect the different trust assumptions of the users. We apply and evaluate our analysis system on the production infrastructure of a global financial institution. For the information flow analysis of dynamic infrastructures we propose the concept of dynamic rule-based information flow graphs and develop a set of algorithms that maintain such information flow graphs for dynamic system models. We generalize the analysis of isolation properties and establish a new generic analysis platform for virtualized infrastructures that allows to express a diverse set of security and operational policies in a formal language. The policy requirements are studied in a case-study with a cloud service provider. We are the first to employ a variety of theorem provers and model checkers to verify the state of a virtualized infrastructure against its policies. Additionally, we analyze dynamic behavior such as VM migrations. For the analysis of dynamic infrastructures we pursue both a reactive as well as a proactive approach. A reactive analysis system is developed that reduces the time between system change and analysis result. The system monitors the infrastructure for changes and employs dynamic information flow graphs to verify, for instance, tenant isolation. For the proactive analysis we propose a new model, the Operations Transition Model, which captures the changes of operations in the virtualized infrastructure as graph transformations. We build a novel analysis system using this model that performs automated run-time analysis of operations and also offers change planning. The operations transition model forms the basis for further research in model checking of virtualized infrastructures

Are Public Intrusion Datasets Fit for Purpose: Characterising the State of the Art in Intrusion Event Datasets

The file attached to this record is the author's final peer reviewed version. The Publisher's final version can be found by following the DOI link.In recent years cybersecurity attacks have caused major disruption and information loss for online organisations, with high profile incidents in the news. One of the key challenges in advancing the state of the art in intrusion detection is the lack of representative datasets. These datasets typically contain millions of time-ordered events (e.g. network packet traces, flow summaries, log entries); subsequently analysed to identify abnormal behavior and specific attacks [1]. Generating realistic datasets has historically required expensive networked assets, specialised traffic generators, and considerable design preparation. Even with advances in virtualisation it remains challenging to create and maintain a representative environment. Major improvements are needed in the design, quality and availability of datasets, to assist researchers in developing advanced detection techniques. With the emergence of new technology paradigms, such as intelligent transport and autonomous vehicles, it is also likely that new classes of threat will emerge [2]. Given the rate of change in threat behavior [3] datasets become quickly obsolete, and some of the most widely cited datasets date back over two decades. Older datasets have limited value: often heavily filtered and anonymised, with unrealistic event distributions, and opaque design methodology. The relative scarcity of (Intrusion Detection System) IDS datasets is compounded by the lack of a central registry, and inconsistent information on provenance. Researchers may also find it hard to locate datasets or understand their relative merits. In addition, many datasets rely on simulation, originating from academic or government institutions. The publication process itself often creates conflicts, with the need to de-identify sensitive information in order to meet regulations such as General Data Protection Act (GDPR) [4]. Another final issue for researchers is the lack of standardised metrics with which to compare dataset quality. In this paper we attempt to classify the most widely used public intrusion datasets, providing references to archives and associated literature. We illustrate their relative utility and scope, highlighting the threat composition, formats, special features, and associated limitations. We identify best practice in dataset design, and describe potential pitfalls of designing anomaly detection techniques based on data that may be either inappropriate, or compromised due to unrealistic threat coverage. Such contributions as made in this paper is expected to facilitate continuous research and development for effectively combating the constantly evolving cyber threat landscape

Trust and integrity in distributed systems

In the last decades, we have witnessed an exploding growth of the Internet. The massive adoption of distributed systems on the Internet allows users to offload their computing intensive work to remote servers, e.g. cloud. In this context, distributed systems are pervasively used in a number of difference scenarios, such as web-based services that receive and process data, cloud nodes where company data and processes are executed, and softwarised networks that process packets. In these systems, all the computing entities need to trust each other and co-operate in order to work properly. While the communication channels can be well protected by protocols like TLS or IPsec, the problem lies in the expected behaviour of the remote computing platforms, because they are not under the direct control of end users and do not offer any guarantee that they will behave as agreed. For example, the remote party may use non-legitimate services for its own convenience (e.g. illegally storing received data and routed packets), or the remote system may misbehave due to an attack (e.g. changing deployed services). This is especially important because most of these computing entities need to expose interfaces towards the Internet, which makes them easier to be attacked. Hence, software-based security solutions alone are insufficient to deal with the current scenario of distributed systems. They must be coupled with stronger means such as hardware-assisted protection. In order to allow the nodes in distributed system to trust each other, their integrity must be presented and assessed to predict their behaviour. The remote attestation technique of trusted computing was proposed to specifically deal with the integrity issue of remote entities, e.g. whether the platform is compromised with bootkit attacks or cracked kernel and services. This technique relies on a hardware chip called Trusted Platform Module (TPM), which is available in most business class laptops, desktops and servers. The TPM plays as the hardware root of trust, which provides a special set of capabilities that allows a physical platform to present its integrity state. With a TPM equipped in the motherboard, the remote attestation is the procedure that a physical node provides hardware-based proof of the software components loaded in this platform, which can be evaluated by other entities to conclude its integrity state. Thanks to the hardware TPM, the remote attestation procedure is resistant to software attacks. However, even though the availability of this chip is high, its actual usage is low. The major reason is that trusted computing has very little flexibility, since its goal is to provide strong integrity guarantees. For instance, remote attestation result is positive if and only if the software components loaded in the platform are expected and loaded in a specific order, which limits its applicability in real-world scenarios. For such reasons, this technique is especially hard to be applied on software services running in application layer, that are loaded in random order and constantly updated. Because of this, current remote attestation techniques provide incomplete solution. They only focus on the boot phase of physical platforms but not on the services, not to mention the services running in virtual instances. This work first proposes a new remote attestation framework with the capability of presenting and evaluating the integrity state not only of the boot phase of physical platforms but also of software services at load time, e.g. whether the software is legitimate or not. The framework allows users to know and understand the integrity state of the whole life cycle of the services they are interacting with, thus the users can make informed decision whether to send their data or trust the received results. Second, based on the remote attestation framework this thesis proposes a method to bind the identity of secure channel endpoint to a specific physical platform and its integrity state. Secure channels are extensively adopted in distributed systems to protect data transmitted from one platform to another. However, they do not convey any information about the integrity state of the platform or the service that generates and receives this data, which leaves ample space for various attacks. With the binding of the secure channel endpoint and the hardware TPM, users are protected from relay attacks (with hardware-based identity) and malicious or cracked platform and software (with remote attestation). Third, with the help of the remote attestation framework, this thesis introduces a new method to include the integrity state of software services running in virtual containers in the evidence generated by the hardware TPM. This solution is especially important for softwarised network environments. Softwarised network was proposed to provide dynamic and flexible network deployment which is an ever complex task nowadays. Its main idea is to switch hardware appliances to softwarised network functions running inside virtual instances, that are full-fledged computational systems and accessible from the Internet, thus their integrity is at stake. Unfortunately, currently remote attestation work is not able to provide hardware-based integrity evidence for software services running inside virtual instances, because the direct link between the internal of virtual instances and hardware root of trust is missing. With the solution proposed in this thesis, the integrity state of the softwarised network functions running in virtual containers can be presented and evaluated with hardware-based evidence, implying the integrity of the whole softwarised network. The proposed remote attestation framework, trusted channel and trusted softwarised network are implemented in separate working prototypes. Their performance was evaluated and proved to be excellent, allowing them to be applied in real-world scenarios. Moreover, the implementation also exposes various APIs to simplify future integration with different management platforms, such as OpenStack and OpenMANO

Intelligent Management of Virtualised Computer Based Workloads and Systems

Managing the complexity within virtualised IT infrastructure platforms is a common problem for many organisations today. Computer systems are often highly consolidated into a relatively small physical footprint compared with previous decades prior to late 2000s, so much thought, planning and control is necessary to effectively operate such systems within the enterprise computing space. With the development of private, hybrid and public cloud utility computing this has become even more relevant; this work examines how such cloud systems are using virtualisation technology and embedded software to leverage advantages, and it uses a fresh approach of developing and creating an Intelligent decision engine (expert system). Its aim is to help reduce the complexity of managing virtualised computer-based platforms, through tight integration, high-levels of automation to minimise human inputs, errors, and enforce standards and consistency, in order to achieve better management and control. The thesis investigates whether an expert system known as the Intelligent Decision Engine (IDE) could aid the management of virtualised computer-based platforms. Through conducting a series of mixed quantitative and qualitative experiments in the areas of research, the initial findings and evaluation are presented in detail, using repeatable and observable processes and provide detailed analysis on the recorded outputs. The results of the investigation establish the advantages of using the IDE (expert system) to achieve the goal of reducing the complexity of managing virtualised computer-based platforms. In each detailed area examined, it is demonstrated how using a global management approach in combination with VM provisioning, migration, failover, and system resource controls can create a powerful autonomous system