Fulcrum: Flexible Network Coding for Heterogeneous Devices

Producción CientíficaWe introduce Fulcrum, a network coding framework that achieves three seemingly conflicting objectives: 1) to reduce the coding coefficient overhead down to nearly n bits per packet in a generation of n packets; 2) to conduct the network coding using only Galois field GF(2) operations at intermediate nodes if necessary, dramatically reducing computing complexity in the network; and 3) to deliver an end-to-end performance that is close to that of a high-field network coding system for high-end receivers, while simultaneously catering to low-end receivers that decode in GF(2). As a consequence of 1) and 3), Fulcrum has a unique trait missing so far in the network coding literature: providing the network with the flexibility to distribute computational complexity over different devices depending on their current load, network conditions, or energy constraints. At the core of our framework lies the idea of precoding at the sources using an expansion field GF(2 h ), h > 1, to increase the number of dimensions seen by the network. Fulcrum can use any high-field linear code for precoding, e.g., Reed-Solomon or Random Linear Network Coding (RLNC). Our analysis shows that the number of additional dimensions created during precoding controls the trade-off between delay, overhead, and computing complexity. Our implementation and measurements show that Fulcrum achieves similar decoding probabilities as high field RLNC but with encoders and decoders that are an order of magnitude faster.Green Mobile Cloud project (grant DFF-0602-01372B)Colorcast project (grant DFF-0602-02661B)TuneSCode project (grant DFF - 1335-00125)Danish Council for Independent Research (grant DFF-4002-00367)Ministerio de Economía, Industria y Competitividad - Fondo Europeo de Desarrollo Regional (grants MTM2012-36917-C03-03 / MTM2015-65764-C3-2-P / MTM2015-69138-REDT)Agencia Estatal de Investigación - Fondo Social Europeo (grant RYC-2016-20208)Aarhus Universitets Forskningsfond Starting (grant AUFF-2017-FLS-7-1

Overview of Caching Mechanisms to Improve Hadoop Performance

Nowadays distributed computing environments, large amounts of data are generated from different resources with a high velocity, rendering the data difficult to capture, manage, and process within existing relational databases. Hadoop is a tool to store and process large datasets in a parallel manner across a cluster of machines in a distributed environment. Hadoop brings many benefits like flexibility, scalability, and high fault tolerance; however, it faces some challenges in terms of data access time, I/O operation, and duplicate computations resulting in extra overhead, resource wastage, and poor performance. Many researchers have utilized caching mechanisms to tackle these challenges. For example, they have presented approaches to improve data access time, enhance data locality rate, remove repetitive calculations, reduce the number of I/O operations, decrease the job execution time, and increase resource efficiency. In the current study, we provide a comprehensive overview of caching strategies to improve Hadoop performance. Additionally, a novel classification is introduced based on cache utilization. Using this classification, we analyze the impact on Hadoop performance and discuss the advantages and disadvantages of each group. Finally, a novel hybrid approach called Hybrid Intelligent Cache (HIC) that combines the benefits of two methods from different groups, H-SVM-LRU and CLQLMRS, is presented. Experimental results show that our hybrid method achieves an average improvement of 31.2% in job execution time

Mobile cloud computing and network function virtualization for 5g systems

The recent growth of the number of smart mobile devices and the emergence of complex multimedia mobile applications have brought new challenges to the design of wireless mobile networks. The envisioned Fifth-Generation (5G) systems are equipped with different technical solutions that can accommodate the increasing demands for high date rate, latency-limited, energy-efficient and reliable mobile communication networks. Mobile Cloud Computing (MCC) is a key technology in 5G systems that enables the offloading of computationally heavy applications, such as for augmented or virtual reality, object recognition, or gaming from mobile devices to cloudlet or cloud servers, which are connected to wireless access points, either directly or through finite-capacity backhaul links. Given the battery-limited nature of mobile devices, mobile cloud computing is deemed to be an important enabler for the provision of such advanced applications. However, computational tasks offloading, and due to the variability of the communication network through which the cloud or cloudlet is accessed, may incur unpredictable energy expenditure or intolerable delay for the communications between mobile devices and the cloud or cloudlet servers. Therefore, the design of a mobile cloud computing system is investigated by jointly optimizing the allocation of radio, computational resources and backhaul resources in both uplink and downlink directions. Moreover, the users selected for cloud offloading need to have an energy consumption that is smaller than the amount required for local computing, which is achieved by means of user scheduling. Motivated by the application-centric drift of 5G systems and the advances in smart devices manufacturing technologies, new brand of mobile applications are developed that are immersive, ubiquitous and highly-collaborative in nature. For example, Augmented Reality (AR) mobile applications have inherent collaborative properties in terms of data collection in the uplink, computing at the cloud, and data delivery in the downlink. Therefore, the optimization of the shared computing and communication resources in MCC not only benefit from the joint allocation of both resources, but also can be more efficiently enhanced by sharing the offloaded data and computations among multiple users. As a result, a resource allocation approach whereby transmitted, received and processed data are shared partially among the users leads to more efficient utilization of the communication and computational resources. As a suggested architecture in 5G systems, MCC decouples the computing functionality from the platform location through the use of software virtualization to allow flexible provisioning of the provided services. Another virtualization-based technology in 5G systems is Network Function Virtualization (NFV) which prescribes the instantiation of network functions on general-purpose network devices, such as servers and switches. While yielding a more flexible and cost-effective network architecture, NFV is potentially limited by the fact that commercial off-the-shelf hardware is less reliable than the dedicated network elements used in conventional cellular deployments. The typical solution for this problem is to duplicate network functions across geographically distributed hardware in order to ensure diversity. For that reason, the development of fault-tolerant virtualization strategies for MCC and NFV is necessary to ensure reliability of the provided services

Reliable Packet Streams with Multipath Network Coding

With increasing computational capabilities and advances in robotics, technology is at the verge of the next industrial revolution. An growing number of tasks can be performed by artificial intelligence and agile robots. This impacts almost every part of the economy, including agriculture, transportation, industrial manufacturing and even social interactions. In all applications of automated machines, communication is a critical component to enable cooperation between machines and exchange of sensor and control signals. The mobility and scale at which these automated machines are deployed also challenges todays communication systems. These complex cyber-physical systems consisting of up to hundreds of mobile machines require highly reliable connectivity to operate safely and efficiently. Current automation systems use wired communication to guarantee low latency connectivity. But wired connections cannot be used to connect mobile robots and are also problematic to deploy at scale. Therefore, wireless connectivity is a necessity. On the other hand, it is subject to many external influences and cannot reach the same level of reliability as the wired communication systems. This thesis aims to address this problem by proposing methods to combine multiple unreliable wireless connections to a stable channel. The foundation for this work is Caterpillar Random Linear Network Coding (CRLNC), a new variant of network code designed to achieve low latency. CRLNC performs similar to block codes in recovery of lost packets, but with a significantly decreased latency. CRLNC with Feedback (CRLNC-FB) integrates a Selective-Repeat ARQ (SR-ARQ) to optimize the tradeoff between delay and throughput of reliable communication. The proposed protocol allows to slightly increase the overhead to reduce the packet delay at the receiver. With CRLNC, delay can be reduced by more than 50 % with only a 10 % reduction in throughput. Finally, CRLNC is combined with a statistical multipath scheduler to optimize the reliability and service availability in wireless network with multiple unreliable paths. This multipath CRLNC scheme improves the reliability of a fixed-rate packet stream by 10 % in a system model based on real-world measurements of LTE and WiFi. All the proposed protocols have been implemented in the software library NCKernel. With NCKernel, these protocols could be evaluated in simulated and emulated networks, and were also deployed in several real-world testbeds and demonstrators.:Abstract 2 Acknowledgements 6 1 Introduction 7 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 1.2 Use Cases and Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 1.3 Opportunities of Multipath . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.4 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 2 State of the Art of Multipath Communication 19 2.1 Physical Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 2.2 Data Link Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2.3 Network Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.4 Transport Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.5 Application Layer and Session Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 2.6 Research Gap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3 NCKernel: Network Coding Protocol Framework 27 3.1 Theory that matters! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.2 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.3 Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.3.1 Socket Buffers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 3.3.2 En-/Re-/Decoder API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 3.3.3 Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3.4 Timers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 3.3.5 Tracing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.4 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.5 Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 4 Low-Latency Network Coding 35 4.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 4.2 Random Linear Network Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 4.3 Low Latency Network Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 4.4 CRLNC: Caterpillar Random Linear Network Coding . . . . . . . . . . . . . . . . . . 38 4.4.1 Encoding and Packet Format . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 4.4.2 Decoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 4.4.3 Computational Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 4.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.5.1 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 4.5.2 Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4.5.3 Packet Loss Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 4.5.4 Delay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 4.5.5 Window Size Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 4.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 5 Delay-Throughput Tradeoff 55 5.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.2 Network Coding with ARQ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 5.3 CRLNC-FB: CRLNC with Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 5.3.1 Encoding and Packet Format . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.3.2 Decoding and Feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 5.3.3 Retransmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 5.4 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.4.1 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 5.4.2 Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.4.3 Systematic Retransmissions . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 5.4.4 Coded Packet Memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 5.4.5 Comparison with other Protocols . . . . . . . . . . . . . . . . . . . . . . . . 67 5.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 6 Multipath for Reliable Low-Latency Packet Streams 73 6.1 Problem Statement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 6.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 6.3 System Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 6.3.1 Traffic Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 6.3.2 Network Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 6.3.3 Channel Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 6.3.4 Reliability Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 6.4 Multipath CRLNC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 6.4.1 Window Size for Heterogeneous Paths . . . . . . . . . . . . . . . . . . . . . 77 6.4.2 Packet Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 6.5 Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 6.5.1 Simulator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 6.5.2 Preliminary Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 6.5.3 Simulation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 6.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 7 Conclusion 94 7.1 Results and Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 7.2 Future Research Topics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Acronyms 99 Publications 101 Bibliography 10

COCAM: a cooperative video edge caching and multicasting approach based on multi-agent deep reinforcement learning in multi-clouds environment

The evolution of the Internet of Things technology (IoT) has boosted the drastic increase in network traffic demand. Caching and multicasting in the multi-clouds scenario are effective approaches to alleviate the backhaul burden of networks and reduce service latency. However, existing works do not jointly exploit the advantages of these two approaches. In this paper, we propose COCAM, a cooperative video edge caching and multicasting approach based on multi-agent deep reinforcement learning to minimize the transmission number in the multi-clouds scenario with limited storage capacity in each edge cloud. Specifically, by integrating a cooperative transmission model with the caching model, we provide a concrete formulation of the joint problem. Then, we cast this decision-making problem as a multi-agent extension of the Markov decision process and propose a multi-agent actor-critic algorithm in which each agent learns a local caching strategy and further encompasses the observations of neighboring agents as constituents of the overall state. Finally, to validate the COCAM algorithm, we conduct extensive experiments on a real-world dataset. The results show that our proposed algorithm outperforms other baseline algorithms in terms of the number of video transmissions