Depto. de Arquitectura de Computadores y AutomáticaFac. de InformáticaFALSEMinisterio de Ciencia e Innovación (MICINN)Comunidad de Madridunpu

Huedo Cuesta, Eduardo

Martín Llorente, Ignacio

Moreno Vozmediano, Rafael

Santiago Montero, Rubén

Ministerio de Ciencia e Innovación (MICINN)Comunidad de MadridDepto. de Arquitectura de Computadores y AutomáticaFac. de InformáticaFALSEunpu

Docta Complutense

Disk Image Storage, Distribution andCaching for Edge Cloud InfrastructuresEduardo Huedo1, Rubén S. Montero1, Rafael Moreno-Vozmediano1, Ignacio M. Llorente21Universidad Complutense de Madrid2OpenNebula Systems1. IntroductionIn this paper, we propose a specialized solution for the management of disk images in highlydistributed environments, like edge cloud infrastructures. The proposed solution has beendesigned to meet the following functional requirements:● Access to external (public) marketplaces  that act as global image repositories.Examples include OpenNebula Marketplace or Docker Hub, but also private HTTPrepositories or container registries.● Minimize image transferring and maximize application I/O performance . Clusters canuse a cloud/edge deployment model over public internet links. Moreover,on-premises provisions can scale to a high number of hosts. Storage should not be abottleneck for any of these situations.● Simple deployment . Reduce the complexity and technology footprint of the solutionby using technologies that already exist in the Linux operating system toaccommodate any deployment model, both on physical and virtual resources, as wellas increasing the reliability of the storage backend.2. Related workEnterprise-class SDS (Software Defined Storage) backends, such as Ceph or Gluster, aredesigned to offer high scalability and availability. Although high-performing and feature rich,the deployment and operation of these storage platforms require extensive experience,higher financial investment, dedicated hardware, and a significant amount of humanresources. In general, their cost, complexity and resource requirements prevent theirpractical use in edge clusters.Ceph delivers object, block, and file storage in one unified system. GlusterFS aggregatesvarious storage servers over different network interfaces and connects them to form a singlelarge parallel network file system.The InterPlanetary File System (IPFS) [1] is a protocol and peer-to-peer network for storingand sharing data. It is a global decentralized file system, where users not only receive butalso host content. Content-addressing is used to uniquely identify each file in a globalnamespace connecting all computing devices, using the Kademlia [2] DHT (Distributed HashTable). Each node caches what it consumes and serves that data to its peers.IPFS Cluster allows creating a cluster of nodes to replicate data, providing redundancy, loadbalancing, and write permission management. They can be run privately or connected to therest of the IPFS network.IPFS could be used for global image distribution, at the cost of increasing the softwarefootprint of storage servers. However, specialized transfer for edge nodes needs VM specificoperations, like image snapshotting and cloning. Also, all the edge nodes would have to beIPFS nodes, increasing complexity and also software footprint.There are many works dealing with object storage for fog or edge environments. Someexisting systems are good candidates for that, for example, Apache Cassandra (NoSQLdatabase) or RADOS [3] (Reliable Autonomous Distributed Object Store, an object storagetechnology on which Ceph is based), and there are also proposals for native edge storagesystems [4]. However, these works deal with data generated from devices and applications.In fact, some authors [5,6] characterize these systems as symmetric or reversed CDNs(Content Distribution Networks), since data is generated on user devices, it is temporarilystored on fog or edge nodes and some part ends up in the core cloud. On the contrary, ourwork focuses on image distribution, which resembles traditional CDNs, where data isdistributed from the core to the edge, but specialized management is needed.Confais et al. [7] evaluated three object store solutions, namely RADOS, Apache Cassandra[8] (a NoSQL database) and IPFS using the Yahoo Cloud System Benchmark (YCSB) [9], awell-known benchmark tool particularly designed to benchmark object stores, on top of theGrid’5000 testbed. The authors conclude that IPFS provides low access times and is theonly system that is able to place the data as close to the user as possible. However,accessing the DHT increases the access times, even to read objects stored locally. To solvethis problem, they propose to add a scale-out NAS (Network Attached Storage) system (i.e.capable of aggregating storage resources from heterogeneous servers) on each edge or fogsite.Delgado Peris et al. [10] proposed a cooperative data caching system for late-bindingscheduling (i.e. based on pilot jobs) on Grid Computing. The system reduces the I/Opressure on storage servers of grid sites by using the Kademlia DHT, which allows pilot jobsto locate and retrieve cached data files from other nodes. In our system, images are alwaystransferred from a replica server, but it could be extended to make transfers from any nodein the cluster.3. Edge Cloud Architecture Based on EdgeClustersThe proposed Edge Cloud Architecture uses lightweight systems and software componentswith modest hardware requirements that are easily healed and operated, while providingmaximum application performance.This architecture is articulated around the concept of edge cluster, which is ahyper-converged  functional set of managed objects that include storage, network and hostresources. A cluster provides all the resources needed to run virtualized or containerizedapplications.Edge clusters are built upon virtualization hosts and an interconnection network. Thevirtualization hosts are responsible for providing applications with execution resourcesthrough a suitable virtualization form, i.e. VMs (QEMU/KVM), system containers (LXC) ormicroVMs (Firecracker/KVM). The actual virtualization technology depends on how thecluster hosts are provisioned and the profile of the workload. Additionally, hosts supply thecluster with storage space needed to run the applications as well as network access.The configuration of the hosts of a cluster is homogeneous in terms of the installed softwarecomponents, accessible storage and administration. Cluster nodes are interconnectedthrough one or more private networks, normally used for storage transfers as well as forapplication communication across virtualization hosts. Finally, access to a public link isrequired to provide applications with Internet connectivity.Cloud management services, including scheduling, monitoring and life-cycle management,run in the cloud front-end and orchestrate the edge clusters. The front-end also providesaccess to the administration tools, user interfaces and API. Thus, the front-end receivesstatus and monitoring updates from cluster hosts and initiates management operations. Thebasic building blocks of the architecture, including the edge clusters, are depicted in Figure1.Figure 1. Edge cloud architecture.As shown in Figure 1, a single front-end can manage clusters from multiple locations at thesame time. In fact, one of the main advantages of this architecture is the ability to deployedge clusters on public bare-metal or virtual providers to enable powerful hybrid cloud,multi-cloud and edge computing.A tool allows the deployment of a fully operational edge cluster in a remote provider and themanagement of its full life-cycle, starting with their provision and maintenance, until theunprovision. Each cloud or edge location is defined as a group of hosts allocated from theremote bare-metal or virtual provider. They are fully configured with the user-selectedhypervisor and enabled in the cloud stack for the end users.Overall, this architecture is able to manage hundreds of these edge clusters, as they operateautonomously in terms of networking and storage, and handle thousands of virtualized hostsand tens of thousands of virtualized applications.4. Image Storage, Distribution and Caching4.1. Edge storage systemThe edge-cloud architecture presented in the previous section provides a lightweight andeasy-to-use storage platform for medium-sized clusters consisting of tens of nodes that canrun on-premises and on-cloud, and on physical and on virtualized resources.The proposed storage system uses the Datastore concept, which is any storage medium tostore disk images holding operating systems or data. The Images Datastore holds the diskimages repository. There are different types of Image Datastores depending on how theimages are stored on the underlying storage technology. On the other hand, the SystemDatastore holds disk images for VMs that are currently running. Images are moved, orcloned, to/from the Images Datastore when the VMs are deployed or terminated; or whendisk images are attached or snapshotted. For example, persistent images save back to thedatastore the changes made inside the VM after it is shut down. Images are transferredbetween the Image and System datastores by the transfer manager drivers, which arespecialized pieces of software that perform low-level storage operations.The proposed image distribution and storage solution for edge clusters combines a 3-tierglobal architecture for image distribution with an Image Datastore Replica on each edgecluster providing replica caching and snapshotting.Figure 2 depicts the 3-tier edge storage system model, which consists of:● Tier 1 - Global Marketplace s. This tier consists of the remote servers and storageimplementing the global application image repositories. It includes third-party siteslike Docker Hub or Linux Containers, as well as OpenNebula Public Marketplace.● Tier 2 - Image Datastores . This tier provides the primary image storage location.This storage area is provided for one or more dedicated nodes. Its contents arecached and replicated on-demand to each edge cluster. This supports the edgecluster deployment on any location as well as to scale the on-premisesinfrastructures.● Tier 3 - Edge Clusters. Application images are cached within the edge clusters in anImage Datastore Replica on dedicated Replica Server hosts to minimize imagetransferring. The Replica Servers make use of a specialized distribution system tomake the images available to all cluster hosts and support snapshotting for hostfailure recovery.Figure 2. Edge storage system for the edge cloud architecture.Within each tier-3 edge cluster, images are transferred between the Image and SystemDatastores by using an enhanced SSH transfer driver with replica caching and snapshottingthat greatly improves its scalability, performance and reliability. The new replica modecaches the images in each cluster so they are available close to the hypervisors to reducethe bandwidth requirements to the tier-2 Image Datastore servers and considerably reducedeployment times. This is especially important in highly distributed edge deployments wherecopying images from the tier-2 Image Datastores to the tier-3 edge clusters could be veryslow.This storage system has been implemented with lightweight technology components thattranslate into modest hardware requirements, like SATA SSDs and 10 gigabit networks.Moreover its deployment follows an hyperconverged approach that does not requirededicated servers to implement a distributed storage system. This reduces the complexity ofthe solution, enabling the use of the local storage area of the cloud cluster hosts.4.2. Application ImagesApplication images are based on files with qcow2 (QEMU copy on write) format, which usesa disk image storage optimization strategy that delays allocation of storage until it is actuallyneeded, allows storing changes made to a read-only base image by using Copy-on-Write,and supports snapshots. Therefore, the use of qcow2 files reduces file transfer andinstantiation times with minimal overhead, eases backup solutions, and provides advancedfeatures like snapshotting in an efficient way.The applications run from the direct attached storage of the hosts, to maximize the availableI/O performance delivered to the applications. The I/O performance is close to that of thenative host and it is only impacted by the virtualization layer. In order to provide anestimation of this overhead we have run the Flexible I/O tester (fio) benchmark1 on both thephysical machine and one VM.The measures have been taken on a t1.small metal instance on Equinix Metal (1x Intel AtomC2550 @ 2.4Ghz, 8 GB RAM, 1x 80 GB SSD and 2x 1 Gbps NICs with bonded ports) and aVM (KVM kernel 4.18.0-147.8.1.el8_1.x86_64, Libvirt-6.0.0 and QEMU with cache=none andio=native) running on the very same server.Table 1 shows the average bandwidth obtained while running the benchmark for bothsequential and random read and write operations. The performance is similar between thephysical machine and the VM, with differences of less than 5%. Note that some latency ishidden by the virtual page cache.Physical machine VMSequential Read 179 171Sequential Write 168 181Random Read 85 90Random Write 90 95Table 1. Average I/O bandwidth (MB/s) for read/write operations on the physical machineand VM.In order to improve the availability of the edge cluster ¿blocks?, live migration is supportedwithin the same cluster. Application snapshots are also kept within the edge cluster (Tier 3)to enable fast recovery from the last application checkpoint.I/O noise generated by the snapshot operation could reduce the I/O of neighbourapplications. However, in our solution, this cost is negligible as it is based on the QEMURedirect-on-Write feature, which avoids changing the original image.To reduce the cost to move the recovery snapshots to the cluster replica server, adelta-transfer algorithm is used to reduce the information transferred to the server. However,this time will increase as the contents of the disk image diverges from the original.1 https://github.com/axboe/fioThe VM recovery time is similar to recovering a VM without any snapshot, as the baseimages are already located on the cluster replica (Tier 3). The only additional overhead is thetransfer of the disk image snapshot, already available in the edge cluster as well.5. Preliminary resultsThe expected deployment times depend on the interconnection links between tiers and diskimage sizes. For a metal remote cluster (Tier 3) in Frankfurt, connected using public internetlinks to a Tier 2 image datastore placed in Amsterdam, we have obtained averagesdeployment times of 5 s (hot caches) and 35 s (first-time, empty cache) for applicationsusing disk images of 0.5 GiB. Average deployment times for multiple VMs on mentionedclusters with cached and without cached images are described in Figure 3. When theclusters are deployed on-premises using 1 Gbps links the deployment time is always 5 s.Figure 3. Comparison of deployment times (s) with and without cached images.AcknowledgmentsThis work was supported by the Ministerio de Ciencia, Innovación y Universidades throughthe EdgeCloud research project (RTI2018-096465-B-I00) and by the Comunidad de Madridthrough the EdgeData research program (P2018/TCS4499).References[1] J. Benet: IPFS - Content Addressed, Versioned, P2P File System (DRAFT 3),https://github.com/ipfs/papers/raw/master/ipfs-cap2pfs/ipfs-p2p-file-system.pdf[2] P. Maymounkov and D. Mazieres. Kademlia: A peer-to-peer information system basedon the xor metric. In Peer-to-Peer Systems, pages 53–65. Springer, 2002.[3] Sage A. Weil, Andrew W. Leung, Scott A. Brandt, and Carlos Maltzahn. 2007. RADOS:a scalable, reliable storage service for petabyte-scale storage clusters. In Proceedingsof the 2nd international workshop on Petascale data storage: held in conjunction withSupercomputing '07 (PDSW '07). Association for Computing Machinery, New York, NY,USA, 35–44. https://doi.org/10.1145/1374596.1374606[4] N. Sreekumar, A. Chandra and J. Weissman, "Position Paper: Towards a RobustEdge-Native Storage System," 2020 IEEE/ACM Symposium on Edge Computing(SEC), San Jose, CA, USA, 2020, pp. 285-292, doi: 10.1109/SEC50012.2020.00040.[5] Bastien Confais, Adrien Lebre, Benoît Parrein. A Fog storage software architecture forthe Internet of Things. Advances in Edge Computing: Massive Parallel Processing andApplications, IOS Press, pp.61-105, 2020, Advances in Parallel Computing,978-1-64368-062-0. doi: ff10.3233/APC200004[6] Mahmud R., Kotagiri R., Buyya R. (2018) Fog Computing: A Taxonomy, Survey andFuture Directions. Internet of Everything. Internet of Things (Technology,Communications and Computing). Springer, Singapore.https://doi.org/10.1007/978-981-10-5861-5_5[7] Confais B., Lebre A., Parrein B. (2017) Performance Analysis of Object Store Systemsin a Fog and Edge Computing Infrastructure. In: Transactions on Large-Scale Data-and Knowledge-Centered Systems XXXIII. Lecture Notes in Computer Science, vol10430. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-55696-2_2[8] Lakshman, A., & Malik, P. (2010). Cassandra: a decentralized structured storagesystem. ACM SIGOPS Operating Systems Review, 44(2), 35-40.[9] Brian F. Cooper, Adam Silberstein, Erwin Tam, Raghu Ramakrishnan, and RussellSears. 2010. Benchmarking cloud serving systems with YCSB. In Proceedings of the1st ACM symposium on Cloud computing (SoCC '10). Association for ComputingMachinery, New York, NY, USA, 143–154.DOI:https://doi.org/10.1145/1807128.1807152[10] Delgado Peris, A., Hernández, J.M. & Huedo, E. Distributed Late-binding Schedulingand Cooperative Data Caching. J Grid Computing 15, 235–256 (2017).https://doi.org/10.1007/s10723-016-9374-y

English

Disk Image Storage, Distribution and Caching for Edge Cloud Infrastructures

https://eprints.ucm.es/75308/1/Paper%20Edge%20Storage%20%281%29.pdf

Disk Image Storage, Distribution and Caching for Edge Cloud Infrastructures

Abstract

Similar works

Full text

Available Versions

Docta Complutense