A Scalable and Economical Method for Distributed Data Processing

This research paper presents a distributed data processing approach that involves the establishment of virtual machines, the creation of a distributed system, and the processing of data to obtain desired results. The proposed method aims to provide a simple and cost-effective solution for distributed data processing, with the ability to scale infrastructure according to the specific needs. Furthermore, a distributed data processing system is introduced, comprising virtual machines equipped with specialized software to facilitate the establishment of the distributed system. The method offers practical advantages in terms of implementation simplicity, reduced infrastructure costs, and improved resource utilization

Proactive cloud management for highly heterogeneous multi-cloud infrastructures

Various literature studies demonstrated that the cloud computing paradigm can help to improve availability and performance of applications subject to the problem of software anomalies. Indeed, the cloud resource provisioning model enables users to rapidly access new processing resources, even distributed over different geographical regions, that can be promptly used in the case of, e.g., crashes or hangs of running machines, as well as to balance the load in the case of overloaded machines. Nevertheless, managing a complex geographically-distributed cloud deploy could be a complex and time-consuming task. Autonomic Cloud Manager (ACM) Framework is an autonomic framework for supporting proactive management of applications deployed over multiple cloud regions. It uses machine learning models to predict failures of virtual machines and to proactively redirect the load to healthy machines/cloud regions. In this paper, we study different policies to perform efficient proactive load balancing across cloud regions in order to mitigate the effect of software anomalies. These policies use predictions about the mean time to failure of virtual machines. We consider the case of heterogeneous cloud regions, i.e regions with different amount of resources, and we provide an experimental assessment of these policies in the context of ACM Framework

Enhancing Job Scheduling of an Atmospheric Intensive Data Application

Nowadays, e-Science applications involve great deal of data to have more accurate analysis. One of its application domains is the Radio Occultation which manages satellite data. Grid Processing Management is a physical infrastructure geographically distributed based on Grid Computing, that is implemented for the overall processing Radio Occultation analysis. After a brief description of algorithms adopted to characterize atmospheric profiles, the paper presents an improvement of job scheduling in order to decrease processing time and optimize resource utilization. Extension of grid computing capacity is implemented by virtual machines in existing physical Grid in order to satisfy temporary job requests. Also scheduling plays an important role in the infrastructure that is handled by a couple of schedulers which are developed to manage data automaticall

Towards Formal Interaction-Based Models of Grid Computing Infrastructures

Grid computing (GC) systems are large-scale virtual machines, built upon a massive pool of resources (processing time, storage, software) that often span multiple distributed domains. Concurrent users interact with the grid by adding new tasks; the grid is expected to assign resources to tasks in a fair, trustworthy way. These distinctive features of GC systems make their specification and verification a challenging issue. Although prior works have proposed formal approaches to the specification of GC systems, a precise account of the interaction model which underlies resource sharing has not been yet proposed. In this paper, we describe ongoing work aimed at filling in this gap. Our approach relies on (higher-order) process calculi: these core languages for concurrency offer a compositional framework in which GC systems can be precisely described and potentially reasoned about.Comment: In Proceedings DCM 2013, arXiv:1403.768

GraphTheta: A Distributed Graph Neural Network Learning System With Flexible Training Strategy

Graph neural networks (GNNs) have been demonstrated as a powerful tool for analysing non-Euclidean graph data. However, the lack of efficient distributed graph learning (GL) systems severely hinders applications of GNNs, especially when graphs are big and GNNs are relatively deep. Herein, we present GraphTheta, a novel distributed and scalable GL system implemented in vertex-centric graph programming model. GraphTheta is the first GL system built upon distributed graph processing with neural network operators implemented as user-defined functions. This system supports multiple training strategies, and enables efficient and scalable big graph learning on distributed (virtual) machines with low memory each. To facilitate graph convolution implementations, GraphTheta puts forward a new GL abstraction named NN-TGAR to bridge the gap between graph processing and graph deep learning. A distributed graph engine is proposed to conduct the stochastic gradient descent optimization with a hybrid-parallel execution. Moreover, we add support for a new cluster-batched training strategy besides global-batch and mini-batch. We evaluate GraphTheta using a number of datasets with network size ranging from small-, modest- to large-scale. Experimental results show that GraphTheta can scale well to 1,024 workers for training an in-house developed GNN on an industry-scale Alipay dataset of 1.4 billion nodes and 4.1 billion attributed edges, with a cluster of CPU virtual machines (dockers) of small memory each (5$\sim$12GB). Moreover, GraphTheta obtains comparable or better prediction results than the state-of-the-art GNN implementations, demonstrating its capability of learning GNNs as well as existing frameworks, and can outperform DistDGL by up to $2.02\times$ with better scalability. To the best of our knowledge, this work presents the largest edge-attributed GNN learning task conducted in the literature.Comment: 18 pages, 14 figures, 5 table

Analysis and evaluation of a XEN based virtual router

Virtualization techniques are applied to improve features like isolation, security, mobility and dynamic reconfiguration in distributed systems. To introduce these advantages into the network where they are highly required, an interesting approach is to virtualize the internet routers themselves. This technique could enable several virtual networks of different types, owners and protocols to coexist inside one physical network. In this study, we propose a model of a virtual router we have implemented with XEN and we evaluate its properties. We show that the performance is close to the performance of non virtualized software routers, but causes an important processing overhead and unfairness in the share of the resources. We study the impact of the virtual machine scheduler parameters on the network performance and we show that the module which is responsible of forwarding the packets between the virtual machines and the physical interfaces is the critical point of network communications

Data location aware scheduling for virtual Hadoop cluster deployment on private cloud computing environment

With the advancements of Internet-of-Things (IoT) and Machine-to-Machine Communications (M2M), the ability to generate massive amount of streaming data from sensory devices in distributed environment is inevitable. A common practice nowadays is to process these data in a high-performance computing infrastructure, such as cloud. Cloud platform has the ability to deploy Hadoop ecosystem on virtual clusters. In cloud configuration with different geographical regions, virtual machines (VMs) that are part of virtual cluster are placed randomly. Prior to processing, data have to be transferred to the regional sites with VMs for data locality purposes. In this paper, a provisioning strategy with data-location aware deployment for virtual cluster will be proposed, as to localize and provision the cluster near to the storage. The proposed mechanism reduces the network distance between virtual cluster and storage, resulting in reduced job completion times