Cloud computing resource scheduling and a survey of its evolutionary approaches

A disruptive technology fundamentally transforming the way that computing services are delivered, cloud computing offers information and communication technology users a new dimension of convenience of resources, as services via the Internet. Because cloud provides a finite pool of virtualized on-demand resources, optimally scheduling them has become an essential and rewarding topic, where a trend of using Evolutionary Computation (EC) algorithms is emerging rapidly. Through analyzing the cloud computing architecture, this survey first presents taxonomy at two levels of scheduling cloud resources. It then paints a landscape of the scheduling problem and solutions. According to the taxonomy, a comprehensive survey of state-of-the-art approaches is presented systematically. Looking forward, challenges and potential future research directions are investigated and invited, including real-time scheduling, adaptive dynamic scheduling, large-scale scheduling, multiobjective scheduling, and distributed and parallel scheduling. At the dawn of Industry 4.0, cloud computing scheduling for cyber-physical integration with the presence of big data is also discussed. Research in this area is only in its infancy, but with the rapid fusion of information and data technology, more exciting and agenda-setting topics are likely to emerge on the horizon

SkyCDS: A resilient content delivery service based on diversified cloud storage

Cloud-based storage is a popular outsourcing solution for organizations to deliver contents to end-users. However, there is a need for contingency plans to ensure service provision when the provider either suffers outages or is going out of business. This paper presents SkyCDS: a resilient content delivery service based on a publish/subscribe overlay over diversified cloud storage. SkyCDS splits the content delivery into metadata and content storage flow layers. The metadata flow layer is based on publish-subscribe patterns for insourcing the metadata control back to content owner. The storage layer is based on dispersal information over multiple cloud locations with which organizations outsource content storage in a controlled manner. In SkyCDS, the content dispersion is performed on the publisher side and the content retrieving process on the end-user side (the subscriber), which reduces the load on the organization side only to metadata management. SkyCDS also lowers the overhead of the content dispersion and retrieving processes by taking advantage of multi-core technology. A new allocation strategy based on cloud storage diversification and failure masking mechanisms minimize side effects of temporary, permanent cloud-based service outages and vendor lock-in. We developed a SkyCDS prototype that was evaluated by using synthetic workloads and a study case with real traces. Publish/subscribe queuing patterns were evaluated by using a simulation tool based on characterized metrics taken from experimental evaluation. The evaluation revealed the feasibility of SkyCDS in terms of performance, reliability and storage space profitability. It also shows a novel way to compare the storage/delivery options through risk assessment. (C) 2015 Elsevier B.V. All rights reserved.The work presented in this paper has been partially supported by EU under the COST programme Action IC1305, Network for Sustainable Ultrascale Computing (NESUS)

Secure Cloud-Edge Deployments, with Trust

Assessing the security level of IoT applications to be deployed to heterogeneous Cloud-Edge infrastructures operated by different providers is a non-trivial task. In this article, we present a methodology that permits to express security requirements for IoT applications, as well as infrastructure security capabilities, in a simple and declarative manner, and to automatically obtain an explainable assessment of the security level of the possible application deployments. The methodology also considers the impact of trust relations among different stakeholders using or managing Cloud-Edge infrastructures. A lifelike example is used to showcase the prototyped implementation of the methodology

Data Placement And Task Mapping Optimization For Big Data Workflows In The Cloud

Data-centric workflows naturally process and analyze a huge volume of datasets. In this new era of Big Data there is a growing need to enable data-centric workflows to perform computations at a scale far exceeding a single workstation\u27s capabilities. Therefore, this type of applications can benefit from distributed high performance computing (HPC) infrastructures like cluster, grid or cloud computing. Although data-centric workflows have been applied extensively to structure complex scientific data analysis processes, they fail to address the big data challenges as well as leverage the capability of dynamic resource provisioning in the Cloud. The concept of “big data workflows” is proposed by our research group as the next generation of data-centric workflow technologies to address the limitations of exist-ing workflows technologies in addressing big data challenges. Executing big data workflows in the Cloud is a challenging problem as work-flow tasks and data are required to be partitioned, distributed and assigned to the cloud execution sites (multiple virtual machines). In running such big data work-flows in the cloud distributed across several physical locations, the workflow execution time and the cloud resource utilization efficiency highly depends on the initial placement and distribution of the workflow tasks and datasets across the multiple virtual machines in the Cloud. Several workflow management systems have been developed for scientists to facilitate the use of workflows; however, data and work-flow task placement issue has not been sufficiently addressed yet. In this dissertation, I propose BDAP strategy (Big Data Placement strategy) for data placement and TPS (Task Placement Strategy) for task placement, which improve workflow performance by minimizing data movement across multiple virtual machines in the Cloud during the workflow execution. In addition, I propose CATS (Cultural Algorithm Task Scheduling) for workflow scheduling, which improve workflow performance by minimizing workflow execution cost. In this dissertation, I 1) formalize data and task placement problems in workflows, 2) propose a data placement algorithm that considers both initial input dataset and intermediate datasets obtained during workflow run, 3) propose a task placement algorithm that considers placement of workflow tasks before workflow run, 4) propose a workflow scheduling strategy to minimize the workflow execution cost once the deadline is provided by user and 5)perform extensive experiments in the distributed environment to validate that our proposed strategies provide an effective data and task placement solution to distribute and place big datasets and tasks into the appropriate virtual machines in the Cloud within reasonable time

InterCloud: Utility-Oriented Federation of Cloud Computing Environments for Scaling of Application Services

Cloud computing providers have setup several data centers at different geographical locations over the Internet in order to optimally serve needs of their customers around the world. However, existing systems do not support mechanisms and policies for dynamically coordinating load distribution among different Cloud-based data centers in order to determine optimal location for hosting application services to achieve reasonable QoS levels. Further, the Cloud computing providers are unable to predict geographic distribution of users consuming their services, hence the load coordination must happen automatically, and distribution of services must change in response to changes in the load. To counter this problem, we advocate creation of federated Cloud computing environment (InterCloud) that facilitates just-in-time, opportunistic, and scalable provisioning of application services, consistently achieving QoS targets under variable workload, resource and network conditions. The overall goal is to create a computing environment that supports dynamic expansion or contraction of capabilities (VMs, services, storage, and database) for handling sudden variations in service demands. This paper presents vision, challenges, and architectural elements of InterCloud for utility-oriented federation of Cloud computing environments. The proposed InterCloud environment supports scaling of applications across multiple vendor clouds. We have validated our approach by conducting a set of rigorous performance evaluation study using the CloudSim toolkit. The results demonstrate that federated Cloud computing model has immense potential as it offers significant performance gains as regards to response time and cost saving under dynamic workload scenarios.Comment: 20 pages, 4 figures, 3 tables, conference pape

BRACELET: Hierarchical Edge-Cloud Microservice Infrastructure for Scientific Instruments’ Lifetime Connectivity

Recent advances in cyber-infrastructure have enabled digital data sharing and ubiquitous network connectivity between scientific instruments and cloud-based storage infrastructure for uploading, storing, curating, and correlating of large amounts of materials and semiconductor fabrication data and metadata. However, there is still a significant number of scientific instruments running on old operating systems that are taken offline and cannot connect to the cloud infrastructure, due to security and performance concerns. In this paper, we propose BRACELET - an edge-cloud infrastructure that augments the existing cloud-based infrastructure with edge devices and helps to tackle the unique performance and security challenges that scientific instruments face when they are connected to the cloud through public network. With BRACELET, we put a networked edge device, called cloudlet, in between the scientific instruments and the cloud as the middle tier of a three-tier hierarchy. The cloudlet will shape and protect the data traffic from scientific instruments to the cloud, and will play a foundational role in keeping the instruments connected throughout its lifetime, and continuously providing the otherwise missing performance and security features for the instrument as its operating system ages.NSF Award Number 1659293NSF Award Number 1443013Ope

DALiuGE: A Graph Execution Framework for Harnessing the Astronomical Data Deluge

The Data Activated Liu Graph Engine - DALiuGE - is an execution framework for processing large astronomical datasets at a scale required by the Square Kilometre Array Phase 1 (SKA1). It includes an interface for expressing complex data reduction pipelines consisting of both data sets and algorithmic components and an implementation run-time to execute such pipelines on distributed resources. By mapping the logical view of a pipeline to its physical realisation, DALiuGE separates the concerns of multiple stakeholders, allowing them to collectively optimise large-scale data processing solutions in a coherent manner. The execution in DALiuGE is data-activated, where each individual data item autonomously triggers the processing on itself. Such decentralisation also makes the execution framework very scalable and flexible, supporting pipeline sizes ranging from less than ten tasks running on a laptop to tens of millions of concurrent tasks on the second fastest supercomputer in the world. DALiuGE has been used in production for reducing interferometry data sets from the Karl E. Jansky Very Large Array and the Mingantu Ultrawide Spectral Radioheliograph; and is being developed as the execution framework prototype for the Science Data Processor (SDP) consortium of the Square Kilometre Array (SKA) telescope. This paper presents a technical overview of DALiuGE and discusses case studies from the CHILES and MUSER projects that use DALiuGE to execute production pipelines. In a companion paper, we provide in-depth analysis of DALiuGE's scalability to very large numbers of tasks on two supercomputing facilities.Comment: 31 pages, 12 figures, currently under review by Astronomy and Computin

Mobiilse värkvõrgu protsessihaldus

Värkvõrk, ehk Asjade Internet (Internet of Things, lüh IoT) edendab lahendusi nagu nn tark linn, kus meid igapäevaselt ümbritsevad objektid on ühendatud infosüsteemidega ja ka üksteisega. Selliseks näiteks võib olla teekatete seisukorra monitoorimissüsteem. Võrku ühendatud sõidukitelt (nt bussidelt) kogutakse videomaterjali, mida seejärel töödeldakse, et tuvastada löökauke või lume kogunemist. Tavaliselt hõlmab selline lahendus keeruka tsentraalse süsteemi ehitamist. Otsuste langetamiseks (nt milliseid sõidukeid parasjagu protsessi kaasata) vajab keskne süsteem pidevat ühendust kõigi IoT seadmetega. Seadmete hulga kasvades võib keskne lahendus aga muutuda pudelikaelaks. Selliste protsesside disaini, haldust, automatiseerimist ja seiret hõlbustavad märkimisväärselt äriprotsesside halduse (Business Process Management, lüh BPM) valdkonna standardid ja tööriistad. Paraku ei ole BPM tehnoloogiad koheselt kasutatavad uute paradigmadega nagu Udu- ja Servaarvutus, mis tuleviku värkvõrgu jaoks vajalikud on. Nende puhul liigub suur osa otsustustest ja arvutustest üksikutest andmekeskustest servavõrgu seadmetele, mis asuvad lõppkasutajatele ja IoT seadmetele lähemal. Videotöötlust võiks teostada mini-andmekeskustes, mis on paigaldatud üle linna, näiteks bussipeatustesse. Arvestades IoT seadmete üha suurenevat hulka, vähendab selline koormuse jaotamine vähendab riski, et tsentraalne andmekeskust ülekoormamist. Doktoritöö uurib, kuidas mobiilsusega seonduvaid IoT protsesse taoliselt ümber korraldada, kohanedes pidevalt muutlikule, liikuvate seadmetega täidetud servavõrgule. Nimelt on ühendused katkendlikud, mistõttu otsuste langetus ja planeerimine peavad arvestama muuhulgas mobiilseadmete liikumistrajektoore. Töö raames valminud prototüüpe testiti Android seadmetel ja simulatsioonides. Lisaks valmis tööriistakomplekt STEP-ONE, mis võimaldab teadlastel hõlpsalt simuleerida ja analüüsida taolisi probleeme erinevais realistlikes stsenaariumites nagu seda on tark linn.The Internet of Things (IoT) promotes solutions such as a smart city, where everyday objects connect with info systems and each other. One example is a road condition monitoring system, where connected vehicles, such as buses, capture video, which is then processed to detect potholes and snow build-up. Building such a solution typically involves establishing a complex centralised system. The centralised approach may become a bottleneck as the number of IoT devices keeps growing. It relies on constant connectivity to all involved devices to make decisions, such as which vehicles to involve in the process. Designing, automating, managing, and monitoring such processes can greatly be supported using the standards and software systems provided by the field of Business Process Management (BPM). However, BPM techniques are not directly applicable to new computing paradigms, such as Fog Computing and Edge Computing, on which the future of IoT relies. Here, a lot of decision-making and processing is moved from central data-centers to devices in the network edge, near the end-users and IoT sensors. For example, video could be processed in mini-datacenters deployed throughout the city, e.g., at bus stops. This load distribution reduces the risk of the ever-growing number of IoT devices overloading the data center. This thesis studies how to reorganise the process execution in this decentralised fashion, where processes must dynamically adapt to the volatile edge environment filled with moving devices. Namely, connectivity is intermittent, so decision-making and planning need to involve factors such as the movement trajectories of mobile devices. We examined this issue in simulations and with a prototype for Android smartphones. We also showcase the STEP-ONE toolset, allowing researchers to conveniently simulate and analyse these issues in different realistic scenarios, such as those in a smart city.  https://www.ester.ee/record=b552551