An improved genetic algorithm for cost-effective data-intensive service composition

The explosion of digital data and the dependence on data-intensive services have been recognized as the most significant characteristics of IT trends in the current decade. Designing workflow of data-intensive services requires data analysis from multiple sources to get required composite services. Composing such services requires effective transfer of large data. Thus many new challenges are posed to control the cost and revenue of the whole composition. This paper addresses the data-intensive service composition and presents an innovative data-intensive service selection algorithm based on a modified genetic algorithm. The performance of this new algorithm is also tested by simulations and compared against other traditional approaches, such as mix integer programming. The contributions of this paper are three folds: 1) An economical model for data-intensive service provision is proposed, 2) An extensible QoS model is also proposed to calculate the QoS values of data-intensive services, 3) Finally, a modified genetic algorithm-based approach is introduced to compose data-intensive services. A local selection method with modifications of crossover and mutation operators is adopted for this algorithm. The results of experiments will demonstrate the scalability and effectiveness of our proposed algorithm

A Taxonomy of Workflow Management Systems for Grid Computing

With the advent of Grid and application technologies, scientists and engineers are building more and more complex applications to manage and process large data sets, and execute scientific experiments on distributed resources. Such application scenarios require means for composing and executing complex workflows. Therefore, many efforts have been made towards the development of workflow management systems for Grid computing. In this paper, we propose a taxonomy that characterizes and classifies various approaches for building and executing workflows on Grids. We also survey several representative Grid workflow systems developed by various projects world-wide to demonstrate the comprehensiveness of the taxonomy. The taxonomy not only highlights the design and engineering similarities and differences of state-of-the-art in Grid workflow systems, but also identifies the areas that need further research.Comment: 29 pages, 15 figure

Improved self-management of datacenter systems applying machine learning

Autonomic Computing is a Computer Science and Technologies research area, originated during mid 2000's. It focuses on optimization and improvement of complex distributed computing systems through self-control and self-management. As distributed computing systems grow in complexity, like multi-datacenter systems in cloud computing, the system operators and architects need more help to understand, design and optimize manually these systems, even more when these systems are distributed along the world and belong to different entities and authorities. Self-management lets these distributed computing systems improve their resource and energy management, a very important issue when resources have a cost, by obtaining, running or maintaining them. Here we propose to improve Autonomic Computing techniques for resource management by applying modeling and prediction methods from Machine Learning and Artificial Intelligence. Machine Learning methods can find accurate models from system behaviors and often intelligible explanations to them, also predict and infer system states and values. These models obtained from automatic learning have the advantage of being easily updated to workload or configuration changes by re-taking examples and re-training the predictors. So employing automatic modeling and predictive abilities, we can find new methods for making "intelligent" decisions and discovering new information and knowledge from systems. This thesis departs from the state of the art, where management is based on administrators expertise, well known data, ad-hoc studied algorithms and models, and elements to be studied from computing machine point of view; to a novel state of the art where management is driven by models learned from the same system, providing useful feedback, making up for incomplete, missing or uncertain data, from a global network of datacenters point of view. - First of all, we cover the scenario where the decision maker works knowing all pieces of information from the system: how much will each job consume, how is and will be the desired quality of service, what are the deadlines for the workload, etc. All of this focusing on each component and policy of each element involved in executing these jobs. -Then we focus on the scenario where instead of fixed oracles that provide us information from an expert formula or set of conditions, machine learning is used to create these oracles. Here we look at components and specific details while some part of the information is not known and must be learned and predicted. - We reduce the problem of optimizing resource allocations and requirements for virtualized web-services to a mathematical problem, indicating each factor, variable and element involved, also all the constraints the scheduling process must attend to. The scheduling problem can be modeled as a Mixed Integer Linear Program. Here we face an scenario of a full datacenter, further we introduce some information prediction. - We complement the model by expanding the predicted elements, studying the main resources (this is CPU, Memory and IO) that can suffer from noise, inaccuracy or unavailability. Once learning predictors for certain components let the decision making improve, the system can become more ¿expert-knowledge independent¿ and research can focus on an scenario where all the elements provide noisy, uncertainty or private information. Also we introduce to the management optimization new factors as for each datacenter context and costs may change, turning the model as "multi-datacenter" - Finally, we review of the cost of placing datacenters depending on green energy sources, and distribute the load according to green energy availability

A Research Perspective on Data Management Techniques for Federated Cloud Environment

Cloud computing has given a large scope of improvement in processing, storage and retrieval of data that is generated in huge amount from devices and users. Heterogenous devices and users generates the multidisciplinary data that needs to take care for easy and efficient storage and fast retrieval by maintaining quality and service level agreements. By just storing the data in cloud will not full fill the user requirements, the data management techniques has to be applied so that data adaptiveness and proactiveness characteristics are upheld. To manage the effectiveness of entire eco system a middleware must be there in between users and cloud service providers. Middleware has set of events and trigger based policies that will act on generated data to intermediate users and cloud service providers. For cloud service providers to deliver an efficient utilization of resources is one of the major issues and has scope of improvement in the federation of cloud service providers to fulfill user’s dynamic demands. Along with providing adaptiveness of data management in the middleware layer is challenging. In this paper, the policies of middleware for adaptive data management have been reviewed extensively. The main objectives of middleware are also discussed to accomplish high throughput of cloud service providers by means of federation and qualitative data management by means of adaptiveness and proactiveness. The cloud federation techniques have been studied thoroughly along with the pros and cons of it. Also, the strategies to do management of data has been exponentially explored

Energy-Efficient Softwarized Networks: A Survey

With the dynamic demands and stringent requirements of various applications, networks need to be high-performance, scalable, and adaptive to changes. Researchers and industries view network softwarization as the best enabler for the evolution of networking to tackle current and prospective challenges. Network softwarization must provide programmability and flexibility to network infrastructures and allow agile management, along with higher control for operators. While satisfying the demands and requirements of network services, energy cannot be overlooked, considering the effects on the sustainability of the environment and business. This paper discusses energy efficiency in modern and future networks with three network softwarization technologies: SDN, NFV, and NS, introduced in an energy-oriented context. With that framework in mind, we review the literature based on network scenarios, control/MANO layers, and energy-efficiency strategies. Following that, we compare the references regarding approach, evaluation method, criterion, and metric attributes to demonstrate the state-of-the-art. Last, we analyze the classified literature, summarize lessons learned, and present ten essential concerns to open discussions about future research opportunities on energy-efficient softwarized networks.Comment: Accepted draft for publication in TNSM with minor updates and editin

Game-theoretic, market and meta-heuristics approaches for modelling scheduling and resource allocation in grid systems

Task scheduling and resource allocation are the crucial issues in any large scale distributed system, such as Computational Grids (CGs). However, traditional computational models and resolution methods cannot effectively tackle the complex nature of Grid, where the resources and users belong to many administrative domains with their own access policies, users' privileges, etc. Recently, researchers are investigating the use of game theoretic approaches for modelling task and resource allocation problems in CGs. In this paper, we present a compact survey of the most relevant research proposals in the literature to use game-based models for the resource allocation problems and their resolution using metaheuristic methods. We emphasize the need of the translation of the traditional economical models into the game scenarios and the use of metaheuristic schedulers for solving such games in order to address the new complex scheduling and allocation criterions. We study the case of asymmetric Stackelberg game used for modelling the Grid users' behavior, where the security and reliability criterions are aggregated and defined as the users' costs functions. The obtained results show the efficiency of the hybridization of heuristic-based approaches with game models, which enables to include additional requirements and features into the computational models and tackle more effectively the resolution of the applied schedulers.Peer ReviewedPostprint (published version

A new MDA-SOA based framework for intercloud interoperability

Cloud computing has been one of the most important topics in Information Technology which aims to assure scalable and reliable on-demand services over the Internet. The expansion of the application scope of cloud services would require cooperation between clouds from different providers that have heterogeneous functionalities. This collaboration between different cloud vendors can provide better Quality of Services (QoS) at the lower price. However, current cloud systems have been developed without concerns of seamless cloud interconnection, and actually they do not support intercloud interoperability to enable collaboration between cloud service providers. Hence, the PhD work is motivated to address interoperability issue between cloud providers as a challenging research objective. This thesis proposes a new framework which supports inter-cloud interoperability in a heterogeneous computing resource cloud environment with the goal of dispatching the workload to the most effective clouds available at runtime. Analysing different methodologies that have been applied to resolve various problem scenarios related to interoperability lead us to exploit Model Driven Architecture (MDA) and Service Oriented Architecture (SOA) methods as appropriate approaches for our inter-cloud framework. Moreover, since distributing the operations in a cloud-based environment is a nondeterministic polynomial time (NP-complete) problem, a Genetic Algorithm (GA) based job scheduler proposed as a part of interoperability framework, offering workload migration with the best performance at the least cost. A new Agent Based Simulation (ABS) approach is proposed to model the inter-cloud environment with three types of agents: Cloud Subscriber agent, Cloud Provider agent, and Job agent. The ABS model is proposed to evaluate the proposed framework.Fundação para a Ciência e a Tecnologia (FCT) - (Referencia da bolsa: SFRH SFRH / BD / 33965 / 2009) and EC 7th Framework Programme under grant agreement n° FITMAN 604674 (http://www.fitman-fi.eu

QoS-aware predictive workflow scheduling

This research places the basis of QoS-aware predictive workflow scheduling. This research novel contributions will open up prospects for future research in handling complex big workflow applications with high uncertainty and dynamism. The results from the proposed workflow scheduling algorithm shows significant improvement in terms of the performance and reliability of the workflow applications