Data Placement And Task Mapping Optimization For Big Data Workflows In The Cloud

Data-centric workflows naturally process and analyze a huge volume of datasets. In this new era of Big Data there is a growing need to enable data-centric workflows to perform computations at a scale far exceeding a single workstation\u27s capabilities. Therefore, this type of applications can benefit from distributed high performance computing (HPC) infrastructures like cluster, grid or cloud computing. Although data-centric workflows have been applied extensively to structure complex scientific data analysis processes, they fail to address the big data challenges as well as leverage the capability of dynamic resource provisioning in the Cloud. The concept of “big data workflows” is proposed by our research group as the next generation of data-centric workflow technologies to address the limitations of exist-ing workflows technologies in addressing big data challenges. Executing big data workflows in the Cloud is a challenging problem as work-flow tasks and data are required to be partitioned, distributed and assigned to the cloud execution sites (multiple virtual machines). In running such big data work-flows in the cloud distributed across several physical locations, the workflow execution time and the cloud resource utilization efficiency highly depends on the initial placement and distribution of the workflow tasks and datasets across the multiple virtual machines in the Cloud. Several workflow management systems have been developed for scientists to facilitate the use of workflows; however, data and work-flow task placement issue has not been sufficiently addressed yet. In this dissertation, I propose BDAP strategy (Big Data Placement strategy) for data placement and TPS (Task Placement Strategy) for task placement, which improve workflow performance by minimizing data movement across multiple virtual machines in the Cloud during the workflow execution. In addition, I propose CATS (Cultural Algorithm Task Scheduling) for workflow scheduling, which improve workflow performance by minimizing workflow execution cost. In this dissertation, I 1) formalize data and task placement problems in workflows, 2) propose a data placement algorithm that considers both initial input dataset and intermediate datasets obtained during workflow run, 3) propose a task placement algorithm that considers placement of workflow tasks before workflow run, 4) propose a workflow scheduling strategy to minimize the workflow execution cost once the deadline is provided by user and 5)perform extensive experiments in the distributed environment to validate that our proposed strategies provide an effective data and task placement solution to distribute and place big datasets and tasks into the appropriate virtual machines in the Cloud within reasonable time

Modelling activity times by hybrid synthetic method

Uncertain (manual) activity times impact a number of manufacturing system modules: plant and layout design, capacity analysis, operator assignment, process planning, scheduling and simulation. Direct observation cannot be used for non-existent production lines. A hybrid direct observation/synthetic method derived from Method Time Measurement available in industry is proposed. To determine accurate activity times required by heuristics and metaheuristics optimisation, manufacturing system modules are modelled by MILP and operator efficiency parameters are used for time standardisation. Among human factors considered are skill and ergonomics. Application to the sterilisation of reusable medical devices is extensively described. Experimental data taken from observation on the field and a worst-case date have shown the model direct applicability for professionals also to non-manufacturing cases

Cost based rescheduling approach to handle disruptions in flexible manufacturing systems

Rescheduling is an essential operating task to efficiently tackle uncertainties and unexpected events frequently encountered in today's complex and flexible manufacturing systems. The main purpose of this thesis is to develop a real time reactive scheduling methodology in order to respond to such disturbances and uncertainties in a cost efficient manner. In order to assess the impact of schedule changes, a compound rescheduling cost function is developed based on machine, job, and material related rescheduling activities. A Total Rescheduling (TR) approach based on the Filtered-Beam-Search-heuristic algorithm (FBS) is proposed to generate a prespecified number of cost efficient suboptimal schedules by using the proposed cost function in case of each disruption. Thereafter, the current schedule is replaced by the alternative schedule which causes the minimum rescheduling cost. Responding to each single disruption with TR may cause system nervousness and increase the operational cost. Hence, a partial rescheduling approach is developed by a Modified Filtered-Beam-Search-heuristic algorithm (MFBSR) in order to generate a prespecified number of sub optimal cost-efficient schedules with a lower rescheduling cost and fewer deviations than TR. In order to validate the performance of the proposed methodologies, TR and MFBSR, different case studies and experimental designs have been performed considering various disruption scenarios. The performance of the suggested methods in terms of rescheduling cost, makespan efficiency and stability have been compared with similar rescheduling and repair methods in the literature. The results reveal that the proposed methodologies could be considered as competitive methods in responding to disruptions in flexible manufacturing system

Optimal Algorithms and a PTAS for Cost-Aware Scheduling

We consider a natural generalization of classical scheduling problems in which using a time unit for processing a job causes some time-dependent cost which must be paid in addition to the standard scheduling cost. We study the scheduling objectives of minimizing the makespan and the sum of (weighted) completion times. It is not dicult to derive a polynomial-time algorithm for preemptive scheduling to minimize the makespan on unrelated machines. The problem of minimizing the total (weighted) completion time is considerably harder, even on a single machine. We present a polynomial-time algorithm that computes for any given sequence of jobs an optimal schedule, i.e., the optimal set of time-slots to be used for scheduling jobs according to the given sequence. This result is based on dynamic programming using a subtle analysis of the structure of optimal solutions and a potential function argument. With this algorithm, we solve the unweighted problem optimally in polynomial time. Furthermore, we argue that there is a (4+")-approximation algorithm for the strongly NP-hard problem with individual job weights. For this weighted version, we also give a PTAS based on a dual scheduling approach introduced for scheduling on a machine of varying speed

Random Keys Genetic Algorithms Scheduling and Rescheduling Systems for Common Production Systems

The majority of scheduling research deals with problems in specific production environments with specific objective functions. However, in many cases, more than one problem type and/or objective function exists, resulting in the need for a more generic and flexible system to generate schedules. Furthermore, most of the published scheduling research focuses on creating an optimal or near optimal initial schedule during the planning phase. However, after production processes start, circumstances like machine breakdowns, urgent jobs, and other unplanned events may render the schedule suboptimal, obsolete or even infeasible resulting in a rescheduling problem, which is typically also addressed for a specific production environment, constraints, and objective functions. This dissertation introduces a generic framework consisting of models and algorithms based on Random Keys Genetic Algorithms (RKGA) to handle both the scheduling and rescheduling problems in the most common production environments and for various types of objective functions. The Scheduling system produces predictive (initial) schedules for environments including single machines, flow shops, job shops and parallel machine production systems to optimize regular objective functions such as the Makespan and the Total Tardiness as well as non-regular objective functions such as the Total Earliness and Tardiness. To deal with the rescheduling problem, and using as a basis the same RKGA, a reactive Rescheduling system capable of repairing initial schedules after the occurrence of unexpected events is introduced. The reactive Rescheduling system was designed not only to optimize regular and non-regular objective functions but also to minimize the instability, a very important aspect in rescheduling to avoid shop chaos due to disruptions. Minimizing both schedule inefficiency and instability, however, turns the problem into a multi-objective optimization problem, which is even more difficult to solve. The computational experiments for the predictive model show that it is able to produce optimal or near optimal schedules to benchmark problems for different production environments and objective functions. Additional computational experiments conducted to test the reactive Rescheduling system under two types of unexpected events, machine breakdowns and the arrival of a rush job, show that the proposed framework and algorithms are robust in handling various problem types and computationally reasonable

Advances and Novel Approaches in Discrete Optimization

Discrete optimization is an important area of Applied Mathematics with a broad spectrum of applications in many fields. This book results from a Special Issue in the journal Mathematics entitled ‘Advances and Novel Approaches in Discrete Optimization’. It contains 17 articles covering a broad spectrum of subjects which have been selected from 43 submitted papers after a thorough refereeing process. Among other topics, it includes seven articles dealing with scheduling problems, e.g., online scheduling, batching, dual and inverse scheduling problems, or uncertain scheduling problems. Other subjects are graphs and applications, evacuation planning, the max-cut problem, capacitated lot-sizing, and packing algorithms

Power Modeling and Resource Optimization in Virtualized Environments

The provisioning of on-demand cloud services has revolutionized the IT industry. This emerging paradigm has drastically increased the growth of data centers (DCs) worldwide. Consequently, this rising number of DCs is contributing to a large amount of world total power consumption. This has directed the attention of researchers and service providers to investigate a power-aware solution for the deployment and management of these systems and networks. However, these solutions could be bene\ufb01cial only if derived from a precisely estimated power consumption at run-time. Accuracy in power estimation is a challenge in virtualized environments due to the lack of certainty of actual resources consumed by virtualized entities and of their impact on applications\u2019 performance. The heterogeneous cloud, composed of multi-tenancy architecture, has also raised several management challenges for both service providers and their clients. Task scheduling and resource allocation in such a system are considered as an NP-hard problem. The inappropriate allocation of resources causes the under-utilization of servers, hence reducing throughput and energy e\ufb03ciency. In this context, the cloud framework needs an e\ufb00ective management solution to maximize the use of available resources and capacity, and also to reduce the impact of their carbon footprint on the environment with reduced power consumption. This thesis addresses the issues of power measurement and resource utilization in virtualized environments as two primary objectives. At \ufb01rst, a survey on prior work of server power modeling and methods in virtualization architectures is carried out. This helps investigate the key challenges that elude the precision of power estimation when dealing with virtualized entities. A di\ufb00erent systematic approach is then presented to improve the prediction accuracy in these networks, considering the resource abstraction at di\ufb00erent architectural levels. Resource usage monitoring at the host and guest helps in identifying the di\ufb00erence in performance between the two. Using virtual Performance Monitoring Counters (vPMCs) at a guest level provides detailed information that helps in improving the prediction accuracy and can be further used for resource optimization, consolidation and load balancing. Later, the research also targets the critical issue of optimal resource utilization in cloud computing. This study seeks a generic, robust but simple approach to deal with resource allocation in cloud computing and networking. The inappropriate scheduling in the cloud causes under- and over- utilization of resources which in turn increases the power consumption and also degrades the system performance. This work \ufb01rst addresses some of the major challenges related to task scheduling in heterogeneous systems. After a critical analysis of existing approaches, this thesis presents a rather simple scheduling scheme based on the combination of heuristic solutions. Improved resource utilization with reduced processing time can be achieved using the proposed energy-e\ufb03cient scheduling algorithm

Energy Efficient Policies, Scheduling, and Design for Sustainable Manufacturing Systems

Climate mitigation, more stringent regulations, rising energy costs, and sustainable manufacturing are pushing researchers to focus on energy efficiency, energy flexibility, and implementation of renewable energy sources in manufacturing systems. This thesis aims to analyze the main works proposed regarding these hot topics, and to fill the gaps in the literature. First, a detailed literature review is proposed. Works regarding energy efficiency in different manufacturing levels, in the assembly line, energy saving policies, and the implementation of renewable energy sources are analyzed. Then, trying to fill the gaps in the literature, different topics are analyzed more in depth. In the single machine context, a mathematical model aiming to align the manufacturing power required to a renewable energy supply in order to obtain the maximum profit is developed. The model is applied to a single work center powered by the electric grid and by a photovoltaic system; afterwards, energy storage is also added to the power system. Analyzing the job shop context, switch off policies implementing workload approach and scheduling considering variable speed of the machines and power constraints are proposed. The direct and indirect workloads of the machines are considered to support the switch on/off decisions. A simulation model is developed to test the proposed policies compared to others presented in the literature. Regarding the job shop scheduling, a fixed and variable power constraints are considered, assuming the minimization of the makespan as the objective function. Studying the factory level, a mathematical model to design a flow line considering the possibility of using switch-off policies is developed. The design model for production lines includes a targeted imbalance among the workstations to allow for defined idle time. Finally, the main findings, results, and the future directions and challenges are presented