Designing a scalable dynamic load -balancing algorithm for pipelined single program multiple data applications on a non-dedicated heterogeneous network of workstations

Dynamic load balancing strategies have been shown to be the most critical part of an efficient implementation of various applications on large distributed computing systems. The need for dynamic load balancing strategies increases when the underlying hardware is a non-dedicated heterogeneous network of workstations (HNOW). This research focuses on the single program multiple data (SPMD) programming model as it has been extensively used in parallel programming for its simplicity and scalability in terms of computational power and memory size.;This dissertation formally defines and addresses the problem of designing a scalable dynamic load-balancing algorithm for pipelined SPMD applications on non-dedicated HNOW. During this process, the HNOW parameters, SPMD application characteristics, and load-balancing performance parameters are identified.;The dissertation presents a taxonomy that categorizes general load balancing algorithms and a methodology that facilitates creating new algorithms that can harness the HNOW computing power and still preserve the scalability of the SPMD application.;The dissertation devises a new algorithm, DLAH (Dynamic Load-balancing Algorithm for HNOW). DLAH is based on a modified diffusion technique, which incorporates the HNOW parameters. Analytical performance bound for the worst-case scenario of the diffusion technique has been derived.;The dissertation develops and utilizes an HNOW simulation model to conduct extensive simulations. These simulations were used to validate DLAH and compare its performance to related dynamic algorithms. The simulations results show that DLAH algorithm is scalable and performs well for both homogeneous and heterogeneous networks. Detailed sensitivity analysis was conducted to study the effects of key parameters on performance

Scheduling Data-Parallel Computations on Heterogeneous and Time-Shared Environments

This paper addresses the problem of load balancing data-parallel computations on heterogeneous and time-shared parallel computing environments. Load imbalance in these environments may be introduced by the different capacities of processors populating a computer, or by the sharing of the same computational resources among several users. A evenly partitioned code, which on a homogeneous system runs with a perfect load balance, may perform poorly if the processors have different speeds or loads. To solve this problem for data-parallel computations, we propose a run-time support for parallel loops based upon a hybrid (static + dynamic) scheduling strategy. Our support balances the processor workloads by dynamically migrating computations from slower to faster processors without requiring a priori knowledge of their relative speeds. The results obtained on many experiments conducted on SGI/Cray T3E and IBM SP2 systems show that our hybrid scheduling strategy which first allocates the sa..

Scheduling Data-Parallel Computations on Heterogeneous and Time-Shared Environments

This paper addresses the problem of load balancing data-parallel computations on heterogeneous and time-shared parallel computing environments, where load imbalance may be introduced by the different capacities of processors populating a computer, or by the sharing of the same computational resources,among several users. To solve this problem we propose a run-time support for parallel loops Based upon a hybrid (static + dynamic) scheduling strategy. The main features of our technique are the absence of centralization and synchronization points, the prefetching of work towards slower processors, and the overlapping of communication latencies with useful computation

Scheduling Data-Parallel Computations on Heterogeneous and Time-Shared Environments

This paper addresses the problem of load balancing data--parallel computations on heterogeneous and time-shared parallel computing environments. Load imbalance in these environments may be introduced by the different capacities of processors populating a computer, or by the sharing of the same computational resources among several users. A evenly partitioned code, which on a homogeneous system runs with a perfect load balance, may perform poorly if the processors have different speeds or loads. To solve this problem for data--parallel computations, we propose a run--time support for parallel loops based upon a hybrid (static + dynamic) scheduling strategy. Our support balances the processor workloads by dynamically migrating computations from slower to faster processors without requiring a priori knowledge of their relative speeds. The results obtained on many experiments conducted on SGI/Cray T3E and IBM SP2 systems show that our hybrid scheduling strategy which first allocates the sa..