A hierarchic task-based programming model for distributed heterogeneous computing

Distributed computing platforms are evolving to heterogeneous ecosystems with Clusters, Grids and Clouds introducing in its computing nodes, processors with different core architectures, accelerators (i.e. GPUs, FPGAs), as well as different memories and storage devices in order to achieve better performance with lower energy consumption. As a consequence of this heterogeneity, programming applications for these distributed heterogeneous platforms becomes a complex task. Additionally to the complexity of developing an application for distributed platforms, developers must also deal now with the complexity of the different computing devices inside the node. In this article, we present a programming model that aims to facilitate the development and execution of applications in current and future distributed heterogeneous parallel architectures. This programming model is based on the hierarchical composition of the COMP Superscalar and Omp Superscalar programming models that allow developers to implement infrastructure-agnostic applications. The underlying runtime enables applications to adapt to the infrastructure without the need of maintaining different versions of the code. Our programming model proposal has been evaluated on real platforms, in terms of heterogeneous resource usage, performance and adaptation.This work has been supported by the European Commission through the Horizon 2020 Research and Innovation program under contract 687584 (TANGO project) by the Spanish Government under contract TIN2015-65316 and grant SEV-2015-0493 (Severo Ochoa Program) and by Generalitat de Catalunya under contracts 2014-SGR-1051 and 2014-SGR-1272.Peer ReviewedPostprint (author's final draft

Аналіз та врахування неоднорідності в системах розподільної обробки даних

В статті розглянуто відомі методи оцінки неоднорідності паралельних обчислювальних систем, зокрема, систем розподільної обробки даних (СРОД). Приведені можливі напрямки покращення кількісно-якісних оцінок неоднорідності таких систем та наведено підхід до створення виразів для оцінки кількісно-якісних характеристик рівня неоднорідності (СРОД). Бібліогр.: 7 назв.В статье рассмотрены известные методы оценки неоднородности параллельных вычислительных систем, в частности, систем распределенной обработки данных (СРОД). Приведены возможные направления улучшения количественно-качественных оценок неоднородности таких систем и подход к созданию выражений для оценки количественно-качественных характеристик уровня неоднородности СРОД. Библиогр.: 7 назв.In article known methods of an estimation of heterogeneity of parallel computing systems, in particular systems of distributed data processing (DDPS) are considered. Possible directions of improvement of quantitative - quality standards of heterogeneity of such systems are given and the approach is given in creation of expressions for an estimation of quantitative - qualitative characteristics of a level of heterogeneity DDPS. Refs.: 7 titles

FGPGA: An Efficient Genetic Approach for Producing Feasible Graph Partitions

Graph partitioning, a well studied problem of parallel computing has many applications in diversified fields such as distributed computing, social network analysis, data mining and many other domains. In this paper, we introduce FGPGA, an efficient genetic approach for producing feasible graph partitions. Our method takes into account the heterogeneity and capacity constraints of the partitions to ensure balanced partitioning. Such approach has various applications in mobile cloud computing that include feasible deployment of software applications on the more resourceful infrastructure in the cloud instead of mobile hand set. Our proposed approach is light weight and hence suitable for use in cloud architecture. We ensure feasibility of the partitions generated by not allowing over-sized partitions to be generated during the initialization and search. Our proposed method tested on standard benchmark datasets significantly outperforms the state-of-the-art methods in terms of quality of partitions and feasibility of the solutions.Comment: Accepted in the 1st International Conference on Networking Systems and Security 2015 (NSysS 2015

Taming Resource Heterogeneity In Distributed ML Training With Dynamic Batching

Current techniques and systems for distributed model training mostly assume that clusters are comprised of homogeneous servers with a constant resource availability. However, cluster heterogeneity is pervasive in computing infrastructure, and is a fundamental characteristic of low-cost transient resources (such as EC2 spot instances). In this paper, we develop a dynamic batching technique for distributed data-parallel training that adjusts the mini-batch sizes on each worker based on its resource availability and throughput. Our mini-batch controller seeks to equalize iteration times on all workers, and facilitates training on clusters comprised of servers with different amounts of CPU and GPU resources. This variable mini-batch technique uses proportional control and ideas from PID controllers to find stable mini-batch sizes. Our empirical evaluation shows that dynamic batching can reduce model training times by more than 4x on heterogeneous clusters

Integrating Biosystem Models Using Waveform Relaxation

Modelling in systems biology often involves the integration of component models into larger composite models. How to do this systematically and efficiently is a significant challenge: coupling of components can be unidirectional or bidirectional, and of variable strengths. We adapt the waveform relaxation (WR) method for parallel computation of ODEs as a general methodology for computing systems of linked submodels. Four test cases are presented: (i) a cascade of unidirectionally and bidirectionally coupled harmonic oscillators, (ii) deterministic and stochastic simulations of calcium oscillations, (iii) single cell calcium oscillations showing complex behaviour such as periodic and chaotic bursting, and (iv) a multicellular calcium model for a cell plate of hepatocytes. We conclude that WR provides a flexible means to deal with multitime-scale computation and model heterogeneity. Global solutions over time can be captured independently of the solution techniques for the individual components, which may be distributed in different computing environments

Scheduling strategies for time-sensitive distributed applications on edge computing

Edge computing is a distributed computing paradigm that shifts the computation capabilities close to the data sources. This new paradigm, coupled with the use of parallel embedded processor architectures, is becoming a very promising solution for time-sensitive distributed applications used in Internet of Things and large Cyber-Physical Systems (e.g., those used in smart cities) to alleviate the pressure on centralized solutions. However, the distribution and heterogeneity nature of the edge computing complicates the response-time analysis on these type of applications. This thesis addresses this challenge by proposing a new Directed Acyclic Graph (DAG)-task based system model to characterize: (1) the distribution nature of applications executed on the edge; and (2) the heterogeneous computation and network communication capabilities of edge computing platforms. Based on this system model, this work presents five different scheduling strategies: four sub-optimal but tractable heuristics and an optimal but costly approach based on a mixed integer linear programming (MILP), that minimize the overall response time of distributed time-sensitive applications. To address both issues, and as a proof of concept, we use COMPSs, a framework composed of a task-based programming model and a runtime used to program and efficiently distribute time-sensitive applications across the compute continuum. However, COMPSs is agnostic of time-sensitive applications, hence in this work we extend it to consider the dynamic scheduling based on the proposed scheduling strategies. Our results show that our scheduling heuristics outperform current scheduling solutions, while providing an average and upper-bound execution time comparable to the optimal one provided by the MILP allocation approach

MPICH-G2: A Grid-Enabled Implementation of the Message Passing Interface

Application development for distributed computing "Grids" can benefit from tools that variously hide or enable application-level management of critical aspects of the heterogeneous environment. As part of an investigation of these issues, we have developed MPICH-G2, a Grid-enabled implementation of the Message Passing Interface (MPI) that allows a user to run MPI programs across multiple computers, at the same or different sites, using the same commands that would be used on a parallel computer. This library extends the Argonne MPICH implementation of MPI to use services provided by the Globus Toolkit for authentication, authorization, resource allocation, executable staging, and I/O, as well as for process creation, monitoring, and control. Various performance-critical operations, including startup and collective operations, are configured to exploit network topology information. The library also exploits MPI constructs for performance management; for example, the MPI communicator construct is used for application-level discovery of, and adaptation to, both network topology and network quality-of-service mechanisms. We describe the MPICH-G2 design and implementation, present performance results, and review application experiences, including record-setting distributed simulations.Comment: 20 pages, 8 figure

A hyper-heuristic for adaptive scheduling in computational grids

In this paper we present the design and implementation of an hyper-heuristic for efficiently scheduling independent jobs in computational grids. An efficient scheduling of jobs to grid resources depends on many parameters, among others, the characteristics of the resources and jobs (such as computing capacity, consistency of computing, workload, etc.). Moreover, these characteristics change over time due to the dynamic nature of grid environment, therefore the planning of jobs to resources should be adaptively done. Existing ad hoc scheduling methods (batch and immediate mode) have shown their efficacy for certain types of resource and job characteristics. However, as stand alone methods, they are not able to produce the best planning of jobs to resources for different types of Grid resources and job characteristics. In this work we have designed and implemented a hyper-heuristic that uses a set of ad hoc (immediate and batch mode) scheduling methods to provide the scheduling of jobs to Grid resources according to the Grid and job characteristics. The hyper-heuristic is a high level algorithm, which examines the state and characteristics of the Grid system (jobs and resources), and selects and applies the ad hoc method that yields the best planning of jobs. The resulting hyper-heuristic based scheduler can be thus used to develop network-aware applications that need efficient planning of jobs to resources. The hyper-heuristic has been tested and evaluated in a dynamic setting through a prototype of a Grid simulator. The experimental evaluation showed the usefulness of the hyper-heuristic for planning of jobs to resources as compared to planning without knowledge of the resource and job characteristics.Peer ReviewedPostprint (author's final draft

3E: Energy-Efficient Elastic Scheduling for Independent Tasks in Heterogeneous Computing Systems

Reducing energy consumption is a major design constraint for modern heterogeneous computing systems to minimize electricity cost, improve system reliability and protect environment. Conventional energy-efficient scheduling strategies developed on these systems do not sufficiently exploit the system elasticity and adaptability for maximum energy savings, and do not simultaneously take account of user expected finish time. In this paper, we develop a novel scheduling strategy named energy-efficient elastic (3E) scheduling for aperiodic, independent and non-real-time tasks with user expected finish times on DVFS-enabled heterogeneous computing systems. The 3E strategy adjusts processors’ supply voltages and frequencies according to the system workload, and makes trade-offs between energy consumption and user expected finish times. Compared with other energy-efficient strategies, 3E significantly improves the scheduling quality and effectively enhances the system elasticity