Application Monitoring in the Grid with GRM and PROVE

Abstract. GRM and PROVE were originally designed and implemented as part of the P-GRADE graphical parallel program development environment running on clusters. In the framework of the biggest European Grid project, the DataGrid project we investigated the possibility of transforming GRM and PROVE to a Grid monitoring infrastructure. This paper presents the results of this work showing how to separate GRM and PROVE from the P-GRADE system and to turn them into standalone Grid monitoring tools

Számítóháló alkalmazások teljesítményanalízise és optimalizációja = Performance analysis and optimisation of grid applications

Számítóhálón (griden) futó alkalmazások, elsősorban workflow-k hatékony végrehajtására kerestünk újszerű megoldásokat a grid teljesítményanalízis és optimalizáció területén. Elkészítettük a Mercury monitort a grid teljesítményanalízis követelményeit figyelembe véve. A párhuzamos programok monitorozására alkalmas GRM monitort integráltuk a relációs adatmodell alapú R-GMA grid információs rendszerrel, illetve a Mercury monitorral. Elkészült a Pulse, és a Prove vizualizációs eszköz grid teljesítményanalízist támogató verziója. Elkészítettünk egy state-of-the-art felmérést grid teljesítményanalízis eszközökről. Kidolgoztuk a P-GRADE rendszer workflow absztrakciós rétegét, melyhez kapcsolódóan elkészült a P-GRADE portál. Ennek segítségével a felhasználók egy web böngészőn keresztül szerkeszthetnek és hajthatnak végre workflow alkalmazásokat számítóhálón. A portál különböző számítóháló implementációkat támogat. Lehetőséget biztosít információ gyűjtésére teljesítményanalízis céljából. Megvizsgáltuk a portál erőforrás brókerekkel való együttműködését, felkészítettük a portált a sikertelen futások javítására. A végrehajtás optimalizálása megkövetelheti az alkalmazás egyes részeinek áthelyezését más erőforrásokra. Ennek támogatására továbbfejlesztettük a P-GRADE alkalmazások naplózhatóságát, és illesztettük a Condor feladatütemezőjéhez. Sikeresen kapcsoltunk a rendszerhez egy terhelés elosztó modult, mely képes a terheltségétől függően áthelyezni a folyamatokat. | We investigated novel approaches for performance analysis and optimization for efficient execution of grid applications, especially workflows. We took into consideration the special requirements of grid performance analysis when elaborated Mercury, a grid monitoring infrastructure. GRM, a performance monitor for parallel applications, has been integrated with R-GMA, a relational grid information system and Mercury as well. We developed Pulse and Prove visualisation tools for supporting grid performance analysis. We wrote a comprehensive state-of-the art survey of grid performance tools. We designed a novel abstraction layer of P-GRADE supporting workflows, and a grid portal. Users can draft and execute workflow applications in the grid via a web browser using the portal. The portal supports multiple grid implementations and provides monitoring capabilities for performance analysis. We tested the integration of the portal with grid resource brokers and also augmented it with some degree of fault-tolerance. Optimization may require the migration of parts of the application to different resources and thus, it requires support for checkpointing. We enhanced the checkpointing facilities of P-GRADE and coupled it to Condor job scheduler. We also extended the system with a load balancer module that is able to migrate processes as part of the optimization

Resource and Application Models for Advanced Grid Schedulers

As Grid computing is becoming an inevitable future, managing, scheduling and monitoring dynamic, heterogeneous resources will present new challenges. Solutions will have to be agile and adaptive, support self-organization and autonomous management, while maintaining optimal resource utilisation. Presented in this paper are basic principles and architectural concepts for efficient resource allocation in heterogeneous Grid environment

Probabilistic grid scheduling based on job statistics and monitoring information

This transfer thesis presents a novel, probabilistic approach to scheduling applications on computational Grids based on their historical behaviour, current state of the Grid and predictions of the future execution times and resource utilisation of such applications. The work lays a foundation for enabling a more intuitive, user-friendly and effective scheduling technique termed deadline scheduling. Initial work has established motivation and requirements for a more efficient Grid scheduler, able to adaptively handle dynamic nature of the Grid resources and submitted workload. Preliminary scheduler research identified the need for a detailed monitoring of Grid resources on the process level, and for a tool to simulate non-deterministic behaviour and statistical properties of Grid applications. A simulation tool, GridLoader, has been developed to enable modelling of application loads similar to a number of typical Grid applications. GridLoader is able to simulate CPU utilisation, memory allocation and network transfers according to limits set through command line parameters or a configuration file. Its specific strength is in achieving set resource utilisation targets in a probabilistic manner, thus creating a dynamic environment, suitable for testing the scheduler’s adaptability and its prediction algorithm. To enable highly granular monitoring of Grid applications, a monitoring framework based on the Ganglia Toolkit was developed and tested. The suite is able to collect resource usage information of individual Grid applications, integrate it into standard XML based information flow, provide visualisation through a Web portal, and export data into a format suitable for off-line analysis. The thesis also presents initial investigation of the utilisation of University College London Central Computing Cluster facility running Sun Grid Engine middleware. Feasibility of basic prediction concepts based on the historical information and process meta-data have been successfully established and possible scheduling improvements using such predictions identified. The thesis is structured as follows: Section 1 introduces Grid computing and its major concepts; Section 2 presents open research issues and specific focus of the author’s research; Section 3 gives a survey of the related literature, schedulers, monitoring tools and simulation packages; Section 4 presents the platform for author’s work – the Self-Organising Grid Resource management project; Sections 5 and 6 give detailed accounts of the monitoring framework and simulation tool developed; Section 7 presents the initial data analysis while Section 8.4 concludes the thesis with appendices and references

Development of Algorithms on the Grid

A parameter study involves analyzing situations myriad times with varying parameters and is both computationally- and data-intensive; yet there is a great need for these studies in many areas of science and engineering. In response to this need a module for the P-GRADE Portal was developed at MTA-SZTAKI in Budapest, Hungary. This allows researchers in all areas of science to perform and visualize these studies with ease as parallel applications on the P-GRADE Portal

GRID Portal Application Visualization

Parameter studies are useful applications for researchers; however, these programs, although helpful, tend to be computationally expensive and due to their long execution time become tedious to execute. In this project we explored a method of implementing a parameter study module for the P-GRADE Portal at MTA-SZTAKI; Budapest, Hungary, an existing parallel application that allows users to create and execute a parallel program in an efficient manner without knowledge of MPI or PVM programming

Autonomous grid scheduling using probabilistic job runtime scheduling

Computational Grids are evolving into a global, service-oriented architecture – a universal platform for delivering future computational services to a range of applications of varying complexity and resource requirements. The thesis focuses on developing a new scheduling model for general-purpose, utility clusters based on the concept of user requested job completion deadlines. In such a system, a user would be able to request each job to finish by a certain deadline, and possibly to a certain monetary cost. Implementing deadline scheduling is dependent on the ability to predict the execution time of each queued job, and on an adaptive scheduling algorithm able to use those predictions to maximise deadline adherence. The thesis proposes novel solutions to these two problems and documents their implementation in a largely autonomous and self-managing way. The starting point of the work is an extensive analysis of a representative Grid workload revealing consistent workflow patterns, usage cycles and correlations between the execution times of jobs and its properties commonly collected by the Grid middleware for accounting purposes. An automated approach is proposed to identify these dependencies and use them to partition the highly variable workload into subsets of more consistent and predictable behaviour. A range of time-series forecasting models, applied in this context for the first time, were used to model the job execution times as a function of their historical behaviour and associated properties. Based on the resulting predictions of job runtimes a novel scheduling algorithm is able to estimate the latest job start time necessary to meet the requested deadline and sort the queue accordingly to minimise the amount of deadline overrun. The testing of the proposed approach was done using the actual job trace collected from a production Grid facility. The best performing execution time predictor (the auto-regressive moving average method) coupled to workload partitioning based on three simultaneous job properties returned the median absolute percentage error centroid of only 4.75%. This level of prediction accuracy enabled the proposed deadline scheduling method to reduce the average deadline overrun time ten-fold compared to the benchmark batch scheduler. Overall, the thesis demonstrates that deadline scheduling of computational jobs on the Grid is achievable using statistical forecasting of job execution times based on historical information. The proposed approach is easily implementable, substantially self-managing and better matched to the human workflow making it well suited for implementation in the utility Grids of the future

Asian Infrastructure Investment Bank (AIIB)'s sustainable safeguard mechanism on energy projects

Asian Infrastructure Investment Bank (AIIB) was officially established by China in 2016 with one of the aims to develop energy infrastructure based on green and sustainable principles in Asia. To achieve that, AIIB has set forth safeguard policies, including the Environmental and Social Framework (ESF) applied for funded projects and the AIIB Energy Sector Strategy, to guide its energy investments. However, the effects of the safeguard policies and further safeguard operations on the energy projects remain unknown. This study reviews AIIB's safeguard mechanism on energy projects, including the safeguard policies, assessment and management plans, and implementations of AIIB's energy projects. We find that AIIB's current safeguard mechanism on energy projects, in comparison with other established multilateral development banks (MDBs), is still at its beginning stage, which does not match its goal and promise on sustainable energy development in Asia

Science opportunities from the Topex/Poseidon mission

The U.S. National Aeronautics and Space Administration (NASA) and the French Centre National d'Etudes Spatiales (CNES) propose to conduct a Topex/Poseidon Mission for studying the global ocean circulation from space. The mission will use the techniques of satellite altimetry to make precise and accurate measurements of sea level for several years. The measurements will then be used by Principal Investigators (selected by NASA and CNES) and by the wider oceanographic community working closely with large international programs for observing the Earth, on studies leading to an improved understanding of global ocean dynamics and the interaction of the ocean with other processes influencing life on Earth. The major elements of the mission include a satellite carrrying an altimetric system for measuring the height of the satellite above the sea surface; a precision orbit determination system for referring the altimetric measurements to geodetic coordinates; a data analysis and distribution system for processing the satellite data, verifying their accuracy, and making them available to the scientific community; and a principal investigator program for scientific studies based on the satellite observations. This document describes the satellite, its sensors, its orbit, the data analysis system, and plans for verifying and distributing the data. It then discusses the expected accuracy of the satellite's measurements and their usefulness to oceanographic, geophysical, and other scientific studies. Finally, it outlines the relationship of the Topex/Poseidon mission to other large programs, including the World Climate Research Program, the U.S. Navy's Remote Ocean Sensing System satellite program and the European Space Agency's ERS-1 satellite program