Parallel job support in the Spanish NGI

Trabajo presentado a la 4th Iberian Grid Infrastructure Conference celebrada en Braga (Portugal) del 24 al 27 de mayo de 2010.-- Los archivos adjuntos corresponden al texto completo de la comunicación de congreso y a la presentación.Execution of parallel applications on a Grid environment is a challenging problem that requires the cooperation of several middleware tools and services. Although current grid middleware stacks have a basic level of support for such applications, this support is very limited. Users face a very heterogeneous environment lacking any standards for managing their jobs. We present a complete framework for the execution of parallel applications in grid environments that hides the underlying complexity with a simple interface to run the applications. The framework is composed of a unified interface layer for starting parallel applications and a grid scheduler that provides transparent and reliable support for such types of applications. The framework is focused on MPI applications, but extensible to other types of parallel applications.The authors acknoledge support of the European Comission FP7 program, under contract number 211804 through the project EUFORIA (http://www.euforia-project.eu)Peer reviewe

Support to MPI Applications on the Grid

The current middleware stacks provide varying support for the Message Passing Interface (MPI) programming paradigm. Users face a complex and heterogeneous environment where too many low level details have to be specified to execute even the simplest parallel jobs. MPI-Start is a tool that provides an interoperable MPI execution framework across the different middleware implementations to abstract the user interfaces from the underlying middleware and to allow users to execute parallel applications in a uniform way, thus bridging the gap between HPC and HTC. In this work we present the latest developments in MPI-Start and how it can be integrated in the different middleware stacks available as part of EMI, providing a unified user experience for MPI jobs

FedCloud in EGI-ENGAGE

Trabajo presentado al Spanish JRU EGI-ENGAGE meeting celebrado en Madrid el 23 de febrero de 2015.EGI-InSPIRE RI-261323. EGI-Engage is co-funded by the Horizon 2020 Framework Programme of the European Union under grant number 654142.N

A unified user experience for MPI jobs in EMI

Trabajo presentado al European Grid Infrastructure User Forum celebrado en Lituania del 11 al 14 de abril de 2011.EMI is partially funded by the European Commission under Grant Agreement INFSO-RI-261611Peer reviewe

Scheduling for interactive and parallel applications on grids

Descripció del recurs: el 23 de febrer de 2010La computación grid constituye uno de los campos más prometedores de los sistemas informáticos. La próxima generación de aplicaciones científicas se beneficiará de una infraestructura de gran escala y multi organizacional que ofrece más potencia de cómputo de la que puede ofrecer cualquier institicuón de forma individual. Los sistemas grid necesitan planificadores de alto nivel que gestionen de forma adecuada los recursos distribuídos en varias organizaciones. Estos sistemas de gestión de recursos grid deben tomar decisiones de planificación sin realmente poseer los recursos y sin tener control total sobre las aplicaciones que en dichos recursos se ejecutan, introduciendo nuevos desafíos a la hora de realizar la planificación de aplicaciones. Aunque los sistemas grids consisten de muchos recursos y las aplicaciones enviados a estos sistemas pueden aprovechar estos recursos usándolos de forma coordinada, la mayoría de los sistemas de gestión de recursos se han centrado en la ejecución de aplicaciones secuenciales, convirtiendo el grid en un gran sistema multi-sitio donde las aplicaciones se ejecutan de forma batch. Sin embargo, en esta tesis nos hemos centrado en un tipo de aplicaciones que ha recibido poca atención hasta el momento: paralelas e interactivas. Las aplicaciones interactivas requieren la posibilidad de iniciar en un futuro inmediato su ejecución. Además, durante su ejecución es necesario proveer de mecanismos que establezcan un canal de comunicación entre el usuario y la aplicación. En el caso de las aplicaciones paralelas, es necesaria la co-asignación, es decir garantizar la disponibilidad simultánea de los recursos cuando la aplicación necesite usarlos. En este trabajo proponemos una nueva arquitectura para la ejecución de estos tipos de aplicaciones y una implementación de la misma: el gestor de recursos CrossBroker. Esta arquitectura incluye mecanismos que permiten la co-asignación de aplicaciones paralelos y la interacción de los usuarios con las aplicaciones en ejecución. Adicionalmente, con la introducción de un mecanismo de multiprogramación, proporcionamos un inicio de aplicaciones rápido incluso en escenarios de alta ocupación de los recursos.Grid computing constitutes one of the most promising fields in computer systems. The next generation of scientific applications can profit from a large-scale, multi-organizational infrastructure that offers more computing power than one institution alone is able to afford. Grids need high-level schedulers that can be used to manage the resources spanning different organizations. These Grid Resource Management Systems (GRMS) have to make scheduling decisions without actually owning the grid resources, or having full control over the jobs that are running there. This introduces new challenges in the scheduling process done by the GRMSs. Although grids consist of many resources, and jobs submitted to grid may benefit from using them in a coordinated way, most of the Grid Resource Management Systems have focused on the execution of sequential jobs, with the grid being a large multi-site environment where jobs run in a batch-like way. However, in this work we concentrate on a kind of jobs that have received little attention to date: interactive and parallel jobs. Interactive jobs require the possibility of starting in the immediate future and need mechanisms to establish a communication channel with the user. Parallel applications introduce the need for co-allocation, guaranteeing the simultaneous availability of the resources when they are accessed by the applications. We address the challenges of executing such jobs with a new architecture for a GRMS and an implementation of that architecture called the CrossBroker. Our architecture includes mechanisms to allow the co-allocation of parallel jobs and the interaction of users with running applications. Additionally with the introduction of a multi-programming mechanism, a fast startup of jobs even in high occupancy scenarios is provided

An efficient cloud scheduler design supporting preemptible instances

Maximizing resource utilization by performing an efficient resource provisioning is a key factor for any cloud provider: commercial actors can maximize their revenues, whereas scientific and non-commercial providers can maximize their infrastructure utilization. Traditionally, batch systems have allowed data centers to fill their resources as much as possible by using backfilling and similar techniques. However, in an IaaS cloud, where virtual machines are supposed to live indefinitely, or at least as long as the user is able to pay for them, these policies are not easily implementable. In this work we present a new scheduling algorithm for IaaS providers that is able to support preemptible instances, that can be stopped by higher priority requests without introducing large modifications in the current cloud schedulers. This scheduler enables the implementation of new cloud usage and payment models that allow more efficient usage of the resources and potential new revenue sources for commercial providers. We also study the correctness and the performace overhead of the proposed scheduler agains existing solutions

ooi: OpenStack OCCI interface

In this document we present an implementation of the Open Grid Forum’s Open Cloud Computing Interface (OCCI) for OpenStack, namely ooi (Openstack occi interface, 2015). OCCI is an open standard for management tasks over cloud resources, focused on interoperability, portability and integration. ooi aims to implement this open interface for the OpenStack cloud middleware, promoting interoperability with other OCCI-enabled cloud management frameworks and infrastructures. ooi focuses on being non-invasive with a vanilla OpenStack installation, not tied to a particular OpenStack release version

Scientific workflow orchestration interoperating HTC and HPC resources

8 páginas, 7 figuras.-- El Pdf del artículo es la versión pre-print.In this work we describe our developments towards the provision of a unified access method to different types of computing infrastructures at the interop- eration level. For that, we have developed a middleware suite which bridges not interoperable middleware stacks used for building distributed computing infrastructues, UNICORE and gLite. Our solution allows to transparently access and operate on HPC and HTC resources from a single interface. Using Kepler as workflow manager, we provide users with the needed integration of codes to create scientific workflows accessing both types of infrastructures.Peer reviewe

Scientific workflow orchestration interoperating HTC and HPC resources

8 páginas, 7 figuras.-- El Pdf del artículo es la versión pre-print.In this work we describe our developments towards the provision of a unified access method to different types of computing infrastructures at the interop- eration level. For that, we have developed a middleware suite which bridges not interoperable middleware stacks used for building distributed computing infrastructues, UNICORE and gLite. Our solution allows to transparently access and operate on HPC and HTC resources from a single interface. Using Kepler as workflow manager, we provide users with the needed integration of codes to create scientific workflows accessing both types of infrastructures.Peer reviewe