Supporting biodiversity studies with the EUBrazilOpenBio Hybrid Data Infrastructure

[EN] EUBrazilOpenBio is a collaborative initiative addressing strategic barriers in biodiversity research by integrating open access data and user-friendly tools widely available in Brazil and Europe. The project deploys the EU-Brazil Hybrid Data Infrastructure that allows the sharing of hardware, software and data on-demand. This infrastructure provides access to several integrated services and resources to seamlessly aggregate taxonomic, biodiversity and climate data, used by processing services implementing checklist cross-mapping and ecological niche modelling. A Virtual Research Environment was created to provide users with a single entry point to processing and data resources. This article describes the architecture, demonstration use cases and some experimental results and validation.EUBrazilOpenBio - Open Data and Cloud Computing e-Infrastructure for Biodiversity (2011-2013) is a Small or medium-scale focused research project (STREP) funded by the European Commission under the Cooperation Programme, Framework Programme Seven (FP7) Objective FP7-ICT-2011- EU-Brazil Research and Development cooperation, and the National Council for Scientific and Technological Development of Brazil (CNPq) of the Brazilian Ministry of Science, Technology and Innovation (MCTI) under the corresponding matching Brazilian Call for proposals MCT/CNPq 066/2010. BSC authors also acknowledge the support of the grant SEV-2011-00067 of Severo Ochoa Program, awarded by the Spanish Government and the Spanish Ministry of Science and Innovation under contract TIN2012-34557 and the Generalitat de Catalunya (contract 2009-SGR-980).Amaral, R.; Badia, RM.; Blanquer Espert, I.; Braga-Neto, R.; Candela, L.; Castelli, D.; Flann, C.... (2015). Supporting biodiversity studies with the EUBrazilOpenBio Hybrid Data Infrastructure. Concurrency and Computation: Practice and Experience. 27(2):376-394. https://doi.org/10.1002/cpe.3238S376394272EUBrazilOpenBio Consortium EU-Brazil Open Data and Cloud Computing e-Infrastructure for Biodiversity http://www.eubrazilopenbio.eu/Triebel, D., Hagedorn, G., & Rambold, G. (2012). An appraisal of megascience platforms for biodiversity information. MycoKeys, 5, 45-63. doi:10.3897/mycokeys.5.4302Edwards, J. L. (2000). Interoperability of Biodiversity Databases: Biodiversity Information on Every Desktop. Science, 289(5488), 2312-2314. doi:10.1126/science.289.5488.2312Grassle, F. (2000). The Ocean Biogeographic Information System (OBIS): An On-line, Worldwide Atlas for Accessing, Modeling and Mapping Marine Biological Data in a Multidimensional Geographic Context. Oceanography, 13(3), 5-7. doi:10.5670/oceanog.2000.01Constable, H., Guralnick, R., Wieczorek, J., Spencer, C., & Peterson, A. T. (2010). VertNet: A New Model for Biodiversity Data Sharing. PLoS Biology, 8(2), e1000309. doi:10.1371/journal.pbio.1000309Roskov Y Kunze T Paglinawan L Orrell T Nicolson D Culham A Bailly N Kirk P Bourgoin T Baillargeon G Hernandez F De Wever A Species 2000 & ITIS Catalogue of Life 2013 www.catalogueoflife.org/col/speciesLink Consortium speciesLink http://splink.cria.org.br 2013 http://splink.cria.org.brList of Species of the Brazilian Flora Consortium List of Species of the Brazilian Flora http://floradobrasil.jbrj.gov.br/ 2013 http://floradobrasil.jbrj.gov.br/Wieczorek, J., Bloom, D., Guralnick, R., Blum, S., Döring, M., Giovanni, R., … Vieglais, D. (2012). Darwin Core: An Evolving Community-Developed Biodiversity Data Standard. PLoS ONE, 7(1), e29715. doi:10.1371/journal.pone.0029715De Giovanni R Copp C Döring M Güntscg A Vieglais D Hobern D Torre J Wieczorek J Gales R Hyam R Blum S Perry S TAPIR - TDWG Access Protocol for Information Retrieval http://www.tdwg.org/activities/abcd/Jetz, W., McPherson, J. M., & Guralnick, R. P. (2012). Integrating biodiversity distribution knowledge: toward a global map of life. Trends in Ecology & Evolution, 27(3), 151-159. doi:10.1016/j.tree.2011.09.007NICE Srl Enginframe 2013 http://www.nice-software.com/products/enginframeHiden, H., Woodman, S., Watson, P., & Cala, J. (2013). Developing cloud applications using the e-Science Central platform. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 371(1983), 20120085. doi:10.1098/rsta.2012.0085Glatard, T., Montagnat, J., Lingrand, D., & Pennec, X. (2008). Flexible and Efficient Workflow Deployment of Data-Intensive Applications On Grids With MOTEUR. The International Journal of High Performance Computing Applications, 22(3), 347-360. doi:10.1177/1094342008096067Kacsuk, P., & Sipos, G. (2005). Multi-Grid, Multi-User Workflows in the P-GRADE Grid Portal. Journal of Grid Computing, 3(3-4), 221-238. doi:10.1007/s10723-005-9012-6Manuali, C., Laganà, A., & Rampino, S. (2010). GriF: A Grid framework for a Web Service approach to reactive scattering. Computer Physics Communications, 181(7), 1179-1185. doi:10.1016/j.cpc.2010.03.001Goecks, J., Nekrutenko, A., Taylor, J., & Galaxy Team, T. (2010). Galaxy: a comprehensive approach for supporting accessible, reproducible, and transparent computational research in the life sciences. Genome Biology, 11(8), R86. doi:10.1186/gb-2010-11-8-r86XSEDE consortium Extreme science and engineering discovery environment 2013 https://www.xsede.org/NanoHUB.org Online simulation and more for nanotechnology http://nanohub.org/SCI-BUS consortium Scientific gateway based user support 2011 https://www.sci-bus.eu/Kacsuk, P., Farkas, Z., Kozlovszky, M., Hermann, G., Balasko, A., Karoczkai, K., & Marton, I. (2012). WS-PGRADE/gUSE Generic DCI Gateway Framework for a Large Variety of User Communities. Journal of Grid Computing, 10(4), 601-630. doi:10.1007/s10723-012-9240-5Candela L Castelli D Pagano P D4science: an e-infrastructure for supporting virtual research environments Post-Proceedings of the 5th Italian Res. Conf. on Digital Libraries - IRCDL 2009 2009 166 169Lobo, J. M., Jiménez-Valverde, A., & Hortal, J. (2010). The uncertain nature of absences and their importance in species distribution modelling. Ecography, 33(1), 103-114. doi:10.1111/j.1600-0587.2009.06039.xGrinnell, J. (1917). Field Tests of Theories Concerning Distributional Control. The American Naturalist, 51(602), 115-128. doi:10.1086/279591Peterson, A. T., Soberón, J., Pearson, R. G., Anderson, R. P., Martínez-Meyer, E., Nakamura, M., & Araújo, M. B. (2011). Ecological Niches and Geographic Distributions (MPB-49). doi:10.23943/princeton/9780691136868.001.0001Brazilian Virtual Herbarium Consortium Brazilian Virtual Herbarium 2013 http://biogeo.inct.florabrasil.net/De Souza Muñoz, M. E., De Giovanni, R., de Siqueira, M. F., Sutton, T., Brewer, P., Pereira, R. S., … Canhos, V. P. (2009). openModeller: a generic approach to species’ potential distribution modelling. GeoInformatica, 15(1), 111-135. doi:10.1007/s10707-009-0090-7Hirzel, A. H., Hausser, J., Chessel, D., & Perrin, N. (2002). ECOLOGICAL-NICHE FACTOR ANALYSIS: HOW TO COMPUTE HABITAT-SUITABILITY MAPS WITHOUT ABSENCE DATA? Ecology, 83(7), 2027-2036. doi:10.1890/0012-9658(2002)083[2027:enfaht]2.0.co;2Anderson, R. P., Lew, D., & Peterson, A. T. (2003). Evaluating predictive models of species’ distributions: criteria for selecting optimal models. Ecological Modelling, 162(3), 211-232. doi:10.1016/s0304-3800(02)00349-6Farber, O., & Kadmon, R. (2003). Assessment of alternative approaches for bioclimatic modeling with special emphasis on the Mahalanobis distance. Ecological Modelling, 160(1-2), 115-130. doi:10.1016/s0304-3800(02)00327-7Phillips, S. J., Anderson, R. P., & Schapire, R. E. (2006). Maximum entropy modeling of species geographic distributions. Ecological Modelling, 190(3-4), 231-259. doi:10.1016/j.ecolmodel.2005.03.026Schölkopf, B., Platt, J. C., Shawe-Taylor, J., Smola, A. J., & Williamson, R. C. (2001). Estimating the Support of a High-Dimensional Distribution. Neural Computation, 13(7), 1443-1471. doi:10.1162/089976601750264965Armbrust, M., Stoica, I., Zaharia, M., Fox, A., Griffith, R., Joseph, A. D., … Rabkin, A. (2010). A view of cloud computing. Communications of the ACM, 53(4), 50. doi:10.1145/1721654.1721672Foster I Zhao Y Raicu I Lu S Cloud computing and grid computing 360-degree compared Grid Computing Environments Workshop, 2008. GCE '08 2008 1 10Candela, L., Castelli, D., & Pagano, P. (2013). Virtual Research Environments: An Overview and a Research Agenda. Data Science Journal, 12(0), GRDI75-GRDI81. doi:10.2481/dsj.grdi-013Tsai W Service-oriented system engineering: a new paradigm Service-oriented system engineering, 2005. sose 2005. IEEE International Workshop 3 6 10.1109/SOSE.2005.34Cattell, R. (2011). Scalable SQL and NoSQL data stores. ACM SIGMOD Record, 39(4), 12. doi:10.1145/1978915.1978919Durão FA Assad RE Silva AF Carvalho JF Garcia VC Trinta FAM USTO.RE: A Private Cloud Storage System 13th International Conference on Web Engineering (ICWE 2013) - Industry Track 2013 452 466Lezzi, D., Rafanell, R., Carrión, A., Espert, I. B., Hernández, V., & Badia, R. M. (2012). Enabling e-Science Applications on the Cloud with COMPSs. Lecture Notes in Computer Science, 25-34. doi:10.1007/978-3-642-29737-3_4Boeres, C., & Rebello, V. E. F. (2004). EasyGrid: towards a framework for the automatic Grid enabling of legacy MPI applications. Concurrency and Computation: Practice and Experience, 16(5), 425-432. doi:10.1002/cpe.821VENUS-C consortium Deliverable 6.1 - report on architecture 2012 http://www.venus-c.eu/Content/Publications.aspx?id=bfac02a9-9bc0-4c8f-80e0-7ceddc5c893bThain, D., Tannenbaum, T., & Livny, M. (2005). Distributed computing in practice: the Condor experience. Concurrency and Computation: Practice and Experience, 17(2-4), 323-356. doi:10.1002/cpe.938Couvares, P., Kosar, T., Roy, A., Weber, J., & Wenger, K. (s. f.). Workflow Management in Condor. Workflows for e-Science, 357-375. doi:10.1007/978-1-84628-757-2_22Sena, A., Nascimento, A., Boeres, C., & Rebello, V. (2008). EasyGrid Enabling of Iterative Tightly-Coupled Parallel MPI Applications. 2008 IEEE International Symposium on Parallel and Distributed Processing with Applications. doi:10.1109/ispa.2008.122Edmonds, A., Metsch, T., & Papaspyrou, A. (2011). Open Cloud Computing Interface in Data Management-Related Setups. Grid and Cloud Database Management, 23-48. doi:10.1007/978-3-642-20045-8_2Livenson, I., & Laure, E. (2011). Towards transparent integration of heterogeneous cloud storage platforms. Proceedings of the fourth international workshop on Data-intensive distributed computing - DIDC ’11. doi:10.1145/1996014.1996020The rOCCI framework http://occi-wg.org/2012/04/02/rocci-a-ruby-occi-framework/Mendelsohn N Gudgin M Ruellan H Nottingham M SOAP message transmission optimization mechanism W3C Recommendation 2005 http://www.w3.org/TR/2005/REC-soap12-mtom-20050125/Sencha Sencha GXT application framework for Google web toolkit 2013 http://www.sencha.com/products/gxt/Vicario, S., Hardisty, A., & Haitas, N. (2011). BioVeL: Biodiversity Virtual e-Laboratory. EMBnet.journal, 17(2), 5. doi:10.14806/ej.17.2.238Lezzi D Rafanell R Torres E De Giovanni R Blanquer I Badia RM Programming ecological niche modeling workflows in the cloud Proceed. of the 27th IEEE Int. Conf. on Advanced Information Networking and Applications 2013 1223 1228Lohmann, L. G. (2006). Untangling the phylogeny of neotropical lianas (Bignonieae, Bignoniaceae). American Journal of Botany, 93(2), 304-318. doi:10.3732/ajb.93.2.304Flann C Use Case Study EUBrazilOpenBio Cross-mapping tool Assessment of usability for regional-GSD comparisons 2013 http://www.eubrazilopenbio.eu/Content/Factfile.aspx?id=0750dcd8-23f2-4bf1-bad4-52aa3277d002Brazilian Ministry of Environment Instrução normativa no. 6, 23 de setembro de 2008 2008 http://www.mma.gov.br/estruturas/179/_arquivos/179_05122008033615.pd

Methodology for malleable applications on distributed memory systems

A la portada logo BSC(English) The dominant programming approach for scientific and industrial computing on clusters is MPI+X. While there are a variety of approaches within the node, denoted by the ``X'', Message Passing interface (MPI) is the standard for programming multiple nodes with distributed memory. This thesis argues that the OmpSs-2 tasking model can be extended beyond the node to naturally support distributed memory, with three benefits: First, at small to medium scale the tasking model is a simpler and more productive alternative to MPI. It eliminates the need to distribute the data explicitly and convert all dependencies into explicit message passing. It also avoids the complexity of hybrid programming using MPI+X. Second, the ability to offload parts of the computation among the nodes enables the runtime to automatically balance the loads in a full-scale MPI+X program. This approach does not require a cost model, and it is able to transparently balance the computational loads across the whole program, on all its nodes. Third, because the runtime handles all low-level aspects of data distribution and communication, it can change the resource allocation dynamically, in a way that is transparent to the application. This thesis describes the design, development and evaluation of OmpSs-2@Cluster, a programming model and runtime system that extends the OmpSs-2 model to allow a virtually unmodified OmpSs-2 program to run across multiple distributed memory nodes. For well-balanced applications it provides similar performance to MPI+OpenMP on up to 16 nodes, and it improves performance by up to 2x for irregular and unbalanced applications like Cholesky factorization. This work also extended OmpSs-2@Cluster for interoperability with MPI and Barcelona Supercomputing Center (BSC)'s state-of-the-art Dynamic Load Balance (DLB) library in order to dynamically balance MPI+OmpSs-2 applications by transparently offloading tasks among nodes. This approach reduces the execution time of a microscale solid mechanics application by 46% on 64 nodes and on a synthetic benchmark, it is within 10% of perfect load balancing on up to 8 nodes. Finally, the runtime was extended to transparently support malleability for pure OmpSs-2@Cluster programs and interoperate with the Resources Management System (RMS). The only change to the application is to explicitly call an API function to control the addition or removal of nodes. In this regard we additionally provide the runtime with the ability to semi-transparently save and recover part of the application status to perform checkpoint and restart. Such a feature hides the complexity of data redistribution and parallel IO from the user while allowing the program to recover and continue previous executions. Our work is a starting point for future research on fault tolerance. In summary, OmpSs-2@Cluster expands the OmpSs-2 programming model to encompass distributed memory clusters. It allows an existing OmpSs-2 program, with few if any changes, to run across multiple nodes. OmpSs-2@Cluster supports transparent multi-node dynamic load balancing for MPI+OmpSs-2 programs, and enables semi-transparent malleability for OmpSs-2@Cluster programs. The runtime system has a high level of stability and performance, and it opens several avenues for future work.(Español) El modelo de programación dominante para clusters tanto en ciencia como industria es actualmente MPI+X. A pesar de que hay alguna variedad de alternativas para programar dentro de un nodo (indicado por la "X"), el estandar para programar múltiples nodos con memoria distribuida sigue siendo Message Passing Interface (MPI). Esta tesis propone la extensión del modelo de programación basado en tareas OmpSs-2 para su funcionamiento en sistemas de memoria distribuida, destacando 3 beneficios principales: En primer lugar; a pequeña y mediana escala, un modelo basado en tareas es más simple y productivo que MPI y elimina la necesidad de distribuir los datos explícitamente y convertir todas las dependencias en mensajes. Además, evita la complejidad de la programacion híbrida MPI+X. En segundo lugar; la capacidad de enviar partes del cálculo entre los nodos permite a la librería balancear la carga de trabajo en programas MPI+X a gran escala. Este enfoque no necesita un modelo de coste y permite equilibrar cargas transversalmente en todo el programa y todos los nodos. En tercer lugar; teniendo en cuenta que es la librería quien maneja todos los aspectos relacionados con distribución y transferencia de datos, es posible la modificación dinámica y transparente de los recursos que utiliza la aplicación. Esta tesis describe el diseño, desarrollo y evaluación de OmpSs-2@Cluster; un modelo de programación y librería que extiende OmpSs-2 permitiendo la ejecución de programas OmpSs-2 existentes en múltiples nodos sin prácticamente necesidad de modificarlos. Para aplicaciones balanceadas, este modelo proporciona un rendimiento similar a MPI+OpenMP hasta 16 nodos y duplica el rendimiento en aplicaciones irregulares o desbalanceadas como la factorización de Cholesky. Este trabajo incluye la extensión de OmpSs-2@Cluster para interactuar con MPI y la librería de balanceo de carga Dynamic Load Balancing (DLB) desarrollada en el Barcelona Supercomputing Center (BSC). De este modo es posible equilibrar aplicaciones MPI+OmpSs-2 mediante la transferencia transparente de tareas entre nodos. Este enfoque reduce el tiempo de ejecución de una aplicación de mecánica de sólidos a micro-escala en un 46% en 64 nodos; en algunos experimentos hasta 8 nodos se pudo equilibrar perfectamente la carga con una diferencia inferior al 10% del equilibrio perfecto. Finalmente, se implementó otra extensión de la librería para realizar operaciones de maleabilidad en programas OmpSs-2@Cluster e interactuar con el Sistema de Manejo de Recursos (RMS). El único cambio requerido en la aplicación es la llamada explicita a una función de la interfaz que controla la adición o eliminación de nodos. Además, se agregó la funcionalidad de guardar y recuperar parte del estado de la aplicación de forma semitransparente con el objetivo de realizar operaciones de salva-reinicio. Dicha funcionalidad oculta al usuario la complejidad de la redistribución de datos y las operaciones de lectura-escritura en paralelo, mientras permite al programa recuperar y continuar ejecuciones previas. Este es un punto de partida para futuras investigaciones en tolerancia a fallos. En resumen, OmpSs-2@Cluster amplía el modelo de programación de OmpSs-2 para abarcar sistemas de memoria distribuida. El modelo permite la ejecución de programas OmpSs-2 en múltiples nodos prácticamente sin necesidad de modificarlos. OmpSs-2@Cluster permite además el balanceo dinámico de carga en aplicaciones híbridas MPI+OmpSs-2 ejecutadas en varios nodos y es capaz de realizar maleabilidad semi-transparente en programas OmpSs-2@Cluster puros. La librería tiene un niveles de rendimiento y estabilidad altos y abre varios caminos para trabajos futuro.Arquitectura de computador

Adaptive Resource Relocation in Virtualized Heterogeneous Clusters

Cluster computing has recently gone through an evolution from single processor systems to multicore/multi-socket systems. This has resulted in lowering the cost/performance ratio of the compute machines. Compute farms that host these machines tend to become heterogeneous over time due to incremental extensions, hardware upgrades and/or nodes being purchased for users with particular needs. This heterogeneity is not surprising given the wide range of processor, memory and network technologies that become available and the relatively small price difference between these various options. Different CPU architectures, memory capacities, communication and I/O interfaces of the participating compute nodes present many challenges to job scheduling and often result in under or over utilization of the compute resources. In general, it is not feasible for the application programmers to specifically optimize their programs for such a set of differing compute n odes, due to the difficulty and time-intensiveness of such a task. The trend of heterogeneous compute farms has coincided with resurgence in the virtualization technology. Virtualization technology is receiving widespread adoption, mainly due to the benefits of server consolidation and isolation, load balancing, security and fault tolerance. Virtualization has also generated considerable interest in the High Performance Computing (HPC) community, due to the resulting high availability, fault tolerance, cluster partitioning and accommodation of conflicting user requirements. However, the HPC community is still wary of the potential overheads associated with‘ virtualization, as it results in slower network communications and disk I/O, which need to be addressed. The live migration feature, available to most virtualization technologies, can be leveraged to improve the throughput of a heterogeneous compute farm (HC) used for HPC applications. For this we mitigated the slow network communication in Xen; an open source virtual machine monitor. We present a detailed analysis of the communication framework of Xen and propose communication configurations that give 50% improvement over the conventional Xen network configuration. From a detailed study of the migration facility in Xen, we propose an improvement in the live migration facility specifically targeting HPC applications. This optimization gives around 50% improvement over the default migration facility of Xen. In this thesis, we also investigate resource scheduling in heterogeneous compute farm with the perspective of dynamic resource re-mapping. Our approach is to profile each job in the compute farm at runtime, and propose a better resource mapping compared to the initial allocation. We then migrate the job(s) to the best-suited homogeneous sub-cluster to improve overall throughput of the HC. For this, we develop a novel heterogeneity and virtualization-aware profiling framework, which is able to predict the CPU and communication characteristics of high performance scientific applications. The prediction accuracy of our performance estimation model is over 80%. The framework implementation is lightweight, with an overhead of 3%. Our experiments show that we are able to improve the throughput of the compute farm by 25% and the time saved by the HC with our framework is over 30%. The framework can be readily extended to HCs supporting a cloud computing environment

Optimization techniques for adaptability in MPI application

The first version of MPI (Message Passing Interface) was released in 1994. At that time, scientific applications for HPC (High Performance Computing) were characterized by a static execution environment. These applications usually had regular computation and communication patterns, operated on dense data structures accessed with good data locality, and ran on homogeneous computing platforms. For these reasons, MPI has become the de facto standard for developing scientific parallel applications for HPC during the last decades. In recent years scientific applications have evolved in order to cope with several challenges posed by different fields of engineering, economics and medicine among others. These challenges include large amounts of data stored in irregular and sparse data structures with poor data locality to be processed in parallel (big data), algorithms with irregular computation and communication patterns, and heterogeneous computing platforms (grid, cloud and heterogeneous cluster). On the other hand, over the last years MPI has introduced relevant improvements and new features in order to meet the requirements of dynamic execution environments. Some of them include asynchronous non-blocking communications, collective I/O routines and the dynamic process management interface introduced in MPI 2.0. The dynamic process management interface allows the application to spawn new processes at runtime and enable communication with them. However, this feature has some technical limitations that make the implementation of malleable MPI applications still a challenge. This thesis proposes FLEX-MPI, a runtime system that extends the functionalities of the MPI standard library and features optimization techniques for adaptability of MPI applications to dynamic execution environments. These techniques can significantly improve the performance and scalability of scientific applications and the overall efficiency of the HPC system on which they run. Specifically, FLEX-MPI focuses on dynamic load balancing and performance-aware malleability for parallel applications. The main goal of the design and implementation of the adaptability techniques is to efficiently execute MPI applications on a wide range of HPC platforms ranging from small to large-scale systems. Dynamic load balancing allows FLEX-MPI to adapt the workload assignments at runtime to the performance of the computing elements that execute the parallel application. On the other hand, performance-aware malleability leverages the dynamic process management interface of MPI to change the number of processes of the application at runtime. This feature allows to improve the performance of applications that exhibit irregular computation patterns and execute in computing systems with dynamic availability of resources. One of the main features of these techniques is that they do not require user intervention nor prior knowledge of the underlying hardware. We have validated and evaluated the performance of the adaptability techniques with three parallel MPI benchmarks and different execution environments with homogeneous and heterogeneous cluster configurations. The results show that FLEXMPI significantly improves the performance of applications when running with the support of dynamic load balancing and malleability, along with a substantial enhancement of their scalability and an improvement of the overall system efficiency.La primera versión de MPI (Message Passing Interface) fue publicada en 1994, cuando la base común de las aplicaciones científicas para HPC (High Performance Computing) se caracterizaba por un entorno de ejecución estático. Dichas aplicaciones presentaban generalmente patrones regulares de cómputo y comunicaciones, accesos a estructuras de datos densas con alta localidad, y ejecución sobre plataformas de computación homogéneas. Esto ha hecho que MPI haya sido la alternativa más adecuada para la implementación de aplicaciones científicas para HPC durante más de 20 años. Sin embargo, en los últimos años las aplicaciones científicas han evolucionado para adaptarse a diferentes retos propuestos por diferentes campos de la ingeniería, la economía o la medicina entre otros. Estos nuevos retos destacan por características como grandes cantidades de datos almacenados en estructuras de datos irregulares con baja localidad para el análisis en paralelo (big data), algoritmos con patrones irregulares de cómputo y comunicaciones, e infraestructuras de computación heterogéneas (cluster heterogéneos, grid y cloud). Por otra parte, MPI ha evolucionado significativamente en cada una de sus sucesivas versiones, siendo algunas de las mejoras más destacables presentadas hasta la reciente versión 3.0 las operaciones de comunicación asíncronas no bloqueantes, rutinas de E/S colectiva, y la interfaz de procesos dinámicos presentada en MPI 2.0. Esta última proporciona un procedimiento para la creación de procesos en tiempo de ejecución de la aplicación. Sin embargo, la implementación de la interfaz de procesos dinámicos por parte de las diferentes distribuciones de MPI aún presenta numerosas limitaciones que condicionan el desarrollo de aplicaciones maleables en MPI. Esta tesis propone FLEX-MPI, un sistema que extiende las funcionalidades de la librería MPI y proporciona técnicas de optimización para la adaptación de aplicaciones MPI a entornos de ejecución dinámicos. Las técnicas integradas en FLEX-MPI permiten mejorar el rendimiento y escalabilidad de las aplicaciones científicas y la eficiencia de las plataformas sobre las que se ejecutan. Entre estas técnicas destacan el balanceo de carga dinámico y maleabilidad para aplicaciones MPI. El diseño e implementación de estas técnicas está dirigido a plataformas de cómputo HPC de pequeña a gran escala. El balanceo de carga dinámico permite a las aplicaciones adaptar de forma eficiente su carga de trabajo a las características y rendimiento de los elementos de procesamiento sobre los que se ejecutan. Por otro lado, la técnica de maleabilidad aprovecha la interfaz de procesos dinámicos de MPI para modificar el número de procesos de la aplicación en tiempo de ejecución, una funcionalidad que permite mejorar el rendimiento de aplicaciones con patrones irregulares o que se ejecutan sobre plataformas de cómputo con disponibilidad dinámica de recursos. Una de las principales características de estas técnicas es que no requieren intervención del usuario ni conocimiento previo de la arquitectura sobre la que se ejecuta la aplicación. Hemos llevado a cabo un proceso de validación y evaluación de rendimiento de las técnicas de adaptabilidad con tres diferentes aplicaciones basadas en MPI, bajo diferentes escenarios de computación homogéneos y heterogéneos. Los resultados demuestran que FLEX-MPI permite obtener un significativo incremento del rendimiento de las aplicaciones, unido a una mejora sustancial de la escalabilidad y un aumento de la eficiencia global del sistema.Programa Oficial de Doctorado en Ciencia y Tecnología InformáticaPresidente: Francisco Fernández Rivera.- Secretario: Florín Daniel Isaila.- Vocal: María Santos Pérez Hernánde