Parallel I/O scheduling in the presence of data duplication on multiprogrammed cluster computing systems

The widespread adoption of cluster computing as a high performance computing platform has seen the growth of data intensive scientific, engineering and commercial applications such as digital libraries, climate modeling, computational chemistry, computational fluid dynamics and image repositories. However, I/O subsystem performance has not been keeping pace with processor and memory performance, and is fast becoming the dominant factor in overall system performance.&nbsp; Thus, parallel I/O has become a necessity in the face of performance improvements in other areas of computing systems. This paper addresses the problem of parallel I/O scheduling on cluster computing systems in the presence of data replication.&nbsp; We propose two new I/O scheduling algorithms and evaluate the relative performance of the proposed policies against two existing approaches.&nbsp; Simulation results show that the proposed policies perform substantially better than the baseline policies.<br /

Improving Parallel I/O Performance Using Interval I/O

Today\u27s most advanced scientific applications run on large clusters consisting of hundreds of thousands of processing cores, access state of the art parallel file systems that allow files to be distributed across hundreds of storage targets, and utilize advanced interconnections systems that allow for theoretical I/O bandwidth of hundreds of gigabytes per second. Despite these advanced technologies, these applications often fail to obtain a reasonable proportion of available I/O bandwidth. The reasons for the poor performance of application I/O include the noncontiguous I/O access patterns used for scientific computing, contention due to false sharing, and the somewhat finicky nature of parallel file system performance. We argue that a more fundamental cause of this problem is the legacy view of a file as a linear sequence of bytes. To address these issues, we introduce a novel approach for parallel I/O called Interval I/O. Interval I/O is an innovative approach that uses application access patterns to partition a file into a series of intervals, which are used as the fundamental unit for subsequent I/O operations. Use of this approach provides superior performance for the noncontiguous access patterns which are frequently used by scientific applications. In addition, the approach reduces false contention and the unnecessary serialization it causes. Interval I/O also significantly increases the performance of atomic mode operations. Finally, the Interval I/O approach includes a technique for supporting parallel I/O for cooperating applications. We provide a prototype implementation of our Interval I/O system and use it to demonstrate performance improvements of as much as 1000% compared to ROMIO when using Interval I/O with several common benchmarks

An evaluation of Java's I/O capabilities for high-performance computing

Java is quickly becoming the preferred language for writing distributed applications because of its inherent support for programming on distributed platforms. In particular, Java provides compile-time and run-time security, automatic garbage collection, inherent support for multithreading, support for persistent objects and object migration, and portability. Given these significant advantages of Java, there is a growing interest in using Java for high-performance computing applications. To be successful in the high-performance computing domain, however, Java must have the capability to efficiently handle the significant I/O requirements commonly found in high-performance computing applications. While there has been significant research in high-performance I/O using languages such as C, C++, and Fortran, there has been relatively little research into the I/O capabilities of Java. In this paper, we evaluate the I/O capabilities of Java for high-performance computing. We examine several approaches that attempt to provide high-performance I/O--many of which are not obvious at first glance--and investigate their performance in both parallel and multithreaded environments. We also provide suggestions for expanding the I/O capabilities of Java to better support the needs of high-performance computing applications

Migración y evaluación de un sistema de E/S paralela para plataformas Windows

La finalidad de este Proyecto Fin de Carrera será el desarrollo de un sistema que permita de una manera sencilla y con las menores restricciones posibles, explotar al máximo la E/S paralela para los supercomputadores basados en el sistema operativo Windows®, mediante la utilización del estándar MPI y de la interfaz de E/S paralela ROMIO. Los objetivos que se han fijado para la realización del presente proyecto son: - Estudio de los sistemas de ficheros paralelos existentes para las plataformas Windows®. Para ello será necesario indagar sobre las posibilidades, tanto comerciales como de software libre disponibles. - Explorar las posibilidades de Windows® como sistema HPC (High Performance Computing). Este estudio se centrará en las capacidades de E/S, no en las de computación. - Estudiar y analizar el sistema de E/S paralela AHPIOS y conocer todas sus posibilidades de funcionamiento. - Migrar la solución comentada en el punto anterior, desarrollado en Linux, a un entorno Windows®. - Evaluar el sistema y averiguar cuál es la configuración más recomendable.Ingeniería en Informátic

Diseño e implementación de un sistema de ficheros en MPI sobre fuse

En la actualidad la mayoría de los supercomputadores pertenecientes al Top500 utilizan sistemas de ficheros como PVFS, GPFS o Lustre. Estos sistemas de ficheros son usados mayoritariamente en aplicaciones de procesamiento de datos y visualización, intensas en datos. Sin embargo, estos sistemas presentan numerosas complicaciones a la hora de instalar y administrar. Este Proyecto Fin de Carrera presenta el diseño, implementación y evaluación de un sistema de ficheros basado en AHPIOS y FUSE. AHPIOS es el primer sistema de E/S salida paralela implementado en MPI. FUSE ofrece a los usuarios la posibilidad de desarrollar nuevos sistemas de ficheros basados en POSIX. En este trabajo se muestra como partiendo de un sistema de E/S paralela basado en MPI, se puede ofrecer una interfaz de fichero basada en POSIX. Para este fin, se ha usado la herramienta FUSE. En el capítulo de evaluación se demuestra que el prototipo implementando obtiene buenos resultados en la mayoría de los casos. Para comprobar el rendimiento del sistema propuesto se han usado distintas herramientas, basadas en aplicaciones usadas frecuentemente por la comunidad científica. ________________________________________________________________Recent advances in storage technologies and high performance interconnects have made possible in the last years to build, more and more potent storage systems that serve thousands of nodes. The majority of storage systems of clusters and super-computers from Top 500 list are managed by one of three scalable parallel file systems: GPFS, PVFS, and Lustre. The following Master Thesis presents the design, implementation, and evaluation of a parallel file system based on AHPIOS file system and FUSE. AHPIOS is the first I/O system implemented in MPI. FUSE offers users the possibility to develop new file systems based on the POSIX interface. In this dissertation we show how starting from an existing MPI-based parallel I/O system, we are able to provide a POSIX-based interface through the FUSE tool-kit. The evaluation section shows that our implemented prototype outperforms the rest of solutions in most of the cases. In order to perform the experimental evaluation, we have been used different well-known benchmarks, which are used by the scientific community.Ingeniería en Informátic