Design and implementation of a parallel I/O runtime system for irregular applications

Many scientific applications are I/O intensive and have tremendous I/O requirements, including checkpointing, periodically writing snapshots of computations. Especially, large number of these applications exhibits irregular access patterns, where accesses to data are performed through one or more level of indirections. A typical computation science analysis cycle for these applications involves several steps: mesh generation, domain decomposition, simulation, visualization, archival of data, and adjustment of parameters. Therefore, two main focus must be considered. The first one is to store data set in a canonical form so that other steps can use it easily without having to reorganize. The second one is that, for a restart of computation with different number of processors, data set should be stored independent of number of processors that produced it. In this dissertation, we present the design, implementation and evaluation of two parallel I/O runtime systems based on collective I/O techniques for irregular applications. The design is motivated by the requirements of a large number of science and engineering applications including teraflops, applications. The first library has been implemented on top of parallel file systems on MPPs. The user application links to the library\u27s client API that issues I/O requests using the I/O commands supported by the parallel file systems. In this library, we designed and implemented two kinds of collective I/O schemes; Collective I/O and Pipelined Collective I/O . In the Collective I/O , all processors participate in the I/O simultaneously, and in the Pipelined Collective I/O , I/O is overlapped with communication by making processor groups. As an optimization, chunking and on-line compression mechanisms are included in the both collective I/O schemes. The second library has been implemented on workstation clusters, called Collective I/O Clustering . This library is based on the client-I/O server model. The I/O architecture of workstation clusters usually relies on a set of I/O servers, having local disks, and a set of diskless nodes. By using the local file system running at each node, such as UNIX, we developed a collective I/O scheme that supports irregular problems. In this library, two I/O configurations are possible. In the first configuration, all nodes have their local disk, thus the data in a client I/O buffer can go to its local disk, removing the communication between the clients and I/O servers. In the second configuration, only subset of nodes have their disk and can serve the incoming I/O requests from the clients. In this environment, the client sends the data to the appropriate I/O server, thus the communication latency between the client and I/O servers should be addressed to improve I/O performance. To optimize the communication latency, we provide a user-controllable stripe technique. Both the library and user applications can control this stripe unit. As did on MPPs, this library is incorporated with compression scheme to optimize I/O costs. In this dissertation, the performance results on large-scale parallel systems including the Intel Paragon at Caltech, and ASCI/Red Teraflops at Sandia National Labs are presented. The results for the collective I/O clustering on IBM-SP located at Argonne National Labs are also presented

Arquitectura multiagente para E/S de alto rendimiento en clusters

La E/S constituye en la actualidad uno de los principales cuellos de botella de los sistemas distribuidos de propósito general, debido al desequilibrio existente entre el tiempo de cómputo y de E/S. Una de las soluciones propuestas para este problema ha sido el uso de la E/S paralela. En esta área, se han originado un gran número de bibliotecas de E/S paralela y sistemas de ficheros paralelos. Este tipo de sistemas adolecen de algunos defectos y carencias. Muchos de ellos están concebidos para máquinas paralelas y no se integran adecuadamente en entornos distribuidos y clusters. El uso intensivo de clusters de estaciones de trabajo durante estos últimos años hace que este tipo de sistemas no sean adecuados en el escenario de computación actual. Otros sistemas, que se adaptan a este tipo de entornos, no incluyen capacidades de reconfiguración dinámica, por lo que tienen una funcionalidad limitada. Por último, la mayoría de los sistemas de E/S que utilizan diferentes optimizaciones de E/S, no ofrecen flexibilidad a las aplicaciones para hacer uso de las mismas, intentando ocultar al usuario este tipo de técnicas. No obstante, a fin de optimizar las operaciones de E/S, es importante que las aplicaciones sean capaces de describir sus patrones de acceso, interactuando con el sistema de E/S. En otro ámbito, dentro del área de los sistemas distribuidos se encuentra el paradigma de agentes, que permite dotar a las aplicaciones de un conjunto de propiedades muy adecuadas para su adaptación a entornos complejos y dinámicos. Las características de este paradigma lo hacen a priori prometedor para abordar algunos de los problemas existentes en el campo de la E/S paralela. Esta tesis propone una solución a la problemática actual de E/S a través de tres líneas principales: (i) el uso de la teoría de agentes en sistemas de E/S de alto rendimiento, (ii) la definición de un formalismo que permita la reconfiguración dinámica de nodos de almacenamiento en un cluster y (iii) el uso de técnicas de optimización de E/S configurables y orientadas a las aplicaciones