PASSION: Parallel And Scalable Software for Input-Output

We are developing a software system called PASSION: Parallel And Scalable Software for Input-Output which provides software support for high performance parallel I/O. PASSION provides support at the language, compiler, runtime as well as file system level. PASSION provides runtime procedures for parallel access to files (read/write), as well as for out-of-core computations. These routines can either be used together with a compiler to translate out-of-core data parallel programs written in a language like HPF, or used directly by application programmers. A number of optimizations such as Two-Phase Access, Data Sieving, Data Prefetching and Data Reuse have been incorporated in the PASSION Runtime Library for improved performance. PASSION also provides an initial framework for runtime support for out-of-core irregular problems. The goal of the PASSION compiler is to automatically translate out- of-core data parallel programs to node programs for distributed memory machines, with calls to the PASSION Runtime Library. At the language level, PASSION suggests extensions to HPF for out-of-core programs. At the file system level, PASSION provides support for buffering and prefetching data from disks. A portable parallel file system is also being developed as part of this project, which can be used across homogeneous or heterogeneous networks of workstations. PASSION also provides support for integrating data and task parallelism using parallel I/O techniques. We have used PASSION to implement a number of out-of-core applications such as a Laplace\u27s equation solver, 2D FFT, Matrix Multiplication, LU Decomposition, image processing applications as well as unstructured mesh kernels in molecular dynamics and computational fluid dynamics. We are currently in the process of using PASSION in applications in CFD (3D turbulent flows), molecular structure calculations, seismic computations, and earth and space science applications such as Four-Dimensional Data Assimilation. PASSION is currently available on the Intel Paragon, Touchstone Delta and iPSC/860. Efforts are underway to port it to the IBM SP-1 and SP-2 using the Vesta Parallel File System

Implementation of collective I/O in the Intel Paragon parallel file system: initial experiences

A majority of parallel applications achieve parallelism by partitioning data over multiple processors. Accessing distributed data structures such as arrays from files often requires each processor to make a large number of small noncontiguous data requests. This problem can be addressed by replacing small non-contiguous requests by large collective requests. This approach, known as collective I/O, has been found to work extremely well in practice. This paper describes implementation and evaluation of a collective I/O prototype in the Parallel File System (PFS) of the Intel Paragon Operating System (Paragon OS). We evaluate the collective I/O performance using its comparison with the PFS MRECORD and KUNIX I/O modes. It is observed that collective I/O provides significant performance improvement over accesses in the H-UNIX mode. However, in many cases, various implementation overheads cause collective I/O to provide lower performance than the H-RECORD I/O mode

Implementation of collective I/O in the Intel Paragon parallel file system: initial experiences

A majority of parallel applications achieve parallelism by partitioning data over multiple processors. Accessing distributed data structures such as arrays from files often requires each processor to make a large number of small noncontiguous data requests. This problem can be addressed by replacing small non-contiguous requests by large collective requests. This approach, known as collective I/O, has been found to work extremely well in practice. This paper describes implementation and evaluation of a collective I/O prototype in the Parallel File System (PFS) of the Intel Paragon Operating System (Paragon OS). We evaluate the collective I/O performance using its comparison with the PFS MRECORD and KUNIX I/O modes. It is observed that collective I/O provides significant performance improvement over accesses in the H-UNIX mode. However, in many cases, various implementation overheads cause collective I/O to provide lower performance than the H-RECORD I/O mode

Implementation of collective I/O in the Intel Paragon parallel file system: initial experiences

A majority of parallel applications achieve parallelism by partitioning data over multiple processors. Accessing distributed data structures such as arrays from files often requires each processor to make a large number of small noncontiguous data requests. This problem can be addressed by replacing small non-contiguous requests by large collective requests. This approach, known as collective I/O, has been found to work extremely well in practice. This paper describes implementation and evaluation of a collective I/O prototype in the Parallel File System (PFS) of the Intel Paragon Operating System (Paragon OS). We evaluate the collective I/O performance using its comparison with the PFS MRECORD and KUNIX I/O modes. It is observed that collective I/O provides significant performance improvement over accesses in the H-UNIX mode. However, in many cases, various implementation overheads cause collective I/O to provide lower performance than the H-RECORD I/O mode

A Case for Compositional File Systems

) Rajesh Bordawekar Center for Advanced Computing Research California Institute of Technology www.cacr.caltech.edu/¸rajesh Abstract This article presents a case for compositional file systems (CFSs). The CFS is designed using the endto -end argument; the basic file system attributes, therefore, are independent of the user requirements. The CFS is designed as a functionally compositional, structurally distributed, and dynamically extendible file system. The article also discusses the advantages and implementation alternatives for such file systems, and outlines possible applications. 1 Motivation File systems form an integral part of modern operating systems. File systems provide administrative support for high-level data abstractions (i.e., files), cooperate with the virtual memory system to cache data in memory, and manage storage media. Existing file systems are designed as functionally monolithic units and are optimized for a particular set of requirements. For example, the di..