77,658 research outputs found
Chunked extendible arrays and its integration with the global array toolkit for parallel image processing
A thesis submitted to the Faculty of Engineering and the Built Environment
in fulfilment of the requirements for the degree of
Doctor of Philosophy, 2016Online resource (xii, 151 leaves)Several meetings of the Extremely Large Databases Community for large scale
scientific applications have advocated the use of multidimensional arrays as the
appropriate model for representing scientific databases. Scientific databases gradually
grow to massive sizes of the order of terabytes and petabytes. As such, the storage of
such databases requires efficient dynamic storage schemes where the array is allowed
to arbitrarily extend the bounds of the dimensions. Conventional multidimensional
array representations in today’s programming environments do not extend or shrink
their bounds without relocating elements of the data-set. In general extendibility of
the bounds of the dimensions is limited to only one dimension. This thesis presents a
technique for storing dense multidimensional arrays by chunks such that the array can
be extended along any dimension without compromising the access time of an element.
This is done with a computed access mapping function that maps the k-dimensional
index onto a linear index of the storage locations. This concept forms the basis for
the implementation of an array file of any number of dimensions, where the bounds
of the array dimension can be extended arbitrarily. Such a feature currently exists in
the Hierarchical Data Format version 5 (HDF5). However, extending the bound of a
dimension in the HDF5 array file can be unusually expensive in time. Such extensions,
in our storage scheme for dense array files, can be performed while still accessing
elements of the array at orders of magnitude faster than in HDF5 or conventional
array-files. We also present Parallel Chunked Extendible Dense Array (PEXTA), a
new parallel I/O model for the Global Array Toolkit. PEXTA provides the necessary
Application Programming Interface (API) for explicit data transfer between the
memory resident global array and its secondary storage counterpart but also allows
the persistent array to be extended on any dimension without compromising the
access time of an element or sub-array elements. Such APIs provide a platform
for high speed and parallel hyperspectral image processing without performance
degradation, even when the imagery files undergo extensions.MT201
HEC: Collaborative Research: SAM^2 Toolkit: Scalable and Adaptive Metadata Management for High-End Computing
The increasing demand for Exa-byte-scale storage capacity by high end computing applications requires a higher level of scalability and dependability than that provided by current file and storage systems. The proposal deals with file systems research for metadata management of scalable cluster-based parallel and distributed file storage systems in the HEC environment. It aims to develop a scalable and adaptive metadata management (SAM2) toolkit to extend features of and fully leverage the peak performance promised by state-of-the-art cluster-based parallel and distributed file storage systems used by the high performance computing community. There is a large body of research on data movement and management scaling, however, the need to scale up the attributes of cluster-based file systems and I/O, that is, metadata, has been underestimated. An understanding of the characteristics of metadata traffic, and an application of proper load-balancing, caching, prefetching and grouping mechanisms to perform metadata management correspondingly, will lead to a high scalability. It is anticipated that by appropriately plugging the scalable and adaptive metadata management components into the state-of-the-art cluster-based parallel and distributed file storage systems one could potentially increase the performance of applications and file systems, and help translate the promise and potential of high peak performance of such systems to real application performance improvements.
The project involves the following components:
1. Develop multi-variable forecasting models to analyze and predict file metadata access patterns. 2. Develop scalable and adaptive file name mapping schemes using the duplicative Bloom filter array technique to enforce load balance and increase scalability 3. Develop decentralized, locality-aware metadata grouping schemes to facilitate the bulk metadata operations such as prefetching. 4. Develop an adaptive cache coherence protocol using a distributed shared object model for client-side and server-side metadata caching. 5. Prototype the SAM2 components into the state-of-the-art parallel virtual file system PVFS2 and a distributed storage data caching system, set up an experimental framework for a DOE CMS Tier 2 site at University of Nebraska-Lincoln and conduct benchmark, evaluation and validation studies
Recommended from our members
Parallel data compression
Data compression schemes remove data redundancy in communicated and stored data and increase the effective capacities of communication and storage devices. Parallel algorithms and implementations for textual data compression are surveyed. Related concepts from parallel computation and information theory are briefly discussed. Static and dynamic methods for codeword construction and transmission on various models of parallel computation are described. Included are parallel methods which boost system speed by coding data concurrently, and approaches which employ multiple compression techniques to improve compression ratios. Theoretical and empirical comparisons are reported and areas for future research are suggested
Computing in the RAIN: a reliable array of independent nodes
The RAIN project is a research collaboration between Caltech and NASA-JPL on distributed computing and data-storage systems for future spaceborne missions. The goal of the project is to identify and develop key building blocks for reliable distributed systems built with inexpensive off-the-shelf components. The RAIN platform consists of a heterogeneous cluster of computing and/or storage nodes connected via multiple interfaces to networks configured in fault-tolerant topologies. The RAIN software components run in conjunction with operating system services and standard network protocols. Through software-implemented fault tolerance, the system tolerates multiple node, link, and switch failures, with no single point of failure. The RAIN-technology has been transferred to Rainfinity, a start-up company focusing on creating clustered solutions for improving the performance and availability of Internet data centers. In this paper, we describe the following contributions: 1) fault-tolerant interconnect topologies and communication protocols providing consistent error reporting of link failures, 2) fault management techniques based on group membership, and 3) data storage schemes based on computationally efficient error-control codes. We present several proof-of-concept applications: a highly-available video server, a highly-available Web server, and a distributed checkpointing system. Also, we describe a commercial product, Rainwall, built with the RAIN technology
Alpha Entanglement Codes: Practical Erasure Codes to Archive Data in Unreliable Environments
Data centres that use consumer-grade disks drives and distributed
peer-to-peer systems are unreliable environments to archive data without enough
redundancy. Most redundancy schemes are not completely effective for providing
high availability, durability and integrity in the long-term. We propose alpha
entanglement codes, a mechanism that creates a virtual layer of highly
interconnected storage devices to propagate redundant information across a
large scale storage system. Our motivation is to design flexible and practical
erasure codes with high fault-tolerance to improve data durability and
availability even in catastrophic scenarios. By flexible and practical, we mean
code settings that can be adapted to future requirements and practical
implementations with reasonable trade-offs between security, resource usage and
performance. The codes have three parameters. Alpha increases storage overhead
linearly but increases the possible paths to recover data exponentially. Two
other parameters increase fault-tolerance even further without the need of
additional storage. As a result, an entangled storage system can provide high
availability, durability and offer additional integrity: it is more difficult
to modify data undetectably. We evaluate how several redundancy schemes perform
in unreliable environments and show that alpha entanglement codes are flexible
and practical codes. Remarkably, they excel at code locality, hence, they
reduce repair costs and become less dependent on storage locations with poor
availability. Our solution outperforms Reed-Solomon codes in many disaster
recovery scenarios.Comment: The publication has 12 pages and 13 figures. This work was partially
supported by Swiss National Science Foundation SNSF Doc.Mobility 162014, 2018
48th Annual IEEE/IFIP International Conference on Dependable Systems and
Networks (DSN
Access to vectors in multi-module memories
The poor bandwidth obtained from memory when conflicts arise in the modules or in the interconnection network degrades the performance of computers. Address transformation schemes, such as interleaving, skewing and linear transformations, have been proposed to achieve conflict-free access for streams with constant stride. However, this is achieved only for some strides. In this paper, we summarize a mechanism to request the elements in an out-of-order way which allows to achieve
conflict-free access for a larger number of strides. We study the cases of a single vector processor and of a vector multiprocessor system. For this latter case, we propose a synchronous mode of accessing memory that can be applied in SIMD machines or in MIMD systems with decoupled access and execution.Peer ReviewedPostprint (published version
A Sparse SCF algorithm and its parallel implementation: Application to DFTB
We present an algorithm and its parallel implementation for solving a self
consistent problem as encountered in Hartree Fock or Density Functional Theory.
The algorithm takes advantage of the sparsity of matrices through the use of
local molecular orbitals. The implementation allows to exploit efficiently
modern symmetric multiprocessing (SMP) computer architectures. As a first
application, the algorithm is used within the density functional based tight
binding method, for which most of the computational time is spent in the linear
algebra routines (diagonalization of the Fock/Kohn-Sham matrix). We show that
with this algorithm (i) single point calculations on very large systems
(millions of atoms) can be performed on large SMP machines (ii) calculations
involving intermediate size systems (1~000--100~000 atoms) are also strongly
accelerated and can run efficiently on standard servers (iii) the error on the
total energy due to the use of a cut-off in the molecular orbital coefficients
can be controlled such that it remains smaller than the SCF convergence
criterion.Comment: 13 pages, 11 figure
- …