93,418 research outputs found
HEC: Collaborative Research: SAM^2 Toolkit: Scalable and Adaptive Metadata Management for High-End Computing
The increasing demand for Exa-byte-scale storage capacity by high end computing applications requires a higher level of scalability and dependability than that provided by current file and storage systems. The proposal deals with file systems research for metadata management of scalable cluster-based parallel and distributed file storage systems in the HEC environment. It aims to develop a scalable and adaptive metadata management (SAM2) toolkit to extend features of and fully leverage the peak performance promised by state-of-the-art cluster-based parallel and distributed file storage systems used by the high performance computing community. There is a large body of research on data movement and management scaling, however, the need to scale up the attributes of cluster-based file systems and I/O, that is, metadata, has been underestimated. An understanding of the characteristics of metadata traffic, and an application of proper load-balancing, caching, prefetching and grouping mechanisms to perform metadata management correspondingly, will lead to a high scalability. It is anticipated that by appropriately plugging the scalable and adaptive metadata management components into the state-of-the-art cluster-based parallel and distributed file storage systems one could potentially increase the performance of applications and file systems, and help translate the promise and potential of high peak performance of such systems to real application performance improvements.
The project involves the following components:
1. Develop multi-variable forecasting models to analyze and predict file metadata access patterns. 2. Develop scalable and adaptive file name mapping schemes using the duplicative Bloom filter array technique to enforce load balance and increase scalability 3. Develop decentralized, locality-aware metadata grouping schemes to facilitate the bulk metadata operations such as prefetching. 4. Develop an adaptive cache coherence protocol using a distributed shared object model for client-side and server-side metadata caching. 5. Prototype the SAM2 components into the state-of-the-art parallel virtual file system PVFS2 and a distributed storage data caching system, set up an experimental framework for a DOE CMS Tier 2 site at University of Nebraska-Lincoln and conduct benchmark, evaluation and validation studies
๋ถ์ฐ ํ์ผ ์์คํ ์ ํจ์ฆ ๋ฉํฐ ์ฐ๋ ๋ ๊ตฌํ์ ํตํ ํจ์ฆ ๋ณ๋ชฉ ํ์ ์ ๊ฑฐ
ํ์๋
ผ๋ฌธ (์์ฌ)-- ์์ธ๋ํ๊ต ๋ํ์ ๊ณต๊ณผ๋ํ ์ปดํจํฐ๊ณตํ๋ถ, 2017. 8. Heon Young YEOM.A popular Filesystem in Userspace (FUSE) is widely used as a data accessing protocol on many modern distributed file systems. FUSE works as a bridge for transferring requests from user application to FUSE-based file system. Actually, it receives requests from user application and then processes some corresponding FUSE operations sequentially by a single FUSE thread. Unfortunately, when a FUSE-based distributed file system performs on workload that consists of a relatively huge number of small-file operations or a very large number of clients, it suffers from overhead of request handling caused by the single FUSE threat. This overhead is the bottleneck of the whole system which significantly degrade the overall performance. In this paper, we propose a multi-threaded FUSE framework that receives and processes requests in parallel which can eliminate the bottleneck caused by that original single FUSE thread. As long as there are requests still available waiting inside the FUSE queue, the other new FUSE thread will be automatically created to receive request and performs some specific FUSE operations simultaneously. We incorporated our mechanism into a GlusterFS distributed file system. The experiment results of our proposed mechanism indicated that depending on the workloads and hardware used, the performance upgradation is improved by 32% on small-file writing workload and 35% on small-file reading workload.Chapter 1 Introduction 1
Chapter 2 Background 4
2.1 Distributed File System 5
2.2 GlusterFS 5
2.3 FUSE 8
2.4 Performance Overhead of FUSE 12
Chapter 3 Problem Statement and Motivation 15
Chapter 4 Design and Implementation 19
Chapter 5 Evaluation 24
5.1 Experimental Setup 24
5.2 Tested Configurations 25
5.3 Benchmark Methodology 26
5.4 Benchmark Results 26
Chapter 6 Discussion 29
Chapter 7 Related Work 32
Chapter 8 Conclusion 34
Chapter 9 Future work 35
Bibliography 36
์์ฝ 39Maste
Scalable Global Grid catalogue for LHC Run3 and beyond
The AliEn (ALICE Environment) file catalogue is a global unique namespace
providing mapping between a UNIX-like logical name structure and the
corresponding physical files distributed over 80 storage elements worldwide.
Powerful search tools and hierarchical metadata information are integral parts
of the system and are used by the Grid jobs as well as local users to store and
access all files on the Grid storage elements. The catalogue has been in
production since 2005 and over the past 11 years has grown to more than 2
billion logical file names. The backend is a set of distributed relational
databases, ensuring smooth growth and fast access. Due to the anticipated fast
future growth, we are looking for ways to enhance the performance and
scalability by simplifying the catalogue schema while keeping the functionality
intact. We investigated different backend solutions, such as distributed key
value stores, as replacement for the relational database. This contribution
covers the architectural changes in the system, together with the technology
evaluation, benchmark results and conclusions.Comment: Proceedings of the 22nd International Conference on Computing in High
Energy and Nuclear Physics, CHEP 2016, 10-14 October 2016, San Francisco.
Submitted to Journal of Physics: Conference Series (JPCS
Predicting Intermediate Storage Performance for Workflow Applications
Configuring a storage system to better serve an application is a challenging
task complicated by a multidimensional, discrete configuration space and the
high cost of space exploration (e.g., by running the application with different
storage configurations). To enable selecting the best configuration in a
reasonable time, we design an end-to-end performance prediction mechanism that
estimates the turn-around time of an application using storage system under a
given configuration. This approach focuses on a generic object-based storage
system design, supports exploring the impact of optimizations targeting
workflow applications (e.g., various data placement schemes) in addition to
other, more traditional, configuration knobs (e.g., stripe size or replication
level), and models the system operation at data-chunk and control message
level.
This paper presents our experience to date with designing and using this
prediction mechanism. We evaluate this mechanism using micro- as well as
synthetic benchmarks mimicking real workflow applications, and a real
application.. A preliminary evaluation shows that we are on a good track to
meet our objectives: it can scale to model a workflow application run on an
entire cluster while offering an over 200x speedup factor (normalized by
resource) compared to running the actual application, and can achieve, in the
limited number of scenarios we study, a prediction accuracy that enables
identifying the best storage system configuration
06472 Abstracts Collection - XQuery Implementation Paradigms
From 19.11.2006 to 22.11.2006, the Dagstuhl Seminar 06472 ``XQuery Implementation Paradigms'' was held in the International Conference and Research Center (IBFI), Schloss Dagstuhl. During the seminar, several participants presented their current research, and ongoing work and open problems were discussed. Abstracts of the presentations given during the seminar as well as abstracts of seminar results and ideas are put together in this paper. The first section describes the seminar topics and goals in general. Links to extended abstracts or full papers are provided, if available
A Benchmark for Image Retrieval using Distributed Systems over the Internet: BIRDS-I
The performance of CBIR algorithms is usually measured on an isolated
workstation. In a real-world environment the algorithms would only constitute a
minor component among the many interacting components. The Internet
dramati-cally changes many of the usual assumptions about measuring CBIR
performance. Any CBIR benchmark should be designed from a networked systems
standpoint. These benchmarks typically introduce communication overhead because
the real systems they model are distributed applications. We present our
implementation of a client/server benchmark called BIRDS-I to measure image
retrieval performance over the Internet. It has been designed with the trend
toward the use of small personalized wireless systems in mind. Web-based CBIR
implies the use of heteroge-neous image sets, imposing certain constraints on
how the images are organized and the type of performance metrics applicable.
BIRDS-I only requires controlled human intervention for the compilation of the
image collection and none for the generation of ground truth in the measurement
of retrieval accuracy. Benchmark image collections need to be evolved
incrementally toward the storage of millions of images and that scaleup can
only be achieved through the use of computer-aided compilation. Finally, our
scoring metric introduces a tightly optimized image-ranking window.Comment: 24 pages, To appear in the Proc. SPIE Internet Imaging Conference
200
- โฆ