2,604 research outputs found
AliEnFS - a Linux File System for the AliEn Grid Services
Among the services offered by the AliEn (ALICE Environment
http://alien.cern.ch) Grid framework there is a virtual file catalogue to allow
transparent access to distributed data-sets using various file transfer
protocols. (AliEn File System) integrates the AliEn file catalogue as
a new file system type into the Linux kernel using LUFS, a hybrid user space
file system framework (Open Source http://lufs.sourceforge.net). LUFS uses a
special kernel interface level called VFS (Virtual File System Switch) to
communicate via a generalised file system interface to the AliEn file system
daemon. The AliEn framework is used for authentication, catalogue browsing,
file registration and read/write transfer operations. A C++ API implements the
generic file system operations. The goal of AliEnFS is to allow users easy
interactive access to a worldwide distributed virtual file system using
familiar shell commands (f.e. cp,ls,rm ...) The paper discusses general aspects
of Grid File Systems, the AliEn implementation and present and future
developments for the AliEn Grid File System.Comment: 9 pages, 12 figure
λ°μ΄ν° μ§μ½μ μμ©μ ν¨μ¨μ μΈ μμ€ν μμ νμ©μ μν λ©λͺ¨λ¦¬ μλΈμμ€ν μ΅μ ν
νμλ
Όλ¬Έ (λ°μ¬) -- μμΈλνκ΅ λνμ : 곡과λν μ κΈ°Β·μ»΄ν¨ν°κ³΅νλΆ, 2020. 8. μΌνμ.With explosive data growth, data-intensive applications, such as relational database and key-value storage, have been increasingly popular in a variety of domains in recent years. To meet the growing performance demands of data-intensive applications, it is crucial to efficiently and fully utilize memory resources for the best possible performance.
However, general-purpose operating systems (OSs) are designed to provide system resources to applications running on a system in a fair manner at system-level. A single application may find it difficult to fully exploit the systems best performance due to this system-level fairness. For performance reasons, many data-intensive applications implement their own mechanisms that OSs already provide, under the assumption that they know better about the data than OSs. They can be greedily optimized for performance but this may result in inefficient use of system resources.
In this dissertation, we claim that simple OS support with minor application modifications can yield even higher application performance without sacrificing system-level resource utilization. We optimize and extend OS memory subsystem for better supporting applications while addressing three memory-related issues in data-intensive applications. First, we introduce a memory-efficient cooperative caching approach between application and kernel buffer to address double caching problem where the same data resides in multiple layers. Second, we present a memory-efficient, transparent zero-copy read I/O scheme to avoid the performance interference problem caused by memory copy behavior during I/O. Third, we propose a memory-efficient fork-based checkpointing mechanism for in-memory database systems to mitigate the memory footprint problem of the existing fork-based checkpointing scheme; memory usage increases incrementally (up to 2x) during checkpointing for update-intensive workloads.
To show the effectiveness of our approach, we implement and evaluate our schemes on real multi-core systems. The experimental results demonstrate that our cooperative approach can more effectively address the above issues related to data-intensive applications than existing non-cooperative approaches while delivering better performance (in terms of transaction processing speed, I/O throughput, or memory footprint).μ΅κ·Ό νλ°μ μΈ λ°μ΄ν° μ±μ₯κ³Ό λλΆμ΄ λ°μ΄ν°λ² μ΄μ€, ν€-λ°Έλ₯ μ€ν λ¦¬μ§ λ±μ λ°μ΄ν° μ§μ½μ μΈ μμ©λ€μ΄ λ€μν λλ©μΈμμ μΈκΈ°λ₯Ό μ»κ³ μλ€. λ°μ΄ν° μ§μ½μ μΈ μμ©μ λμ μ±λ₯ μꡬλ₯Ό μΆ©μ‘±νκΈ° μν΄μλ μ£Όμ΄μ§ λ©λͺ¨λ¦¬ μμμ ν¨μ¨μ μ΄κ³ μλ²½νκ² νμ©νλ κ²μ΄ μ€μνλ€. κ·Έλ¬λ, λ²μ© μ΄μ체μ (OS)λ μμ€ν
μμ μν μ€μΈ λͺ¨λ μμ©λ€μ λν΄ μμ€ν
μ°¨μμμ 곡ννκ² μμμ μ 곡νλ κ²μ μ°μ νλλ‘ μ€κ³λμ΄μλ€. μ¦, μμ€ν
μ°¨μμ 곡νμ± μ μ§λ₯Ό μν μ΄μ체μ μ§μμ νκ³λ‘ μΈν΄ λ¨μΌ μμ©μ μμ€ν
μ μ΅κ³ μ±λ₯μ μμ ν νμ©νκΈ° μ΄λ ΅λ€. μ΄λ¬ν μ΄μ λ‘, λ§μ λ°μ΄ν° μ§μ½μ μμ©μ μ΄μ체μ μμ μ 곡νλ κΈ°λ₯μ μμ§νμ§ μκ³ λΉμ·ν κΈ°λ₯μ μμ© λ 벨μ ꡬννκ³€ νλ€. μ΄λ¬ν μ κ·Ό λ°©λ²μ νμμ μΈ μ΅μ νκ° κ°λ₯νλ€λ μ μμ μ±λ₯ μ μ΄λμ΄ μμ μ μμ§λ§, μμ€ν
μμμ λΉν¨μ¨μ μΈ μ¬μ©μ μ΄λν μ μλ€.
λ³Έ λ
Όλ¬Έμμλ μ΄μ체μ μ μ§μκ³Ό μ½κ°μ μμ© μμ λ§μΌλ‘λ λΉν¨μ¨μ μΈ μμ€ν
μμ μ¬μ© μμ΄ λ³΄λ€ λμ μμ© μ±λ₯μ λ³΄μΌ μ μμμ μ¦λͺ
νκ³ μ νλ€. κ·Έλ¬κΈ° μν΄ μ΄μ체μ μ λ©λͺ¨λ¦¬ μλΈμμ€ν
μ μ΅μ ν λ° νμ₯νμ¬ λ°μ΄ν° μ§μ½μ μΈ μμ©μμ λ°μνλ μΈ κ°μ§ λ©λͺ¨λ¦¬ κ΄λ ¨ λ¬Έμ λ₯Ό ν΄κ²°νμλ€. 첫째, λμΌν λ°μ΄ν°κ° μ¬λ¬ κ³μΈ΅μ μ‘΄μ¬νλ μ€λ³΅ μΊμ± λ¬Έμ λ₯Ό ν΄κ²°νκΈ° μν΄ μμ©κ³Ό 컀λ λ²νΌ κ°μ λ©λͺ¨λ¦¬ ν¨μ¨μ μΈ νλ ₯ μΊμ± λ°©μμ μ μνμλ€. λμ§Έ, μ
μΆλ ₯μ λ°μνλ λ©λͺ¨λ¦¬ 볡μ¬λ‘ μΈν μ±λ₯ κ°μ λ¬Έμ λ₯Ό νΌνκΈ° μν΄ λ©λͺ¨λ¦¬ ν¨μ¨μ μΈ λ¬΄λ³΅μ¬ μ½κΈ° μ
μΆλ ₯ λ°©μμ μ μνμλ€. μ
μ§Έ, μΈ-λ©λͺ¨λ¦¬ λ°μ΄ν°λ² μ΄μ€ μμ€ν
μ μν λ©λͺ¨λ¦¬ ν¨μ¨μ μΈ fork κΈ°λ° μ²΄ν¬ν¬μΈνΈ κΈ°λ²μ μ μνμ¬ κΈ°μ‘΄ ν¬ν¬ κΈ°λ° μ²΄ν¬ν¬μΈνΈ κΈ°λ²μμ λ°μνλ λ©λͺ¨λ¦¬ μ¬μ©λ μ¦κ° λ¬Έμ λ₯Ό μννμλ€; κΈ°μ‘΄ λ°©μμ μ
λ°μ΄νΈ μ§μ½μ μν¬λ‘λμ λν΄ μ²΄ν¬ν¬μΈν
μ μννλ λμ λ©λͺ¨λ¦¬ μ¬μ©λμ΄ μ΅λ 2λ°°κΉμ§ μ μ§μ μΌλ‘ μ¦κ°ν μ μμλ€.
λ³Έ λ
Όλ¬Έμμλ μ μν λ°©λ²λ€μ ν¨κ³Όλ₯Ό μ¦λͺ
νκΈ° μν΄ μ€μ λ©ν° μ½μ΄ μμ€ν
μ ꡬννκ³ κ·Έ μ±λ₯μ νκ°νμλ€. μ€νκ²°κ³Όλ₯Ό ν΅ν΄ μ μν νλ ₯μ μ κ·Όλ°©μμ΄ κΈ°μ‘΄μ λΉνλ ₯μ μ κ·Όλ°©μλ³΄λ€ λ°μ΄ν° μ§μ½μ μμ©μκ² ν¨μ¨μ μΈ λ©λͺ¨λ¦¬ μμ νμ©μ
κ°λ₯νκ² ν¨μΌλ‘μ¨ λ λμ μ±λ₯μ μ 곡ν μ μμμ νμΈν μ μμλ€.Chapter 1 Introduction 1
1.1 Motivation 1
1.1.1 Importance of Memory Resources 1
1.1.2 Problems 2
1.2 Contributions 5
1.3 Outline 6
Chapter 2 Background 7
2.1 Linux Kernel Memory Management 7
2.1.1 Page Cache 7
2.1.2 Page Reclamation 8
2.1.3 Page Table and TLB Shootdown 9
2.1.4 Copy-on-Write 10
2.2 Linux Support for Applications 11
2.2.1 fork 11
2.2.2 madvise 11
2.2.3 Direct I/O 12
2.2.4 mmap 13
Chapter 3 Memory Efficient Cooperative Caching 14
3.1 Motivation 14
3.1.1 Problems of Existing Datastore Architecture 14
3.1.2 Proposed Architecture 17
3.2 Related Work 17
3.3 Design and Implementation 19
3.3.1 Overview 19
3.3.2 Kernel Support 24
3.3.3 Migration to DBIO 25
3.4 Evaluation 27
3.4.1 System Configuration 27
3.4.2 Methodology 28
3.4.3 TPC-C Benchmarks 30
3.4.4 YCSB Benchmarks 32
3.5 Summary 37
Chapter 4 Memory Efficient Zero-copy I/O 38
4.1 Motivation 38
4.1.1 The Problems of Copy-Based I/O 38
4.2 Related Work 40
4.2.1 Zero Copy I/O 40
4.2.2 TLB Shootdown 42
4.2.3 Copy-on-Write 43
4.3 Design and Implementation 44
4.3.1 Prerequisites for z-READ 44
4.3.2 Overview of z-READ 45
4.3.3 TLB Shootdown Optimization 48
4.3.4 Copy-on-Write Optimization 52
4.3.5 Implementation 55
4.4 Evaluation 55
4.4.1 System Configurations 56
4.4.2 Effectiveness of the TLB Shootdown Optimization 57
4.4.3 Effectiveness of CoW Optimization 59
4.4.4 Analysis of the Performance Improvement 62
4.4.5 Performance Interference Intensity 63
4.4.6 Effectiveness of z-READ in Macrobenchmarks 65
4.5 Summary 67
Chapter 5 Memory Efficient Fork-based Checkpointing 69
5.1 Motivation 69
5.1.1 Fork-based Checkpointing 69
5.1.2 Approach 71
5.2 Related Work 73
5.3 Design and Implementation 74
5.3.1 Overview 74
5.3.2 OS Support 78
5.3.3 Implementation 79
5.4 Evaluation 80
5.4.1 Experimental Setup 80
5.4.2 Performance 81
5.5 Summary 86
Chapter 6 Conclusion 87
μμ½ 100Docto
Using khazana to support distributed application development
technical reportOne of the most important services required by most distributed applications is some form of shared data management, e.g., a directory service manages shared directory entries while groupware manages shared documents. Each such application currently must implement its own data management mechanisms, because existing runtime systems are not flexible enough to support all distributed applications efficiently. For example, groupware can be efficiently supported by a distributed object system, while a distributed database would prefer a more low-level storage abstraction. The goal of Khazana is to provide programmer's with configurable components that support the data management services required by a wide variety of distributed applications, including: consistent caching, automated replication and migration of data, persistence, access control, and fault tolerance. It does so via a carefully designed set of interfaces that supports a hierarchy of data abstractions, ranging from flat data to C++/Java objects, and that give programmers a great of control over how their data is managed. To demonstrate the effectiveness of our design, we report on our experience porting three applications to Khazana: a distributed file system, a distributed directory service, and a shared whiteboard
Interest-Based Access Control for Content Centric Networks (extended version)
Content-Centric Networking (CCN) is an emerging network architecture designed
to overcome limitations of the current IP-based Internet. One of the
fundamental tenets of CCN is that data, or content, is a named and addressable
entity in the network. Consumers request content by issuing interest messages
with the desired content name. These interests are forwarded by routers to
producers, and the resulting content object is returned and optionally cached
at each router along the path. In-network caching makes it difficult to enforce
access control policies on sensitive content outside of the producer since
routers only use interest information for forwarding decisions. To that end, we
propose an Interest-Based Access Control (IBAC) scheme that enables access
control enforcement using only information contained in interest messages,
i.e., by making sensitive content names unpredictable to unauthorized parties.
Our IBAC scheme supports both hash- and encryption-based name obfuscation. We
address the problem of interest replay attacks by formulating a mutual trust
framework between producers and consumers that enables routers to perform
authorization checks when satisfying interests from their cache. We assess the
computational, storage, and bandwidth overhead of each IBAC variant. Our design
is flexible and allows producers to arbitrarily specify and enforce any type of
access control on content, without having to deal with the problems of content
encryption and key distribution. This is the first comprehensive design for CCN
access control using only information contained in interest messages.Comment: 11 pages, 2 figure
Dynamic Virtual Page-based Flash Translation Layer with Novel Hot Data Identification and Adaptive Parallelism Management
Solid-state disks (SSDs) tend to replace traditional motor-driven hard disks in high-end storage devices in past few decades. However, various inherent features, such as out-of-place update [resorting to garbage collection (GC)] and limited endurance (resorting to wear leveling), need to be reduced to a large extent before that day comes. Both the GC and wear leveling fundamentally depend on hot data identification (HDI). In this paper, we propose a hot data-aware flash translation layer architecture based on a dynamic virtual page (DVPFTL) so as to improve the performance and lifetime of NAND flash devices. First, we develop a generalized dual layer HDI (DL-HDI) framework, which is composed of a cold data pre-classifier and a hot data post-identifier. Those can efficiently follow the frequency and recency of information access. Then, we design an adaptive parallelism manager (APM) to assign the clustered data chunks to distinct resident blocks in the SSD so as to prolong its endurance. Finally, the experimental results from our realized SSD prototype indicate that the DVPFTL scheme has reliably improved the parallelizability and endurance of NAND flash devices with improved GC-costs, compared with related works.Peer reviewe
Recommended from our members
Computing infrastructure issues in distributed communications systems : a survey of operating system transport system architectures
The performance of distributed applications (such as file transfer, remote login, tele-conferencing, full-motion video, and scientific visualization) is influenced by several factors that interact in complex ways. In particular, application performance is significantly affected both by communication infrastructure factors and computing infrastructure factors. Several communication infrastructure factors include channel speed, bit-error rate, and congestion at intermediate switching nodes. Computing infrastructure factors include (among other things) both protocol processing activities (such as connection management, flow control, error detection, and retransmission) and general operating system factors (such as memory latency, CPU speed, interrupt and context switching overhead, process architecture, and message buffering). Due to a several orders of magnitude increase in network channel speed and an increase in application diversity, performance bottlenecks are shifting from the network factors to the transport system factors.This paper defines an abstraction called an "Operating System Transport System Architecture" (OSTSA) that is used to classify the major components and services in the computing infrastructure. End-to-end network protocols such as TCP, TP4, VMTP, XTP, and Delta-t typically run on general-purpose computers, where they utilize various operating system resources such as processors, virtual memory, and network controllers. The OSTSA provides services that integrate these resources to support distributed applications running on local and wide area networks.A taxonomy is presented to evaluate OSTSAs in terms of their support for protocol processing activities. We use this taxonomy to compare and contrast five general-purpose commercial and experimental operating systems including System V UNIX, BSD UNIX, the x-kernel, Choices, and Xinu
- β¦