Making the Most out of Direct-Access Network Attached Storage

The performance of high-speed network-attached storage applications is often limited by end-system overhead, caused primarily by memory copying and network protocol processing. In this paper, we examine alternative strategies for reducing overhead in such systems. We consider optimizations to remote procedure call (RPC)-based data transfer using either remote direct memory access (RDMA) or network interface support for pre-posting of application receive buffers. We demonstrate that both mechanisms enable file access throughput that saturates a 2Gb/s network link when performing large I/Os on relatively slow, commodity PCs. However, for multi-client workloads dominated by small I/Os, throughput is limited by the per-I/O overhead of processing RPCs in the server. For such workloads, we propose the use of a new network I/O mechanism, Optimistic RDMA (ORDMA). ORDMA is an alternative to RPC that aims to improve server throughput and response time for small I/Os. We measured performance improvements of up to 32% in server throughput and 36% in response time with use of ORDMA in our prototype.Engineering and Applied Science

CacheDAFS: User Level Client-Side Caching for the Direct Access File System

This thesis focuses on the design, implementation, and evaluation of user-level client-side caching for the Direct Access File System (DAFS). DAFS is a high performance file access protocol designed for local sharing in high-speed, low latency networked environments. DAFS operates over memory-to-memory interconnects such as Visual Interface (VI). VI provides a standard for efficient network communication by moving software overheads into hardware and eliminating the operating system from common data transfers. While much work has been done on message passing and distributed shared memory in VI-like environments, DAFS is one of the first attempts to extend user-level networking to network file systems. In the environment of high-speed networks with virtual interfaces, software overheads such as data copies and translation, buffer management and context switches become important bottlenecks. The DAFS protocol departs from traditional network file system practices to enhance performance. Distributed fil systems use client-side caching to improve performance by reducing network traffic, disk traffic, and server load. The DAFS client omits any caching. This thesis presents a user-space cache for DAFS called cacheDAFS with a careful design that avoids most bottlenecks in network file system protocols and user-level networking environments. CacheDAFS maintains perfect consistency among DAFS clients using NFSv4-like open delegations. Changes to the DAFS API in order to add caching are minimal and results show that DAFS applications can use cacheDAFS to reap all the standard benefits of caching.Engineering and Applied Science

Application Performance on the Direct Access File System

The Direct Access File System (DAFS) is a distributed file system built on top of direct-access transports (DAT). Direct-access transports are characterized by using remote direct memory access (RDMA) for data transfer and user-level networking. The motivation behind the DAT-enabled distributed file system architecture is the reduction of the CPU overhead on the I/O data path. In collaboration with Duke University we have created and made available an open-source implementation of DAFS for the FreeBSD platform. In this paper we describe a performance evaluation study of DAFS that was performed with this software. The goal of this work is to determine whether the architecture of DAFS brings any fundamental performance benefits to applications compared to traditional distributed file systems. In our study we compare DAFS to a version of NFS optimized to reduce the I/O overhead. We conclude that DAFS can accomplish superior performance for latency-sensitive applications, outperforming NFS by up to a factor of 2. Bandwidth-sensitive applications do equally well on both systems, unless they are CPU-intensive, in which case they perform better on DAFS. We also found that RDMA is a less restrictive mechanism to achieve copy avoidance than that used by the optimized NFS.Engineering and Applied Science

Application performance on the Direct Access File System

distributed file system built on top of direct-access transports (DAT). Direct-access transports are characterized by using remote direct memory access (RDMA) for data transfer and user-level networking. The motivation behind the DAT-enabled distributed file system architecture is the reduction of the CPU overhead on the I/O data path. In collaboration with Duke University we have created and made available an open-source implementation of DAFS for the FreeBSD platform. In this paper we describe a performance evaluation study of DAFS that was performed with this software. The goal of this work is to determine whether the architecture of DAFS brings any fundamental performance benefits to applications compared to traditional distributed file systems. In our study we compare DAFS to a version of NFS optimized to reduce the I/O overhead. We conclude that DAFS can accomplish superior performance for latency-sensitive applications, outperforming NFS by up to a factor of 2. Bandwidthsensitive applications do equally well on both systems, unless they are CPU-intensive, in which case they perform better on DAFS. We also found that RDMA is a less restrictive mechanism to achieve copy avoidance than that used by the optimized NFS. 1

Making the Most out of Direct-Access Network Attached Storage

The performance of high-speed network-attached storage applications is often limited by end-system overhead, caused primarily by memory copying and network protocol processing. In this paper, we examine alternative strategies for reducing overhead in such systems. We consider optimizations to remote procedure call (RPC)-based data transfer using either remote direct memory access (RDMA) or network interface support for pre-posting of application receive buffers. We demonstrate that both mechanisms enable file access throughput that saturates a 2Gb/s network link when performing large I/Os on relatively slow, commodity PCs. However, for multi-client workloads dominated by small I/Os, throughput is limited by the per-I/O overhead of processing RPCs in the server. For such workloads, we propose the use of a new network I/O mechanism, Optimistic RDMA (ORDMA). ORDMA is an alternative to RPC that aims to improve server throughput and response time for small I/Os. We measured performance improvements of up to 32 % in server throughput and 36 % in response time with use of ORDMA in our prototype.