LSM-trees have emerged as the write-optimized index of choice for key-value stores and relational database systems. LSM-trees typically rely on a storage manager on top of a file system for storing data on Solid-State Drives (SSDs). The I/O path thus comprises four layers, each independently managing similar indirection, journaling, and garbage collection mechanisms. Such overhead isincreasingly problematic. First, the advent of microsecond-scale SSDs makes it necessary to streamline the I/O software stack. Second, the increasing performance gap between storage and CPU makes it necessary to reduce CPU storage overhead. A solution is to collapse LSM, file system, and SSD management layers into a single software layer embedded on computational storage. Specific commercial solutions are already available. In this short paper, we describe the design space for LSM management on computational storage

Bonnet, Philippe

Picoli, Ivan Luiz

Tözün, Pinar

English

Crossref

LSM Management on Computational Storage

The IT University of Copenhagen's Repository

LSM Management on Computational StorageIvan Luiz PicoliIT University of Copenhagenivpi@itu.dkPhilippe BonnetIT University of Copenhagenphbo@itu.dkPınar TözünIT University of Copenhagenpito@itu.dkABSTRACTLSM-trees have emerged as the write-optimized index of choicefor key-value stores and relational database systems. LSM-treestypically rely on a storage manager on top of a file system forstoring data on Solid-State Drives (SSDs). The I/O path thus com-prises four layers, each independently managing similar indirection,journaling, and garbage collection mechanisms. Such overhead isincreasingly problematic. First, the advent of microsecond-scaleSSDs makes it necessary to streamline the I/O software stack. Sec-ond, the increasing performance gap between storage and CPUmakes it necessary to reduce CPU storage overhead. A solution isto collapse LSM, file system, and SSD management layers into asingle software layer embedded on computational storage. Specificcommercial solutions are already available. In this short paper, wedescribe the design space for LSM management on computationalstorage.ACM Reference Format:Ivan Luiz Picoli, Philippe Bonnet, and Pınar Tözün. 2019. LSM Manage-ment on Computational Storage. In Proceedings of ACM Conference (Con-ference’17). ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn1 INTRODUCTIONIn recent years, the rate at which data can be read from or writtento storage and network devices has increased exponentially, whilethe rate at which data can be read from or written to the memoryof a host processor (CPU) has only increased linearly. This trendis expected to continue in the coming years. Soon, CPUs will notbe able to keep up with the rate at which data moves through stor-age and network. Computational storage, or near-data processing,architectures are re-emerging to tackle this issue by offloading low-level storage processing from host CPU to storage controllers [8].At the same time, the emergence of Solid State Drives (SSDs) withmicrosecond-scale latency makes it necessary to streamline thestorage stack to avoid redundancies and to bypass bottlenecks.The confluence of these two trends motivates a profound re-design of the storage stack. Two of the key questions in this contextare: (a) How to streamline existing layers on the I/O path? and (b)What components of the storage stack can be offloaded from thehost CPU to the storage controller?As initially pioneered by active disks, it makes sense to offloadaccess methods on the storage controller to minimize data move-ment in database systems. In particular, LSM-trees constitute idealcandidates for collapsing layers on the storage stack in order tostreamline the I/O path. Indeed, LSM-trees and Flash TranslationLayers both rely on immutable storage structuctures associatedwith a form of garbage collection. Initial work by Baidu [9] andCNEX Labs [7] focused on supporting an LSM-based key-valuestore on open-channel SSDs by implementing an LSM-Tree specificFTL on the host. These solutions are defined in the context of tradi-tional host-SSD architectures. On the other hand, recently, Samsungannounced KV-SSD, an SSD equippedwith an LSM-tree specific FTL[3]. Can these solutions be applied in the context of computationalstorage? What are the possible architectures for LSM managementon computational storage? What is the associated design space?We tackle these questions in this short paper.2 ARCHITECTURELet us briefly present an overview of the architectural impact ofcomputational storage, illustrated in Figure 1. In traditional archi-tectures (Figure 1a), a KV-store implemented on the host relieson the file system and block layer from the operating system aswell as a generic FTL embedded on the SSD. Open-channel SSDs(Figure 1b) [2, 4] make it possible to specialize the FTL for LSMManagement [7]. While Figure 1b improves on Figure 1a by stream-lining the data path, it does so by increasing the load on the host.In order to reduce the load on the host, it is necessary to offloadprocessing to the SSD.SSDFPGANICkey-value store(e.g., RocksDB)customkey-value store(d) FPGA-basedstorage controller(c) KV-SSDNVMe driverNVMe driverSSDKV-FTLkey-value store(e.g., RocksDB)open-channel SSDlightlsmlightnvmkey-value store(e.g., RocksDB)lightlsmlightnvmopen-channel SSDFTLuser-spacekernel-space(b) local OC-SSD(e) NIC-basedstorage controllerstoragenetworkinterfacekey-value store(e.g., RocksDB)(a) traditional SSDPOSIX APISSDFTLfilesystemblock layerNVMe driverFabricPCIePCIePCIePCIe PCIeFigure 1: State-of-the-art architectures with (a) traditionalSSDs and (b) open-channel SSDs compared to (c) a customKV-store on a specialized KV-SSD and upcoming architec-tures based on (d) FPGA-based computational storage aswellas (e) NIC-based computational storage.Samsung recently announced KV-SSD, an SSD equipped withan LSM-specialized FTL (Figure 1c). KV-SSD requires a specializedNVMe driver on the host, which is accessed from a custom KV-store.Conference’17, July 2017, Washington, DC, USA Ivan Luiz Picoli, Philippe Bonnet, and Pınar TözünAn alternative to this specialized solution is to consider general-purpose computational storage. So far, two types of computationalstorage have emerged. The first type integrates a programmablestorage controller directly onto an SSD (Figure 1d). Commercialproducts are available in this space (e.g., Samsung SmartSSD, Scale-Flux, NGD). The programmable storage is either a Linux-basedARM processor or programmable hardware (FPGA). The secondtype combines Open-Channel SSDs with a programmable storagecontroller integrated with a Network Interface Card (Figure 1e).Commercial products are available from Broadcom or Mellanox,while research products include the DFC Platform [1]. While botharchitectures support functionality offload from the host, the feasi-bility to collapse layers and streamline the I/O path on the Figure 1darchitecture is unclear as some products only expose storage mediato the storage controller via a proprietary generic FTL.In the rest of this paper, we assume the Figure 1e architecture todiscuss the design space.3 DESIGN SPACEIn this section, we highlight the opportunities and trade-offs as-sociated with LSM management on computational storage with aspecialized FTL. While previous work has focused on leveragingLSM-tree compaction as a form of garbage collection, here we focuson the challenges of interface, mapping, and journaling.An LSM-tree is composed of levels. The top level is an in-memorybuffer called mem-table, while the rest of the levels are kept as fileson secondary storage called Sorted Sequence Table (SST). We referreaders to Dayan et al. [6] for a thorough description of LSM-treeinternals.3.1 InterfaceAn initial design decision is whether LSM management should befully offloaded to computational storage, or whether one shouldsplit the functionalities. An obvious split is to offload SST manage-ment on computational storage and leave mem-table managementon the host. The rationale is two-fold. First, offloading SST manage-ment on computational storage makes it possible to handle com-paction within storage, thus avoiding unnecessary data movementto the host, and enabling compaction to be the sole form of garbagecollection in an LSM-specific FTL. Second, keeping mem-tables onthe host leverages locality to minimize data movement.An LSM-based storage manager manages auxiliary data struc-tures in addition to mem-tables and SST. For example, RocksDB hasWAL for recovering mem-tables and manifest files to keep track ofSSTs with their corresponding key range. Ideally, those data struc-tures are maintained in persistent memory with an in-memory datastructure for the manifest contents for faster access. Alternatively,computational storage should support storing/retrieving these datastructures. On completion of a command flushing an SST to stor-age, the WAL entry for the given SST can be discarded. The idealmanagement of these data structures over computational storagehas to be studied.3.2 MappingAny FTL maps a logical name space onto the physical address space.FTLs specialized for LSM management must map SSTs onto thephysical address space. In open-channel SSDs, the physical addressspace is represented as a hierarchical space of groups, parallel units(PUs), chunks, and logical blocks. The minimal unit of read andwrite is a logical block. This is a clean abstraction of the physicalspace used by any FTL. We consider that the maximum size ofan SST is fixed in such a way that it corresponds to the productnumber_o f _parallel_units ∗ chunk_size1. On an Open-ChannelSSD with 32 PUs and 1MB chunks, we get a maximum SST size of32MB, with 4KB as the unit of reads.Such an SST size gives us a range of mapping options. The fistoption is a horizontal mapping, where the contents of each SST isstriped across all PUs. This option maximizes write throughput, butit may cause interference that will negatively impact read latency,as any read will interfere with any write of an SST. The second op-tion is a vertical mapping, where the contents of each SST is stripedacross the PUs of a given group. This option trades write through-put for improved concurrency as a maximum ofnumber_o f _дroupsreads and SST writes can be executed concurrently without inter-ference. A third option is a hybrid mapping, where the contentsof each SST is striped across PUs in a subset of all groups. Such amapping might offer a compromise between write throughput andread latency. In vertical and hybrid mapping, the groups on whichSSTs are striped are picked in a round-robin fashion.A thorough investigation of the trade-off between write through-put and read latency for LSM-tree management on computationalstorage is needed.3.3 JournalingThe state of the software embedded on computational storage mustbe re-constructed whenever it starts up or recovery is needed. It isnatural to consider the modifications of this state as transactionsfor which the durability property is guaranteed. The transactionalnature of the FTL for traditional SSDs has been observed in priorwork [5].A first option to ensure durability is write-ahead logging. Specificportions of the storage space (e.g., specific chunks) must be reservedfor this purpose. A second option is to use the out-of-bound area onflash storage to store metadata (e.g., reverse mapping in a traditionalFTL). A third option is to store snapshots at fixed locations in theaddress space.The key tradeoff here is between the speed of recovery and spaceamplification, which has to be studied to design efficient recoveryprocedures for LSM-trees on computational storage.4 CONCLUSIONWe have given an overview of the fundamental trade-offs for thedesign of LSMmanagement solutions on computational storage. Wehave identified the design of interfaces, mapping, and journalingas open issues. Exploring this design space on various forms ofcomputational storage is an interesting topic for future work.REFERENCES[1] DFC Platform. https://github.com/DFC-OpenSource.[2] Open-Channel SSD. https://openchannelssd.readthedocs.io/en/latest/.[3] Samsung KVSSD. https://github.com/OpenMPDK/KVSSD.1The size of each SST might vary, but it will not exceed the given limit.LSM Management on Computational Storage Conference’17, July 2017, Washington, DC, USA[4] M. Bjørling, J. González, and P. Bonnet. LightNVM: The Linux Open-channel SSDSubsystem. In FAST, pages 359–373, 2017.[5] J.-Y. Choi, E. H. Nam, Y. J. Seong, J. H. Yoon, S. Lee, H. S. Kim, J. Park, Y.-J. Woo,S. Lee, and S. L. Min. HIL: A Framework for Compositional FTL Development andProvably-Correct Crash Recovery. ACM Transactions on Storage (TOS), 14(4), 2018.[6] N. Dayan, M. Athanassoulis, and S. Idreos. Monkey: Optimal Navigable Key-ValueStore. In SIGMOD, pages 79–94, 2017.[7] J. González, M. Bjørling, S. Lee, C. Dong, and Y. Ronnie Huang. Application-DrivenFlash Translation Layers on Open-Channel SSDs. In NVMW, 2016.[8] J. Gray. Disk Architecture Directions: Put Everything In the Disk Controller. InNASD, 1998.[9] P. Wang, G. Sun, S. Jiang, J. Ouyang, S. Lin, C. Zhang, and J. Cong. An EfficientDesign and Implementation of LSM-tree Based Key-value Store on Open-channelSSD. In EuroSys, pages 16:1–16:14, 2014.

https://pure.itu.dk/ws/files/84771288/lsm_ocssd.pdf

LSM Management on Computational Storage

Abstract

Similar works

Full text

Available Versions

Crossref

The IT University of Copenhagen's Repository

The IT University of Copenhagen's Repository