959 research outputs found
The End of Slow Networks: It's Time for a Redesign
Next generation high-performance RDMA-capable networks will require a
fundamental rethinking of the design and architecture of modern distributed
DBMSs. These systems are commonly designed and optimized under the assumption
that the network is the bottleneck: the network is slow and "thin", and thus
needs to be avoided as much as possible. Yet this assumption no longer holds
true. With InfiniBand FDR 4x, the bandwidth available to transfer data across
network is in the same ballpark as the bandwidth of one memory channel, and it
increases even further with the most recent EDR standard. Moreover, with the
increasing advances of RDMA, the latency improves similarly fast. In this
paper, we first argue that the "old" distributed database design is not capable
of taking full advantage of the network. Second, we propose architectural
redesigns for OLTP, OLAP and advanced analytical frameworks to take better
advantage of the improved bandwidth, latency and RDMA capabilities. Finally,
for each of the workload categories, we show that remarkable performance
improvements can be achieved
Recommended from our members
Building Distributed Systems with Non-Volatile Main Memories and RDMA Networks
High-performance, byte-addressable non-volatile main memories (NVMMs) allow application developers to combine storage and memory into a single layer. These high-performance storage systems would be especially useful in large-scale data center environments where data is distributed and replicated across multiple servers.Unfortunately, existing approaches of providing remote storage access rest on the assumption that storage is slow, so the cost of the software and protocols is acceptable. Such assumption no longer holds for the fast NVMM. As a result, taking full advantage of NVMMsâ potential will require changes in system software and networking protocol. This thesis focuses on accessing remote NVMM efficiently using remote direct memory access (RDMA) network. RDMA enables a client to directly access memory on a remote machine without involving its local CPU.This thesis first presents Mojim, a system that provides replicated, reliable, and highly-available NVMM as an operating system service. Applications can access data in Mojim using normal load and store instructions while controlling when and how updates propagate to replicas using system calls. Our evaluation shows Mojim adds little overhead to the un-replicated system and provides 0.4x to 2.7x the throughput of the un-replicated system.This thesis then presents Orion, a distributed file system designed from for NVMM and RDMA networks. Traditional distributed file systems are designed for slower hard drives. These slower media incentivizes complex optimizations (e.g., queuing, striping, and batching) around disk accesses. Orion combines file system functions and network operations into a single layer. It provides low latency metadata accesses and outperforms existing distributed file systems by a large margin.Finally, an NVMM application can map files backed by an NVMM file system into its address space, and accesses them using CPU instructions. In this case, RDMA and NVMM file systems introduce duplication of effort on permissions, naming, and address translation. We introduce two changes to the existing RDMA protocol: the file memory region (FileMR) and range based address translation. By eliminating redundant translations, FileMR minimizes the number of translations done at the NIC, reducing the load on the NICâs translation cache and resulting in application performance improvement by 1.8x - 2.0x
Assise: Performance and Availability via NVM Colocation in a Distributed File System
The adoption of very low latency persistent memory modules (PMMs) upends the
long-established model of disaggregated file system access. Instead, by
colocating computation and PMM storage, we can provide applications much higher
I/O performance, sub-second application failover, and strong consistency. To
demonstrate this, we built the Assise distributed file system, based on a
persistent, replicated coherence protocol for managing a set of
server-colocated PMMs as a fast, crash-recoverable cache between applications
and slower disaggregated storage, such as SSDs. Unlike disaggregated file
systems, Assise maximizes locality for all file IO by carrying out IO on
colocated PMM whenever possible and minimizes coherence overhead by maintaining
consistency at IO operation granularity, rather than at fixed block sizes.
We compare Assise to Ceph/Bluestore, NFS, and Octopus on a cluster with Intel
Optane DC PMMs and SSDs for common cloud applications and benchmarks, such as
LevelDB, Postfix, and FileBench. We find that Assise improves write latency up
to 22x, throughput up to 56x, fail-over time up to 103x, and scales up to 6x
better than its counterparts, while providing stronger consistency semantics.
Assise promises to beat the MinuteSort world record by 1.5x
Towards Scalable OLTP Over Fast Networks
Online Transaction Processing (OLTP) underpins real-time data processing in many mission-critical applications, from banking to e-commerce.
These applications typically issue short-duration, latency-sensitive transactions that demand immediate processing.
High-volume applications, such as Alibaba's e-commerce platform, achieve peak transaction rates as high as 70 million transactions per second, exceeding the capacity of a single machine.
Instead, distributed OLTP database management systems (DBMS) are deployed across multiple powerful machines.
Historically, such distributed OLTP DBMSs have been primarily designed to avoid network communication, a paradigm largely unchanged since the 1980s.
However, fast networks challenge the conventional belief that network communication is the main bottleneck.
In particular, emerging network technologies, like Remote Direct Memory Access (RDMA), radically alter how data can be accessed over a network.
RDMA's primitives allow direct access to the memory of a remote machine within an order of magnitude of local memory access.
This development invalidates the notion that network communication is the primary bottleneck.
Given that traditional distributed database systems have been designed with the premise that the network is slow, they cannot efficiently exploit these fast network primitives, which requires us to reconsider how we design distributed OLTP systems.
This thesis focuses on the challenges RDMA presents and its implications on the design of distributed OLTP systems.
First, we examine distributed architectures to understand data access patterns and scalability in modern OLTP systems.
Drawing on these insights, we advocate a distributed storage engine optimized for high-speed networks.
The storage engine serves as the foundation of a database, ensuring efficient data access through three central components: indexes, synchronization primitives, and buffer management (caching).
With the introduction of RDMA, the landscape of data access has undergone a significant transformation.
This requires a comprehensive redesign of the storage engine components to exploit the potential of RDMA and similar high-speed network technologies.
Thus, as the second contribution, we design RDMA-optimized tree-based indexes â especially applicable for disaggregated databases to access remote data efficiently.
We then turn our attention to the unique challenges of RDMA.
One-sided RDMA, one of the network primitives introduced by RDMA, presents a performance advantage in enabling remote memory access while bypassing the remote CPU and the operating system.
This allows the remote CPU to process transactions uninterrupted, with no requirement to be on hand for network communication. However, that way, specialized one-sided RDMA synchronization primitives are required since traditional CPU-driven primitives are bypassed.
We found that existing RDMA one-sided synchronization schemes are unscalable or, even worse, fail to synchronize correctly, leading to hard-to-detect data corruption.
As our third contribution, we address this issue by offering guidelines to build scalable and correct one-sided RDMA synchronization primitives.
Finally, recognizing that maintaining all data in memory becomes economically unattractive, we propose a distributed buffer manager design that efficiently utilizes cost-effective NVMe flash storage.
By leveraging low-latency RDMA messages, our buffer manager provides a transparent memory abstraction, accessing the aggregated DRAM and NVMe storage across nodes.
Central to our approach is a distributed caching protocol that dynamically caches data.
With this approach, our system can outperform RDMA-enabled in-memory distributed databases while managing larger-than-memory datasets efficiently
SABRes: Atomic Object Reads for In-Memory Rack-Scale Computing
Modern in-memory services rely on large distributed object stores to achieve the high scalability essential to service thousands of requests concurrently. The independent and unpredictable nature of incoming requests results in random accesses to the object store, triggering frequent remote memory accesses. State-of-the-art distributed memory frameworks leverage the one-sided operations offered by RDMA technology to mitigate the traditionally high cost of remote memory access. Unfortunately, the limited semantics of RDMA one-sided operations bound remote memory access atomicity to a single cache block; therefore, atomic remote object access relies on software mechanisms. Emerging highly integrated rack-scale systems that reduce the latency of one-sided operations to a small multiple of DRAM latency expose the overhead of these software mechanisms as a major latency contributor. This technology-triggered paradigm shift calls for new one-sided operations with stronger semantics. We take a step in that direction by proposing SABRes, a new one-sided operation that provides atomic remote object reads in hardware. We then present LightSABRes, a lightweight hardware accelerator for SABRes that removes all atomicity-associated software overheads. Compared to a state-of-the-art software atomicity mechanism, LightSABRes improve the throughput of a microbenchmark atomically accessing 128B-8KB objects from remote memory by 15-97%, and the throughput of a modern in-memory distributed object store by 30-60%
Aspects of Code Generation and Data Transfer Techniques for Modern Parallel Architectures
Im Bereich der Prozessorarchitekturen hat sich der Fokus neuer Entwicklungen von immer höheren Taktfrequenzen hin zu immer mehr Kernen auf einem Chip verschoben.
Eine hohe Kernanzahl ermöglicht es unterschiedlich leistungsfÀhige Kerne anzubieten, und sogar dedizierte Kerne mit speziellen BefehlssÀtzen.
Die Entwicklung fĂŒr solch heterogene Plattformen ist herausfordernd und benötigt entsprechende UnterstĂŒtzung von Entwicklungswerkzeugen, wie beispielsweise Ăbersetzern.
Neben ihrer heterogenen Kernstruktur gibt es eine zweite Dimension, die die Entwicklung fĂŒr solche Architekturen anspruchsvoll macht: ihre Speicherstruktur.
Die Aufrechterhaltung von globaler Cache-KohÀrenz erschwert das Erreichen hoher Kernzahlen.
Hardwarebasierte Cache-KohĂ€renz-Protokolle skalieren entweder schlecht, oder sind kompliziert und fĂŒhren zu Problemen bei AusfĂŒhrungszeit und Energieeffizienz.
Eine radikale Lösung dieses Problems stellt die Abschaffung der globalen Cache-KohÀrenz dar.
Jedoch ist es schwierig, bestehende Programmiermodelle effizient auf solch eine Hardware-Architektur mit schwachen Garantien abzubilden.
Der erste Teil dieser Dissertation beschĂ€ftigt sich Datentransfertechniken fĂŒr nicht-cache-kohĂ€rente Architekturen mit gemeinsamem Speicher.
Diese Architekturen bieten einen gemeinsamen physikalischen Adressraum, implementieren aber keine hardwarebasierte KohÀrenz zwischen allen Caches des Systems.
Die logische Partitionierung des gemeinsamen Speichers ermöglicht die sichere Programmierung einer solchen Plattform.
Im Allgemeinen erzeugt dies die Notwendigkeit Daten zwischen Speicherpartitionen zu kopieren.
Wir untersuchen die Ăbersetzung fĂŒr invasive Architekturen, einer Familie von nicht-cache-kohĂ€renten Vielkernarchitekturen.
Wir betrachten die effiziente Implementierung von Datentransfers sowohl einfacher als auch komplexer Datenstrukturen auf invasiven Architekturen.
Insbesondere schlagen wir eine neuartige Technik zum Kopieren komplexer verzeigerter Datenstrukturen vor, die ohne Serialisierung auskommt.
Hierzu verallgemeinern wir den Objekt-Klon-Ansatz mit ĂŒbersetzergesteuerter automatischer software-basierter KohĂ€renz, sodass er auch im Kontext nicht-kohĂ€renter Caches funktioniert.
Wir prĂ€sentieren Implementierungen mehrerer Datentransfertechniken im Rahmen eines existierenden Ăbersetzers und seines Laufzeitsystems.
Wir fĂŒhren eine ausfĂŒhrliche Auswertung dieser Implementierungen auf einem FPGA-basierten Prototypen einer invasiven Architektur durch.
SchlieĂlich schlagen wir vor, HardwareunterstĂŒtzung fĂŒr bereichsbasierte Cache-Operationen hinzuzufĂŒgen und beschreiben und bewerten mögliche Implementierungen und deren Kosten.
Der zweite Teil dieser Dissertation befasst sich mit der Beschleunigung von Shuffle-Code, der bei der Registerzuteilung auftritt, durch die Verwendung von Permutationsbefehlen.
Die Aufgabe der Registerzuteilung wĂ€hrend der ProgrammĂŒbersetzung ist die Abbildung von Programmvariablen auf Maschinenregister.
WĂ€hrend der Registerzuteilung erzeugt der Ăbersetzer Shuffle-Code, der aus Kopier- und Tauschbefehlen besteht, um Werte zwischen Registern zu transferieren.
AbhĂ€ngig von der QualitĂ€t der Registerzuteilung und der Zahl der verfĂŒgbaren Register kann eine groĂe Menge an Shuffle-Code erzeugt werden.
Wir schlagen vor, die AusfĂŒhrung von Shuffle-Code mit Hilfe von neuartigen Permutationsbefehlen zu beschleunigen, die die Inhalte von einigen Registern in einem Taktzyklus beliebig permutieren.
Um die Machbarkeit dieser Idee zu demonstrieren, erweitern wir zunÀchst ein bestehendes RISC-Befehlsformat um Permutationsbefehle.
AnschlieĂend beschreiben wir, wie die vorgeschlagenen Permutationsbefehle in einer bestehenden RISC-Architektur implementiert werden können.
Dann entwickeln wir zwei Verfahren zur Codeerzeugung, die die Permutationsbefehle ausnutzen, um Shuffle-Code zu beschleunigen: eine schnelle Heuristik und einen auf dynamischer Programmierung basierenden optimalen Ansatz.
Wir beweisen QualitÀts- und Korrektheitseingeschaften beider AnsÀtze und zeigen die OptimalitÀt des zweiten Ansatzes.
Im Folgenden implementieren wir beide Codeerzeugungsverfahren in einem Ăbersetzer und untersuchen sowie vergleichen deren CodequalitĂ€t ausfĂŒhrlich mit Hilfe standardisierter Benchmarks.
ZunĂ€chst messen wir die genaue Zahl der dynamisch ausgefĂŒhrten Befehle, welche wir folgend validieren, indem wir Programmlaufzeiten auf einer FPGA-basierten Prototypimplementierung der um Permutationsbefehle erweiterten RISC-Architektur messen.
SchlieĂlich argumentieren wir, dass Permutationsbefehle auf modernen Out-Of-Order-Prozessorarchitekturen, die bereits Registerumbenennung unterstĂŒtzen, mit wenig Aufwand implementierbar sind
- âŠ