Practical View-Change-Less Protocol through Rapid View Synchronization

The emergence of blockchain technology has renewed the interest in consensus-based data management systems that are resilient to failures. To maximize throughput of these systems, we have recently seen several prototype consensus solutions that optimize for throughput at the expense of overall implementation complexity, high costs, and reliability. Due to this, it remains unclear how these prototypes will perform in real-world environments. In this paper, we present the Practical View-Change-Less Protocol PVP, a high-throughput, simple, and reliable consensus protocol. Central to PVP is the combination of (1) a chained consensus design for replicating requests with a reduced message cost; (2) the novel Rapid View Synchronization protocol that enables robust and low-cost failure recovery; and (3) a high-performance concurrent consensus architecture in which independent instances of the chained consensus operate concurrently to process requests with high throughput and without single-replica bottlenecks. Due to the concurrent consensus architecture, PVP greatly outperforms traditional primary-backup consensus protocols such as PBFT (by up to 430%), Narwhal (by up to 296%), and HotStuff (by up to 3803%). Due to its reduced message cost, PVP is even able to outperform RCC, a state-of-the-art high-throughput concurrent consensus protocol, by up to 23%. Furthermore, PVP is able to maintain a stable and low latency and consistently high throughput even during failures.Comment: 16 pages, 14 figure

Consensus in high performance computing

Dissertação de mestrado integrado em Informatics EngineeringFor a long time, developers and scientists designed consensus solutions that sacrificed thread scalability or load balance to decrease the latency. The appearance of networking and memory with microseconds requests reshaped how high-throughput consensus solutions are designed today. Today’s protocols must be multi-threaded to scale and seize the high hardware-level parallelism available. Moreover, their focus should be on not overloading the system instead of decreasing the latency. In the last decade, storage devices underwent an enormous development in their performance. The NVMe devices bought the request’s latency of solid-state devices to the microseconds range. Additionally, the software available to manage and operate these devices kept up with the hardware development. Libraries such as the Storage Performance Tool Kit (SPDK) appeared to ease the development of high-performance storage applications. In this context, this work aimed to answer a fundamental question: can NVMe Devices and SPDK improve the existing consensus-related work? This dissertation describes a solution for the distributed consensus problem that combines Disk Paxos (a consensus algorithm that relies upon writing and reading in a network of storage devices to achieve a distributed agreement) with NVMe Devices and SPDK. We conceived this solution using C++ and conducted a performance evaluation that, in the end, compared our solution with LibPaxos. Our findings describe the issues and benefits of the usage of these technologies to solve consensus. With our approach, we increase the understanding of the potential of these new technologies in enabling better solutions for the consensus problem in the future.Durante bastante tempo, as abordagens adotadas para resolver o consenso distribuído sacrificavam a escalabilidade do número de “threads” ou o balanceamento da carga com o objetivo de diminuir a latência da solução. O aparecimento de redes e latências de memória, com pedidos na ordem dos microssegundos, alterou a forma como os algoritmos de alto desempenho são desenhos, hoje em dia. Os protocolos atuais têm de ser “multithreaded” para conseguirem escalar e aproveitar o elevado paralelismo existente do hardware disponível. Para além disso, o foco dos protocolos deve residir em evitar a sobrecarga do sistema e não na redução da latência. Na última década, houve um grande desenvolvimento nos dispositivos de armazenamento. Os discos “NVMe” introduziram latências na gama dos microssegundos aumentando significativamente o desempenho dos discos de armazenamento. Adicionalmente, o “software” disponível, para gerir e operar sobre estes dispositivos, também acompanhou o seu desenvolvimento. Bibliotecas, como o “Storage Performance Tool Kit (SPDK)”, apareceram para facilitar o desenvolvimento de aplicações de armazenamento de alto desempenho. Neste contexto, este trabalho teve, como principal objetivo, responder à seguinte questão: Podem os dispositivos NVMe e o SPDK melhorar o trabalho existente sobre o consenso distribuído? Esta dissertação descreve uma solução para o consenso distribuído que combina o Disk Paxos, um algoritmo que resolve o consenso distribuído, através de leituras e escritas numa rede partilhada de discos, com os dispositivos NVMe e a biblioteca SPDK. Esta solução foi desenvolvida em C++ e foi realizada uma análise de desempenho que, na sua fase final, compara a mesma com outra solução existente, o LibPaxos. A análise desenvolvida descreve os problemas e os benefícios de usar estas tecnologias para resolver o problema de consenso distribuído. Com esta abordagem, foi possível explorar e descobrir o potencial que as tecnologias usadas têm para possibilitar o desenvolvimento de melhores soluções no futuro

Compositional verification using interface recognizers/suppliers (IRS)

Thèse numérisée par la Direction des bibliothèques de l'Université de Montréal

Hermes: a Fast, Fault-Tolerant and Linearizable Replication Protocol

Today's datacenter applications are underpinned by datastores that are responsible for providing availability, consistency, and performance. For high availability in the presence of failures, these datastores replicate data across several nodes. This is accomplished with the help of a reliable replication protocol that is responsible for maintaining the replicas strongly-consistent even when faults occur. Strong consistency is preferred to weaker consistency models that cannot guarantee an intuitive behavior for the clients. Furthermore, to accommodate high demand at real-time latencies, datastores must deliver high throughput and low latency. This work introduces Hermes, a broadcast-based reliable replication protocol for in-memory datastores that provides both high throughput and low latency by enabling local reads and fully-concurrent fast writes at all replicas. Hermes couples logical timestamps with cache-coherence-inspired invalidations to guarantee linearizability, avoid write serialization at a centralized ordering point, resolve write conflicts locally at each replica (hence ensuring that writes never abort) and provide fault-tolerance via replayable writes. Our implementation of Hermes over an RDMA-enabled reliable datastore with five replicas shows that Hermes consistently achieves higher throughput than state-of-the-art RDMA-based reliable protocols (ZAB and CRAQ) across all write ratios while also significantly reducing tail latency. At 5% writes, the tail latency of Hermes is 3.6X lower than that of CRAQ and ZAB.Comment: Accepted in ASPLOS 202