A Generic Undo Support for State-Based CRDTs

CRDTs (Conflict-free Replicated Data Types) have properties desirable for large-scale distributed systems with variable network latency or transient partitions. With CRDT, data are always available for local updates and data states converge when the replicas have incorporated the same updates. Undo is useful for correcting human mistakes and for restoring system-wide invariant violated due to long delays or network partitions. There is currently no generally applicable undo support for CRDTs. There are at least two reasons for this. First, there is currently no abstraction that we can practically use to capture the relations between undo and normal operations with respect to concurrency and causality. Second, using inverse operations as the existing partial solutions, the CRDT designer has to hard-code certain rules and design a new CRDT for almost every operation that needs undo support. In this paper, we present an approach to generic support of undo for CRDTs. The approach consists of two major parts. We first work out an abstraction that captures the semantics of concurrent undo and redo operations through equivalence classes. The abstraction is a natural extension of undo and redo in sequential applications and is straightforward to implement in practice. By using this abstraction, we then device a mechanism to augment existing CRDTs. The mechanism provides an "out of the box" support for undo without the involvement of the CRDT designers. We also present a practical application of the approach in collaborative editing

The Only Undoable CRDTs are Counters

In comparing well-known CRDTs representing sets that can grow and shrink, we find caveats. In one, the removal of an element cannot be reliably undone. In another, undesirable states are attainable, such as when an element is present -1 times (and so must be added for the set to become empty). The first lacks a general-purpose undo, while the second acts less like a set and more like a tuple of counters, one per possible element. Using some group theory, we show that this trade-off is unavoidable: every undoable CRDT is a tuple of counters

Building Tunable CRDTs

Nowadays multiple large-scale services are hosted on the Internet, many of them with millions of daily users. These systems need to scale efficiently, providing fast access and being always available, despite failures of the servers or of the network and high amounts of users accessing the service. As such, these services typically trade strong consistency for high availability and low latency. However, not having strong consistency implies that conflicts arising from concurrent updates will occur, which need to be solved. Conflict-Free Replicated Data Types (CRDTs) provide low latency and solve conflicts automatically, ensuring eventual state convergence. However, one shortcoming of CRDTs is the way they deal with concurrency conflicts – usually they solve them automatically by applying a specific policy. For example, in a set, e priority to the add. These policies are limited and not adequate for all applications, especially since CRDTs don’t allow for policies dependent on the application context, such as, give priority to add if it’s element e, but give priority to remove if it’s element f. As such, in this thesis we propose a new type of CRDTs, called tunable CRDT (t- CRDT), which allows the programmer to specify for each operation what is the desired conflict solving policy, by supplying a simple boolean function. The programmer can either supply his own policy or use one of the many we provide in our t-CRDT library. T-CRDTs solve conflicts automatically by applying the policies in each operation. This new type of CRDTs adapt more easily to each application specific needs, as it gives more control to the programmer while still having the main properties of CRDTs, i.e., eventual convergence of state and low latency. With this, it is expected that more applications can start using CRDTs as their data solution and benefit from their properties

Key-CRDT stores

Dissertação para obtenção do Grau de Mestre em Engenharia InformáticaThe Internet has opened opportunities to create world scale services. These systems require highavailability and fault tolerance, while preserving low latency. Replication is a widely adopted technique to provide these properties. Different replication techniques have been proposed through the years, but to support these properties for world scale services it is necessary to trade consistency for availability, fault-tolerance and low latency. In weak consistency models, it is necessary to deal with possible conflicts arising from concurrent updates. We propose the use of conflict free replicated data types (CRDTs) to address this issue. Cloud computing systems support world scale services, often relying on Key-Value stores for storing data. These systems partition and replicate data over multiple nodes, that can be geographically disperse over the network. For handling conflict, these systems either rely on solutions that lose updates (e.g. last-write-wins) or require application to handle concurrent updates. Additionally, these systems provide little support for transactions, a widely used abstraction for data access. In this dissertation, we present the design and implementation of SwiftCloud, a Key-CRDT store that extends a Key-Value store by incorporating CRDTs in the system’s data-model. The system provides automatic conflict resolution relying on properties of CRDTs. We also present a version of SwiftCloud that supports transactions. Unlike traditional transactional systems, transactions never abort due to write/write conflicts, as the system leverages CRDT properties to merge concurrent transactions. For implementing SwiftCloud, we have introduced a set of new techniques, including versioned CRDTs, composition of CRDTs and alternative serialization methods. The evaluation of the system, with both micro-benchmarks and the TPC-W benchmark, shows that SwiftCloud imposes little overhead over a key-value store. Allowing clients to access a datacenter close to them with SwiftCloud, can reduce latency without requiring any complex reconciliation mechanism. The experience of using SwiftCloud has shown that adapting an existing application to use SwiftCloud requires low effort.Project PTDC/EIA-EIA/108963/200

Delta state replicated data types

Conflict-free Replicated Data Types (CRDTs) are distributed data types that make eventual consistency of a distributed object possible and non ad-hoc. Specifically, state-based CRDTs ensure convergence through disseminating the entire state, that may be large, and merging it to other replicas. We introduce Delta State Conflict-Free Replicated Data Types (delta-CRDT) that can achieve the best of both operation-based and state-based CRDTs: small messages with an incremental nature, as in operation-based CRDTs, disseminated over unreliable communication channels, as in traditional state-based CRDTs. This is achieved by defining delta-mutators to return a delta-state, typically with a much smaller size than the full state, that to be joined with both local and remote states. We introduce the delta-CRDT framework, and we explain it through establishing a correspondence to current state-based CRDTs. In addition, we present an anti-entropy algorithm for eventual convergence, and another one that ensures causal consistency. Finally, we introduce several delta-CRDT specifications of both well-known replicated datatypes and novel datatypes, including a generic map composition. (C) 2017 Elsevier Inc. All rights reserved.The work presented was partially supported by EU FP7 SyncFree project (609551), EU H2020 LightKone project (732505), and SMILES line in project TEC4Growth (NORTE-01-0145-FEDER-000020)

Update Consistency for Wait-free Concurrent Objects

In large scale systems such as the Internet, replicating data is an essential feature in order to provide availability and fault-tolerance. Attiya and Welch proved that using strong consistency criteria such as atomicity is costly as each operation may need an execution time linear with the latency of the communication network. Weaker consistency criteria like causal consistency and PRAM consistency do not ensure convergence. The different replicas are not guaranteed to converge towards a unique state. Eventual consistency guarantees that all replicas eventually converge when the participants stop updating. However, it fails to fully specify the semantics of the operations on shared objects and requires additional non-intuitive and error-prone distributed specification techniques. This paper introduces and formalizes a new consistency criterion, called update consistency, that requires the state of a replicated object to be consistent with a linearization of all the updates. In other words, whereas atomicity imposes a linearization of all of the operations, this criterion imposes this only on updates. Consequently some read operations may return out-dated values. Update consistency is stronger than eventual consistency, so we can replace eventually consistent objects with update consistent ones in any program. Finally, we prove that update consistency is universal, in the sense that any object can be implemented under this criterion in a distributed system where any number of nodes may crash.Comment: appears in International Parallel and Distributed Processing Symposium, May 2015, Hyderabad, Indi

Conflict-Free Replicated Relations for Multi-Synchronous Database Management at Edge

International audienceIn a cloud-edge environment, edge devices may not always be connected to the network. Still, applications may need to access the data on edge devices even when they are not connected. With support for multi-synchronous access, data on an edge device are kept synchronous with the data in the cloud as long as the device is online. When the device is off-line, the application can still access the data on the device, asynchronously with concurrent data updates either in the cloud or on other edge devices. Conflict-free Replicated Data Types (CRDTs) emerged as a technology for multi-synchronous data access. CRDTs guarantee that when all sites have applied the same set of updates, the replicated data converge. However, CRDTs have not been successfully applied to relational databases (RDBs) for multi-synchronous access. In this paper, we present Conflict-free Replicated Relations (CRRs) that apply CRDTs to RDBs for support of multi-synchronous data access. With CRR, existing RDB applications, with very little modification, can be enhanced with multi-synchronous access. We also present a prototype implementation of CRR with some preliminary performance results