Building Regular Registers with Rational Malicious Servers and Anonymous Clients

The paper addresses the problem of emulating a regular register in a synchronous distributed system where clients invoking $$\mathsf{read}()$$ and $$\mathsf{write}()$$ operations are anonymous while server processes maintaining the state of the register may be compromised by rational adversaries (i.e., a server might behave as rational malicious Byzantine process). We first model our problem as a Bayesian game between a client and a rational malicious server where the equilibrium depends on the decisions of the malicious server (behave correctly and not be detected by clients vs returning a wrong register value to clients with the risk of being detected and then excluded by the computation). We prove such equilibrium exists and finally we design a protocol implementing the regular register that forces the rational malicious server to behave correctly

A Demand Based Load Balanced Service Replication Model

Cloud computing allows service users and providers to access the applications, logical resources and files on any computer with ease. A cloud service has three distinct characteristics that differentiate it from traditional hosting. It is sold on demand, typically by the minute or the hour; it is elastic. It is a way to increase capacity or add capabilities on the fly without investing in new infrastructure, training new personnel, or licensing new software. It not only promises reliable services delivered through next-generation data centers that are built on compute and storage virtualization technologies but also addresses the key issues such as scalability, reliability, fault tolerance and file load balancing. The one way to achieve this is through service replication across different machines coupled with load balancing. Though replication potentially improves fault tolerance, it leads to the problem of ensuring consistency of replicas when certain service is updated or modified. However, fewer replicas also decrease concurrency and the level of service availability. A balanced synchronization between replication mechanism and consistency not only ensures highly reliable and fault tolerant system but also improves system performance significantly. This paper presents a load balancing based service replication model that creates a replica on other servers on the basis of number of service accesses. The simulation results indicate that the proposed model reduces the number of messages exchanged for service replication by 25-55% thus improving the overall system performance significantly. Also in case of CPU load based file replication, it is observed that file access time reduces by 5.56%-7.65%

A decentralized framework for cross administrative domain data sharing

Federation of messaging and storage platforms located in remote datacenters is an essential functionality to share data among geographically distributed platforms. When systems are administered by the same owner data replication reduces data access latency bringing data closer to applications and enables fault tolerance to face disaster recovery of an entire location. When storage platforms are administered by different owners data replication across different administrative domains is essential for enterprise application data integration. Contents and services managed by different software platforms need to be integrated to provide richer contents and services. Clients may need to share subsets of data in order to enable collaborative analysis and service integration. Platforms usually include proprietary federation functionalities and specific APIs to let external software and platforms access their internal data. These different techniques may not be applicable to all environments and networks due to security and technological restrictions. Moreover the federation of dispersed nodes under a decentralized administration scheme is still a research issue. This thesis is a contribution along this research direction as it introduces and describes a framework, called \u201cWideGroups\u201d, directed towards the creation and the management of an automatic federation and integration of widely dispersed platform nodes. It is based on groups to exchange messages among distributed applications located in different remote datacenters. Groups are created and managed using client side programmatic configuration without touching servers. WideGroups enables the extension of the software platform services to nodes belonging to different administrative domains in a wide area network environment. It lets different nodes form ad-hoc overlay networks on-the-fly depending on message destinations located in distinct administrative domains. It supports multiple dynamic overlay networks based on message groups, dynamic discovery of nodes and automatic setup of overlay networks among nodes with no server-side configuration. I designed and implemented platform connectors to integrate the framework as the federation module of Message Oriented Middleware and Key Value Store platforms, which are among the most widespread paradigms supporting data sharing in distributed systems

Dependable data storage with state machine replication

Tese de mestrado em Informática, Universidade de Lisboa, Faculdade de Ciências, 2014State Machine Replication (SMR) is a technique to replicate information across servers, also called replicas, providing fault tolerance to services. Instead of execute in a single server, requests from multiple clients are ordered and executed in a set of replicas. Results are confirmed to the clients once a predefined quorum of replicas replies. Several studies prove possible to tolerate up to f faults using 2f + 1 replicas. Byzantine Fault Tolerant (BFT) SMR configurations, where replicas can behave in an arbitrary mode, require f additional replicas, with the total of 3f + 1 replicas. When a replica is detected faulty, it has to be recovered with an updated state to reduce the vulnerability of the system. This state is generated during the service execution, when write operations are logged. To bind the size of the log and the time to replay it, periodic snapshots of the service state, or checkpoints, are taken and the log reset. During recovery the checkpoint and the log are transferred from other replicas. To provide resilience to co-related faults, information has to be logged in durable storage. Synchronous writes in durable storage and constant checkpoints can affect throughput and latency of the system as replicas have to wait for information to be stored before reply. When a checkpoint is being taken the system cannot make progress because the state cannot be changed. This may cause the service to be interrupted for several seconds during a checkpoint. The state transfer to a recovering replica can also cause perturbations in the system execution, as correct replicas has to read and transfer the state, composed by the checkpoint, log and digests of messages in case of BFT systems. In this dissertation we present three techniques to improve the performance of state storage and transfer in a BFT SMR protocol - BFT-SMART. The first, Parallel Logging stores information in the log in parallel with its execution by the application. The second, Sequential Checkpointing makes only one replica take a checkpoint at a time, in a round-robin fashion, allowing the system to make progress during that period. The last technique, Collaborative State Transfer (CST) reduces the perturbation in a system during state transfer to a recovering replica by having one replica providing the checkpoint and the remaining providing portions of the log. We also present algorithms that address the problem of co-related failures. When several replicas fail at the same time it is possible to start them simultaneously and compare the stored state before having the service available again. After presenting the techniques, we provide a prototype implementation called Dura-SMaRt with an evaluation against BFT-SMART to compare the efficiency of the new techniques. We performed the evaluation with two applications: a consistent key-value store – SCKV-store – and a coordination service that stores information in tuple spaces – DepSpace. Next, we evaluate Dura-SMaRt in a complex use, having a database replication middleware built on top of it. SteelDB, provide fault tolerance for transaction processing in database management systems (DBMS). Transactional databases provide durability for information systems executing operations inside boundaries called transactions. Transactions guarantee several properties, amongst which, atomicity and isolation. Atomicity enforces that all operations executed inside a transaction are confirmed, or none is. Isolation guarantees that operations inside a transaction are only visible for other transactions after it is finished. Concurrency mechanisms implementations allow several transactions, from several clients to be executed at the same time, improving the performance of a DBMS. To provide dependability to DBMS, several DBMS vendors provide replications mechanisms that usually rely on the efficiency of fail detection and recovery. Such replication mechanisms are also attached to the vendor implementation. With SteelDB we provide transparent Byzantine fault tolerance with 3f + 1 replicated databases. SteelDB requires no changes in the client code as it provides a driver implementation of the JDBC specification. Clients have only to switch the current driver provided by the database vendor it is using to the driver provided by SteelDB. After describing the concepts and implementation of SteelDB we present an evaluation performed on SteelDB during the last year of the FP7 TClouds project. We evaluated SteelDB for functional and performance aspects with a real application executing different types of transactions and comparing results with executions on different environments. We compared SteelDB executions in local area networks, private, public and hybrid clouds discussing the differences in performance and efficiency of optimizations present in the middleware. After SteelDB evaluation we discuss the related work to state management in SMR and database replication middlewares. Finally we will conclude the work with a discussion on the results obtained and purposes for future work.Replicação de Máquina de Estados (SMR) é uma técnica para replicar informações entre vários servidores, também chamados de réplicas, provendo tolerância a faltas para aplicações. Ao invés de executar os pedidos dos clientes em um único servidor, pedidos de vários clientes que alteram o estado de uma aplicação passam por um protocolo de ordenação e são entregues na mesma ordem para um conjunto de réplicas. Os resultados somente são confirmados aos clientes após um quórum pré-definido de réplicas responder. Vários estudos provaram ser possível tolerar até f faltas com o uso de 2f + 1 réplicas. Configurações para SMR com Tolerância a Faltas Bizantinas (BFT), onde réplicas podem apresentar comportamento arbitrário, necessitam de f réplicas adicionais, com o total de 3f + 1 réplicas. Quando uma réplica percebe que esta atrasada em relação às demais, ou uma nova réplica é adicionada ao sistema, ela precisa instalar uma a versão atualizada do estado, para poder participar do protocolo de ordenação e processamento dos pedidos, restaurando assim a tolerância do sistema a faltas. Réplicas geram um log das operações executadas para terem uma cópia atualizada do estado, necessária a uma possível recuperação. As operações de escrita são armazenadas de forma sequencial no log. Para limitar seu tamanho e o tempo para reproduzí-lo em uma réplica que está recuperar-se, as réplicas tiram cópias do estado periodicamente em checkpoints e, apagam o log em seguida. Durante a recuperação de uma réplica, o checkpoint e o log são transferidos pelas demais. A réplica que está a recuperar-se instala o checkpoint recebido e executa as operações do log antes de confirmar às demais que está pronta a processar pedidos novamente. Para oferecer tolerância a faltas co-relacionadas, onde várias réplicas podem apresentar falhas ao mesmo tempo, informações precisam ser armazenadas em mídia durável. Escritas síncronas em mídia durável e checkpoints constantes podem diminuir o throughput e aumentar a latência do sistema pois as réplicas precisam esperar até que a escrita seja concluída, antes de confirmar a operação ao cliente. De outra forma, no caso de uma falha antes do fim da escrita, poderíamos ter dados confirmados ao cliente mas não armazenados. Realizamos experimentos que provam que a substituição da mídia por opções mais rápidas, nomeadamente, disco rígido por SSD, apesar de diminuir o tempo de escrita ainda afeta consideravelmente o throughput da aplicação. Enquanto um checkpoint do estado é gerado, a aplicação não pode estar a processar operações de escrita, pois estas podem alterar este estado. Isto faz com que o throughput do sistema seja zero durante este período, que pode demorar vários segundos, dependendo do tamanho do estado. Conforme demonstramos através de gráficos de desempenho da aplicação, a transferência de estado a uma réplica que está a recuperar-se pode também causar perturbações nas réplicas que estão a transferí-lo, pois estas precisam ler dados em mídia durável e transferir o estado pela rede. Em situações onde o tamanho do estado é grande, a transferência pode afectar a comunicação com as demais réplicas e com os clientes. Apresentamos neste trabalho três técnicas puramente algorítmicas que melhoram o desempenho no armazenamento e transferência de estado em um protocolo BFT SMR chamado BFT-SMART. A primeira, Parallel Logging, faz as réplicas armazenarem as operações no log em paralelo com sua execução¸ ao pela aplicação. Em aplicações onde o tempo para se executar uma operação é considerável, pode-se reduzir o tempo total ao executar a operação e o log em threads diferentes. A segunda, Sequential Checkpointing faz somente uma das réplicas tirar um checkpoint por vez, sequencialmente, permitindo ao sistema fazer progresso nesse período. A terceira técnica, Collaborative State Transfer (CST) define uma estratégia para transferência de estado onde uma réplica envia o checkpoint da aplicação e as demais enviam partes do log, reduzindo o efeito da transferência de estado nas réplicas que estão a enviá-lo. Apresentamos também algoritmos para resolver o problema de faltas co-relacionadas. No caso de uma falta onde todas as réplicas vão abaixo, é possível fazê-las retomar o serviço e iniciar a execução¸ ao novamente, após iniciadas. Implementamos as novas técnicas apresentadas em um protótipo chamado Dura-SMaRt para obtermos uma avaliação de seu efeito no desempenho de um sistema replicado. Apresentamos uma avaliação do protótipo e do BFT-SMART com duas aplicações diferentes construídas sobre estes, uma consistent key-value store chamada SCKV-Store e um serviço de coordenação que utiliza um espaço de tuplos para armazenamento de dados chamado DepSpace. Comparamos os resultados de diversos experimentos para demonstrar que as novas técnicas reduzem o impacto da escrita de operações em mídia durável. Apresentamos resultados que mostram que a execução das operações de escrita em paralelo com seu armazenamento no log não afectam o throughput em para aplicações onde o tempo de execução de mensagens é considerável. As novas técnicas também reduzem o impacto que a geração de um checkpoint tem no throughput do sistema. Por fim demonstramos que a transferência de estado tem menor impacto no throughput do sistema com as novas técnicas quando comparadas ao modelo anterior onde uma réplica era responsável por enviar o checkpoint e o log das operações. De seguida, avaliamos o Dura-SMaRt em um caso de uso complexo: um middleware para replicação de bases de dados chamado SteelDB. Este middleware utilizou o Dura-SMaRt para replicação de dados, oferecendo tolerˆancia a faltas para transações em sistemas de gerenciamento de bases de dados (DBMS). Bases de dados transacionais fornecem durabilidade para sistemas de informação ao executar operações dentro de barreiras chamadas transações. Uma transação garante algumas propriedades, entre as quais atomicidade e isolamento. Atomicidade implica que todas as operações executadas são confirmadas, ou nenhuma é. Isolamento garante que alterações presentes dentro de uma transação só serão visíveis às demais após o fim desta. Estas propriedades permitem a utilização da base de dados simultaneamente por vários clientes, aumentando a concorrência na execução de operações. Para aumentar a disponibilidade e recuperação a faltas, vários desenvolvedores de DBMS fornecem mecanismos de replicação de dados. Estes mecanismos geralmente estão ligados a eficiência dos sistemas de deteccão de falha e recuperação. Eles também estão intrinsecamente ligados ao fabricante da base de dados escolhido. Com o SteelDB n´os oferecemos tolerância transparente a faltas Byzantinas, com o uso de 3f + 1 bases de dados. O SteelDB fornece aos clientes uma implementação da especificação JDBC, portanto, clientes que já utilizam um driver JDBC para aceder a uma base de dados, somente precisam trocá-lo pelo driver fornecido pelo SteelDB. Depois de descrever os conceitos e implementação do middleware SteelDB, apresentamos uma avaliação deste, realizada no último ano do projeto FP7 TClouds. Esta avaliação testou diversos aspectos de desempenho e funcionalidade em uma aplicação real com diversos tipos de transações, fornecida por um dos parceiros do projeto. Descrevemos a configuração e execução do SteelDB em diversos ambientes como redes locais, clouds privadas, públicas e híbridas. Comparamos de seguida os resultados da execução nestes diferentes ambientes para avaliar a eficiência de optimizações incluídas no middleware. Apesar da utilização de bases locais ter desempenho consideravelmente melhor com relação à replicação com o SteelDB, bases locais não fornecem tolerância a faltas. Também demonstramos que quando o tamanho das transações aumenta, a diferença entre os tempos de execução diminui, evidenciando o custo da troca de mensagens entre redes remotas. Otimizações incluídas no SteelDB, entretanto, diminuem o número de mensagens necessárias por operação, reduzindo também o seu tempo de execução total. Avaliamos também o desempenho do SteelDB em simulações com diferentes tipos de faltas. Nos casos de teste que avaliamos, as faltas não afectam consideravelmente o desempenho do SteelDB, uma vez que o protocolo de replicação Dura-SMaRt não precisa esperar por respostas de todas as réplicas antes de confirmar as operaçõees aos clientes. Após apresentarmos a avaliação do SteelDB, discutimos os trabalhos relacionados com o gerenciamento de estado em sistemas SMR e também estudos e alternativas para replicação de bases de dados com o uso de SMR. Concluímos com uma discussão dos resulados obtidos e propostas de trabalhos futuros

CERN openlab Whitepaper on Future IT Challenges in Scientific Research

This whitepaper describes the major IT challenges in scientific research at CERN and several other European and international research laboratories and projects. Each challenge is exemplified through a set of concrete use cases drawn from the requirements of large-scale scientific programs. The paper is based on contributions from many researchers and IT experts of the participating laboratories and also input from the existing CERN openlab industrial sponsors. The views expressed in this document are those of the individual contributors and do not necessarily reflect the view of their organisations and/or affiliates

Building global and scalable systems with atomic multicast

The rise of worldwide Internet-scale services demands large distributed systems. Indeed, when handling several millions of users, it is common to operate thousands of servers spread across the globe. Here, replication plays a central role, as it contributes to improve the user experience by hiding failures and by providing acceptable latency. In this thesis, we claim that atomic multicast, with strong and well-defined properties, is the appropriate abstraction to efficiently design and implement globally scalable distributed systems. Internet-scale services rely on data partitioning and replication to provide scalable performance and high availability. Moreover, to reduce user-perceived response times and tolerate disasters (i.e., the failure of a whole datacenter), services are increasingly becoming geographically distributed. Data partitioning and replication, combined with local and geographical distribution, introduce daunting challenges, including the need to carefully order requests among replicas and partitions. One way to tackle this problem is to use group communication primitives that encapsulate order requirements. While replication is a common technique used to design such reliable distributed systems, to cope with the requirements of modern cloud based ``always-on'' applications, replication protocols must additionally allow for throughput scalability and dynamic reconfiguration, that is, on-demand replacement or provisioning of system resources. We propose a dynamic atomic multicast protocol which fulfills these requirements. It allows to dynamically add and remove resources to an online replicated state machine and to recover crashed processes. Major efforts have been spent in recent years to improve the performance, scalability and reliability of distributed systems. In order to hide the complexity of designing distributed applications, many proposals provide efficient high-level communication abstractions. Since the implementation of a production-ready system based on this abstraction is still a major task, we further propose to expose our protocol to developers in the form of distributed data structures. B-trees for example, are commonly used in different kinds of applications, including database indexes or file systems. Providing a distributed, fault-tolerant and scalable data structure would help developers to integrate their applications in a distribution transparent manner. This work describes how to build reliable and scalable distributed systems based on atomic multicast and demonstrates their capabilities by an implementation of a distributed ordered map that supports dynamic re-partitioning and fast recovery. To substantiate our claim, we ported an existing SQL database atop of our distributed lock-free data structure. Here, replication plays a central role, as it contributes to improve the user experience by hiding failures and by providing acceptable latency. In this thesis, we claim that atomic multicast, with strong and well-defined properties, is the appropriate abstraction to efficiently design and implement globally scalable distributed systems. Internet-scale services rely on data partitioning and replication to provide scalable performance and high availability. Moreover, to reduce user-perceived response times and tolerate disasters (i.e., the failure of a whole datacenter), services are increasingly becoming geographically distributed. Data partitioning and replication, combined with local and geographical distribution, introduce daunting challenges, including the need to carefully order requests among replicas and partitions. One way to tackle this problem is to use group communication primitives that encapsulate order requirements. While replication is a common technique used to design such reliable distributed systems, to cope with the requirements of modern cloud based ``always-on'' applications, replication protocols must additionally allow for throughput scalability and dynamic reconfiguration, that is, on-demand replacement or provisioning of system resources. We propose a dynamic atomic multicast protocol which fulfills these requirements. It allows to dynamically add and remove resources to an online replicated state machine and to recover crashed processes. Major efforts have been spent in recent years to improve the performance, scalability and reliability of distributed systems. In order to hide the complexity of designing distributed applications, many proposals provide efficient high-level communication abstractions. Since the implementation of a production-ready system based on this abstraction is still a major task, we further propose to expose our protocol to developers in the form of distributed data structures. B- trees for example, are commonly used in different kinds of applications, including database indexes or file systems. Providing a distributed, fault-tolerant and scalable data structure would help developers to integrate their applications in a distribution transparent manner. This work describes how to build reliable and scalable distributed systems based on atomic multicast and demonstrates their capabilities by an implementation of a distributed ordered map that supports dynamic re-partitioning and fast recovery. To substantiate our claim, we ported an existing SQL database atop of our distributed lock-free data structure