110,310 research outputs found
The End of a Myth: Distributed Transactions Can Scale
The common wisdom is that distributed transactions do not scale. But what if
distributed transactions could be made scalable using the next generation of
networks and a redesign of distributed databases? There would be no need for
developers anymore to worry about co-partitioning schemes to achieve decent
performance. Application development would become easier as data placement
would no longer determine how scalable an application is. Hardware provisioning
would be simplified as the system administrator can expect a linear scale-out
when adding more machines rather than some complex sub-linear function, which
is highly application specific.
In this paper, we present the design of our novel scalable database system
NAM-DB and show that distributed transactions with the very common Snapshot
Isolation guarantee can indeed scale using the next generation of RDMA-enabled
network technology without any inherent bottlenecks. Our experiments with the
TPC-C benchmark show that our system scales linearly to over 6.5 million
new-order (14.5 million total) distributed transactions per second on 56
machines.Comment: 12 page
Instant restore after a media failure
Media failures usually leave database systems unavailable for several hours
until recovery is complete, especially in applications with large devices and
high transaction volume. Previous work introduced a technique called
single-pass restore, which increases restore bandwidth and thus substantially
decreases time to repair. Instant restore goes further as it permits read/write
access to any data on a device undergoing restore--even data not yet
restored--by restoring individual data segments on demand. Thus, the restore
process is guided primarily by the needs of applications, and the observed mean
time to repair is effectively reduced from several hours to a few seconds.
This paper presents an implementation and evaluation of instant restore. The
technique is incrementally implemented on a system starting with the
traditional ARIES design for logging and recovery. Experiments show that the
transaction latency perceived after a media failure can be cut down to less
than a second and that the overhead imposed by the technique on normal
processing is minimal. The net effect is that a few "nines" of availability are
added to the system using simple and low-overhead software techniques
Security, Performance and Energy Trade-offs of Hardware-assisted Memory Protection Mechanisms
The deployment of large-scale distributed systems, e.g., publish-subscribe
platforms, that operate over sensitive data using the infrastructure of public
cloud providers, is nowadays heavily hindered by the surging lack of trust
toward the cloud operators. Although purely software-based solutions exist to
protect the confidentiality of data and the processing itself, such as
homomorphic encryption schemes, their performance is far from being practical
under real-world workloads.
The performance trade-offs of two novel hardware-assisted memory protection
mechanisms, namely AMD SEV and Intel SGX - currently available on the market to
tackle this problem, are described in this practical experience.
Specifically, we implement and evaluate a publish/subscribe use-case and
evaluate the impact of the memory protection mechanisms and the resulting
performance. This paper reports on the experience gained while building this
system, in particular when having to cope with the technical limitations
imposed by SEV and SGX.
Several trade-offs that provide valuable insights in terms of latency,
throughput, processing time and energy requirements are exhibited by means of
micro- and macro-benchmarks.Comment: European Commission Project: LEGaTO - Low Energy Toolset for
Heterogeneous Computing (EC-H2020-780681
Implementing the Gaia Astrometric Global Iterative Solution (AGIS) in Java
This paper provides a description of the Java software framework which has
been constructed to run the Astrometric Global Iterative Solution for the Gaia
mission. This is the mathematical framework to provide the rigid reference
frame for Gaia observations from the Gaia data itself. This process makes Gaia
a self calibrated, and input catalogue independent, mission. The framework is
highly distributed typically running on a cluster of machines with a database
back end. All code is written in the Java language. We describe the overall
architecture and some of the details of the implementation.Comment: Accepted for Experimental Astronom
High performance deferred update replication
Replication is a well-known approach to implementing storage systems that can tolerate failures. Replicated storage systems are designed such that the state of the system is kept at several replicas. A replication protocol ensures that the failure of a replica is masked by the rest of the system, in a way that is transparent to its users. Replicated storage systems are among the most important building blocks in the design of large scale applications. Applications at scale are often deployed on top of commodity hardware, store a vast amount of data, and serve a large number of users. The larger the system, the higher its vulnerability to failures. The ability to tolerate failures is not the only desirable feature in a replicated system. Storage systems need to be efficient in order to accommodate requests from a large user base while achieving low response times. In that respect, replication can leverage multiple replicas to parallelize the execution of user requests. This thesis focuses on Deferred Update Replication (DUR), a well-established database replication approach. It provides high availability in that every replica can execute client transactions. In terms of performance, it is better than other replication techniques in that only one replica executes a given transaction while the other replicas only apply state changes. However, DUR suffers from the following drawback: each replica stores a full copy of the database, which has consequences in terms of performance. The first consequence is that DUR cannot take advantage of the aggregated memory available to the replicas. Our first contribution is a distributed caching mechanism that addresses the problem. It makes efficient use of the main memory of an entire cluster of machines, while guaranteeing strong consistency. The second consequence is that DUR cannot scale with the number of replicas. The throughput of a fully replicated system is inherently limited by the number of transactions that a single replica can apply to its local storage. We propose a scalable version of the DUR approach where the system state is partitioned in smaller replica sets. Transactions that access disjoint partitions are parallelized. The last part of the thesis focuses on latency. We show that the scalable DUR-based approach may have detrimental effects on response time, especially when replicas are geographically distributed. The thesis considers different deployments and their implications on latency. We propose optimizations that provide substantial gains in geographically distributed environments
- âŠ