17,723 research outputs found
Parallelizing Windowed Stream Joins in a Shared-Nothing Cluster
The availability of large number of processing nodes in a parallel and
distributed computing environment enables sophisticated real time processing
over high speed data streams, as required by many emerging applications.
Sliding window stream joins are among the most important operators in a stream
processing system. In this paper, we consider the issue of parallelizing a
sliding window stream join operator over a shared nothing cluster. We propose a
framework, based on fixed or predefined communication pattern, to distribute
the join processing loads over the shared-nothing cluster. We consider various
overheads while scaling over a large number of nodes, and propose solution
methodologies to cope with the issues. We implement the algorithm over a
cluster using a message passing system, and present the experimental results
showing the effectiveness of the join processing algorithm.Comment: 11 page
A New Framework for Join Product Skew
Different types of data skew can result in load imbalance in the context of
parallel joins under the shared nothing architecture. We study one important
type of skew, join product skew (JPS). A static approach based on frequency
classes is proposed which takes for granted the data distribution of join
attribute values. It comes from the observation that the join selectivity can
be expressed as a sum of products of frequencies of the join attribute values.
As a consequence, an appropriate assignment of join sub-tasks, that takes into
consideration the magnitude of the frequency products can alleviate the join
product skew. Motivated by the aforementioned remark, we propose an algorithm,
called Handling Join Product Skew (HJPS), to handle join product skew
Scaling and Load-Balancing Equi-Joins
The task of joining two tables is fundamental for querying databases. In this
paper, we focus on the equi-join problem, where a pair of records from the two
joined tables are part of the join results if equality holds between their
values in the join column(s). While this is a tractable problem when the number
of records in the joined tables is relatively small, it becomes very
challenging as the table sizes increase, especially if hot keys (join column
values with a large number of records) exist in both joined tables.
This paper, an extended version of [metwally-SIGMOD-2022], proposes
Adaptive-Multistage-Join (AM-Join) for scalable and fast equi-joins in
distributed shared-nothing architectures. AM-Join utilizes (a) Tree-Join, a
proposed novel algorithm that scales well when the joined tables share hot
keys, and (b) Broadcast-Join, the known fastest when joining keys that are hot
in only one table.
Unlike the state-of-the-art algorithms, AM-Join (a) holistically solves the
join-skew problem by achieving load balancing throughout the join execution,
and (b) supports all outer-join variants without record deduplication or custom
table partitioning. For the fastest AM-Join outer-join performance, we propose
the Index-Broadcast-Join (IB-Join) family of algorithms for Small-Large joins,
where one table fits in memory and the other can be up to orders of magnitude
larger. The outer-join variants of IB-Join improves on the state-of-the-art
Small-Large outer-join algorithms.
The proposed algorithms can be adopted in any shared-nothing architecture. We
implemented a MapReduce version using Spark. Our evaluation shows the proposed
algorithms execute significantly faster and scale to more skewed and
orders-of-magnitude bigger tables when compared to the state-of-the-art
algorithms
GLB: Lifeline-based Global Load Balancing library in X10
We present GLB, a programming model and an associated implementation that can
handle a wide range of irregular paral- lel programming problems running over
large-scale distributed systems. GLB is applicable both to problems that are
easily load-balanced via static scheduling and to problems that are hard to
statically load balance. GLB hides the intricate syn- chronizations (e.g.,
inter-node communication, initialization and startup, load balancing,
termination and result collection) from the users. GLB internally uses a
version of the lifeline graph based work-stealing algorithm proposed by
Saraswat et al. Users of GLB are simply required to write several pieces of
sequential code that comply with the GLB interface. GLB then schedules and
orchestrates the parallel execution of the code correctly and efficiently at
scale. We have applied GLB to two representative benchmarks: Betweenness
Centrality (BC) and Unbalanced Tree Search (UTS). Among them, BC can be
statically load-balanced whereas UTS cannot. In either case, GLB scales well--
achieving nearly linear speedup on different computer architectures (Power,
Blue Gene/Q, and K) -- up to 16K cores
Learning a Partitioning Advisor with Deep Reinforcement Learning
Commercial data analytics products such as Microsoft Azure SQL Data Warehouse
or Amazon Redshift provide ready-to-use scale-out database solutions for
OLAP-style workloads in the cloud. While the provisioning of a database cluster
is usually fully automated by cloud providers, customers typically still have
to make important design decisions which were traditionally made by the
database administrator such as selecting the partitioning schemes.
In this paper we introduce a learned partitioning advisor for analytical
OLAP-style workloads based on Deep Reinforcement Learning (DRL). The main idea
is that a DRL agent learns its decisions based on experience by monitoring the
rewards for different workloads and partitioning schemes. We evaluate our
learned partitioning advisor in an experimental evaluation with different
databases schemata and workloads of varying complexity. In the evaluation, we
show that our advisor is not only able to find partitionings that outperform
existing approaches for automated partitioning design but that it also can
easily adjust to different deployments. This is especially important in cloud
setups where customers can easily migrate their cluster to a new set of
(virtual) machines
Adaptive Lock-Free Data Structures in Haskell: A General Method for Concurrent Implementation Swapping
A key part of implementing high-level languages is providing built-in and
default data structures. Yet selecting good defaults is hard. A mutable data
structure's workload is not known in advance, and it may shift over its
lifetime - e.g., between read-heavy and write-heavy, or from heavy contention
by multiple threads to single-threaded or low-frequency use. One idea is to
switch implementations adaptively, but it is nontrivial to switch the
implementation of a concurrent data structure at runtime. Performing the
transition requires a concurrent snapshot of data structure contents, which
normally demands special engineering in the data structure's design. However,
in this paper we identify and formalize an relevant property of lock-free
algorithms. Namely, lock-freedom is sufficient to guarantee that freezing
memory locations in an arbitrary order will result in a valid snapshot. Several
functional languages have data structures that freeze and thaw, transitioning
between mutable and immutable, such as Haskell vectors and Clojure transients,
but these enable only single-threaded writers. We generalize this approach to
augment an arbitrary lock-free data structure with the ability to gradually
freeze and optionally transition to a new representation. This augmentation
doesn't require changing the algorithm or code for the data structure, only
replacing its datatype for mutable references with a freezable variant. In this
paper, we present an algorithm for lifting plain to adaptive data and prove
that the resulting hybrid data structure is itself lock-free, linearizable, and
simulates the original. We also perform an empirical case study in the context
of heating up and cooling down concurrent maps.Comment: To be published in ACM SIGPLAN Haskell Symposium 201
- …