20,808 research outputs found
A New Framework for Join Product Skew
Different types of data skew can result in load imbalance in the context of
parallel joins under the shared nothing architecture. We study one important
type of skew, join product skew (JPS). A static approach based on frequency
classes is proposed which takes for granted the data distribution of join
attribute values. It comes from the observation that the join selectivity can
be expressed as a sum of products of frequencies of the join attribute values.
As a consequence, an appropriate assignment of join sub-tasks, that takes into
consideration the magnitude of the frequency products can alleviate the join
product skew. Motivated by the aforementioned remark, we propose an algorithm,
called Handling Join Product Skew (HJPS), to handle join product skew
맵리듀스에서의 병렬 조인을 위한 다차원 범위 분할 기법
학위논문 (박사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2014. 8. 이상구.Joins are fundamental operations for many data analysis tasks, but are not directly supported by the MapReduce framework. This is because 1) the framework is basically designed to process a single input data set, and 2) MapReduce's key-equality based data grouping method makes it difficult to support complex join conditions. As a result, a large number of MapReduce-based join algorithms have been proposed.
As in traditional shared-nothing systems, one of the major issues in join algorithms using MapReduce is handling of data skew. We propose a new skew handling method, called Multi-Dimensional Range Partitioning (MDRP), and show that the proposed method outperforms traditional skew handling methods: range-based and randomized methods. Specifically, the proposed method has the following advantages: 1) Compared to the range-based method, it considers the number of output tuples at each machine, which leads better handling of join product skew. 2) Compared with the randomized method, it exploits given join conditions before the actual join begins, so that unnecessary input duplication can be reduced.
The MDRP method can be used to support advanced join operations such as theta-joins and multi-way joins. With extensive experiments using real and synthetic data sets, we evaluate the effectiveness of the proposed algorithm.Abstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i
I. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
II. Backgrounds and RelatedWork . . . . . . . . . . . . . . . . 8
2.1 MapReduce . . . . . . . . . . . . . . . . . . . . . . . . . . 8
2.2 Join Algorithms in MapReduce . . . . . . . . . . . . . . . . 11
2.2.1 Two-Way Join Algorithms . . . . . . . . . . . . . . 11
2.2.2 Multi-Way Join Algorithms . . . . . . . . . . . . . 17
2.3 Data Skew in Join Algorithms . . . . . . . . . . . . . . . . 18
2.4 Skew Handling Approaches in MapReduce . . . . . . . . . 22
2.4.1 Hash-Based Approach . . . . . . . . . . . . . . . . 22
2.4.2 Range-Based Approach . . . . . . . . . . . . . . . 24
2.4.3 Randomized Approach . . . . . . . . . . . . . . . . 26
III. Our Approach . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.1 Multi-Dimensional Range Partitioning . . . . . . . . . . . . 29
3.1.1 Creation of a Partitioning Matrix . . . . . . . . . . . 29
3.1.2 Identifying and Chopping of Heavy Cells . . . . . . 31
3.1.3 Assigning Cells to Reducers . . . . . . . . . . . . . 33
3.1.4 Join Processing using the Partitioning Matrix . . . . 35
3.2 Theoretical Analysis . . . . . . . . . . . . . . . . . . . . . 39
3.3 Complex Join Conditions . . . . . . . . . . . . . . . . . . . 41
3.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.4.1 Scalar Skew Experiments . . . . . . . . . . . . . . . 44
3.4.2 Zipfs Distribution . . . . . . . . . . . . . . . . . . 49
3.4.3 Non-Equijoin Experiments . . . . . . . . . . . . . . 50
3.4.4 Scalability Experiments . . . . . . . . . . . . . . . 52
3.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
3.5.1 Sampling . . . . . . . . . . . . . . . . . . . . . . . 55
3.5.2 Memory-Awareness . . . . . . . . . . . . . . . . . 58
3.5.3 Handling of Heavy Cells . . . . . . . . . . . . . . . 59
3.5.4 Existing Histograms . . . . . . . . . . . . . . . . . 60
3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
IV. Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
4.1 Joining Multiple Relations in a MapReduce Job . . . . . . . 65
4.1.1 Example: SPARQL Basic Graph Pattern . . . . . . . 65
4.1.2 Example: Matrix Chain Multiplication . . . . . . . . 67
4.1.3 Single-Key Join and Multiple-Key Join Queries . . . 69
4.2 Skew Handling for Multi-Way Joins . . . . . . . . . . . . . 71
4.2.1 Skew Handling for SK-Join Queries . . . . . . . . . 71
4.2.2 Skew Handling for MK-Join Queires . . . . . . . . 72
4.3 Combinations of SK-Join and MK-Join . . . . . . . . . . . 74
4.3.1 Complex Queries . . . . . . . . . . . . . . . . . . . 74
4.3.2 Iteration-Based Algorithms . . . . . . . . . . . . . . 75
4.3.3 Replication-Based Algorithms . . . . . . . . . . . . 77
4.3.4 Iteration-Based vs. Replication-Based . . . . . . . . 78
4.4 Join-Key Selection Algorithms for Complex Queries . . . . 83
4.4.1 Greedy Key Selection . . . . . . . . . . . . . . . . 84
4.4.2 Multiple Key Selection . . . . . . . . . . . . . . . . 85
4.4.3 Hybrid Key Selection . . . . . . . . . . . . . . . . . 86
4.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 87
4.5.1 SK-Join Experiments . . . . . . . . . . . . . . . . . 87
4.5.2 MK-Join Experiments . . . . . . . . . . . . . . . . 89
4.5.3 Analysis of TV Watching Logs . . . . . . . . . . . . 90
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
V. Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
5.1 Algorithms for SPARQL Basic Graph Pattern . . . . . . . . 94
5.1.1 MR-Selection . . . . . . . . . . . . . . . . . . . . . 95
5.1.2 MR-Join . . . . . . . . . . . . . . . . . . . . . . . 98
5.1.3 Performance Evaluation . . . . . . . . . . . . . . . 101
5.1.4 Discussion . . . . . . . . . . . . . . . . . . . . . . 105
5.2 Algorithms for Matrix Chain Multiplication . . . . . . . . . 107
5.2.1 Serial Two-Way Join (S2) . . . . . . . . . . . . . . 109
5.2.2 Parallel M-Way Join (P2, PM) . . . . . . . . . . . . 111
5.2.3 Serial Two-Way vs. Parallel M-Way . . . . . . . . . 115
5.2.4 Performance Evaluation . . . . . . . . . . . . . . . 116
5.2.5 Discussion . . . . . . . . . . . . . . . . . . . . . . 119
5.2.6 Extension: Embedded MapReduce . . . . . . . . . . 119
VI. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
초록 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133Docto
Distributed Triangle Counting in the Graphulo Matrix Math Library
Triangle counting is a key algorithm for large graph analysis. The Graphulo
library provides a framework for implementing graph algorithms on the Apache
Accumulo distributed database. In this work we adapt two algorithms for
counting triangles, one that uses the adjacency matrix and another that also
uses the incidence matrix, to the Graphulo library for server-side processing
inside Accumulo. Cloud-based experiments show a similar performance profile for
these different approaches on the family of power law Graph500 graphs, for
which data skew increasingly bottlenecks. These results motivate the design of
skew-aware hybrid algorithms that we propose for future work.Comment: Honorable mention in the 2017 IEEE HPEC's Graph Challeng
Parallelizing Windowed Stream Joins in a Shared-Nothing Cluster
The availability of large number of processing nodes in a parallel and
distributed computing environment enables sophisticated real time processing
over high speed data streams, as required by many emerging applications.
Sliding window stream joins are among the most important operators in a stream
processing system. In this paper, we consider the issue of parallelizing a
sliding window stream join operator over a shared nothing cluster. We propose a
framework, based on fixed or predefined communication pattern, to distribute
the join processing loads over the shared-nothing cluster. We consider various
overheads while scaling over a large number of nodes, and propose solution
methodologies to cope with the issues. We implement the algorithm over a
cluster using a message passing system, and present the experimental results
showing the effectiveness of the join processing algorithm.Comment: 11 page
Variable sets over an algebra of lifetimes: a contribution of lattice theory to the study of computational topology
A topos theoretic generalisation of the category of sets allows for modelling
spaces which vary according to time intervals. Persistent homology, or more
generally, persistence is a central tool in topological data analysis, which
examines the structure of data through topology. The basic techniques have been
extended in several different directions, permuting the encoding of topological
features by so called barcodes or equivalently persistence diagrams. The set of
points of all such diagrams determines a complete Heyting algebra that can
explain aspects of the relations between persistent bars through the algebraic
properties of its underlying lattice structure. In this paper, we investigate
the topos of sheaves over such algebra, as well as discuss its construction and
potential for a generalised simplicial homology over it. In particular we are
interested in establishing a topos theoretic unifying theory for the various
flavours of persistent homology that have emerged so far, providing a global
perspective over the algebraic foundations of applied and computational
topology.Comment: 20 pages, 12 figures, AAA88 Conference proceedings at Demonstratio
Mathematica. The new version has restructured arguments, clearer intuition is
provided, and several typos correcte
- …