96 research outputs found
Finding Top-k Dominance on Incomplete Big Data Using Map-Reduce Framework
Incomplete data is one major kind of multi-dimensional dataset that has random-distributed missing nodes in its dimensions. It is very difficult to retrieve information from this type of dataset when it becomes huge. Finding top-k dominant values in this type of dataset is a challenging procedure. Some algorithms are present to enhance this process but are mostly efficient only when dealing with a small-size incomplete data. One of the algorithms that make the application of TKD query possible is the Bitmap Index Guided (BIG) algorithm. This algorithm strongly improves the performance for incomplete data, but it is not originally capable of finding top-k dominant values in incomplete big data, nor is it designed to do so. Several other algorithms have been proposed to find the TKD query, such as Skyband Based and Upper Bound Based algorithms, but their performance is also questionable. Algorithms developed previously were among the first attempts to apply TKD query on incomplete data; however, all these had weak performances or were not compatible with the incomplete data. This thesis proposes MapReduced Enhanced Bitmap Index Guided Algorithm (MRBIG) for dealing with the aforementioned issues. MRBIG uses the MapReduce framework to enhance the performance of applying top-k dominance queries on huge incomplete datasets. The proposed approach uses the MapReduce parallel computing approach using multiple computing nodes. The framework separates the tasks between several computing nodes that independently and simultaneously work to find the result. This method has achieved up to two times faster processing time in finding the TKD query result in comparison to previously presented algorithms
Distributed Indexing Schemes for k-Dominant Skyline Analytics on Uncertain Edge-IoT Data
Skyline queries typically search a Pareto-optimal set from a given data set
to solve the corresponding multiobjective optimization problem. As the number
of criteria increases, the skyline presumes excessive data items, which yield a
meaningless result. To address this curse of dimensionality, we proposed a
k-dominant skyline in which the number of skyline members was reduced by
relaxing the restriction on the number of dimensions, considering the
uncertainty of data. Specifically, each data item was associated with a
probability of appearance, which represented the probability of becoming a
member of the k-dominant skyline. As data items appear continuously in data
streams, the corresponding k-dominant skyline may vary with time. Therefore, an
effective and rapid mechanism of updating the k-dominant skyline becomes
crucial. Herein, we proposed two time-efficient schemes, Middle Indexing (MI)
and All Indexing (AI), for k-dominant skyline in distributed edge-computing
environments, where irrelevant data items can be effectively excluded from the
compute to reduce the processing duration. Furthermore, the proposed schemes
were validated with extensive experimental simulations. The experimental
results demonstrated that the proposed MI and AI schemes reduced the
computation time by approximately 13% and 56%, respectively, compared with the
existing method.Comment: 13 pages, 8 figures, 12 tables, to appear in IEEE Transactions on
Emerging Topics in Computin
Integration of Skyline Queries into Spark SQL
Skyline queries are frequently used in data analytics and multi-criteria
decision support applications to filter relevant information from big amounts
of data. Apache Spark is a popular framework for processing big, distributed
data. The framework even provides a convenient SQL-like interface via the Spark
SQL module. However, skyline queries are not natively supported and require
tedious rewriting to fit the SQL standard or Spark's SQL-like language. The
goal of our work is to fill this gap. We thus provide a full-fledged
integration of the skyline operator into Spark SQL. This allows for a simple
and easy to use syntax to input skyline queries. Moreover, our empirical
results show that this integrated solution of skyline queries by far
outperforms a solution based on rewriting into standard SQL
Dynamic Skyline Computation with the Skyline Breaker Algorithm
Given a sequential data input, we tackle parallel dynamic skyline computation of the read data by means of a spatial tree structure for indexing fine-grained feature vectors. For this purpose, we modified the Skyline Breaker algorithm that solves skyline computation with multiple local split decision trees concurrently. With this approach, we propose an algorithm for dynamic skyline computation that inherits the robustness against the dimension curse and different data distributions
- …