9,937 research outputs found
A survey of parallel execution strategies for transitive closure and logic programs
An important feature of database technology of the nineties is the use of parallelism for speeding up the execution of complex queries. This technology is being tested in several experimental database architectures and a few commercial systems for conventional select-project-join queries. In particular, hash-based fragmentation is used to distribute data to disks under the control of different processors in order to perform selections and joins in parallel. With the development of new query languages, and in particular with the definition of transitive closure queries and of more general logic programming queries, the new dimension of recursion has been added to query processing. Recursive queries are complex; at the same time, their regular structure is particularly suited for parallel execution, and parallelism may give a high efficiency gain. We survey the approaches to parallel execution of recursive queries that have been presented in the recent literature. We observe that research on parallel execution of recursive queries is separated into two distinct subareas, one focused on the transitive closure of Relational Algebra expressions, the other one focused on optimization of more general Datalog queries. Though the subareas seem radically different because of the approach and formalism used, they have many common features. This is not surprising, because most typical Datalog queries can be solved by means of the transitive closure of simple algebraic expressions. We first analyze the relationship between the transitive closure of expressions in Relational Algebra and Datalog programs. We then review sequential methods for evaluating transitive closure, distinguishing iterative and direct methods. We address the parallelization of these methods, by discussing various forms of parallelization. Data fragmentation plays an important role in obtaining parallel execution; we describe hash-based and semantic fragmentation. Finally, we consider Datalog queries, and present general methods for parallel rule execution; we recognize the similarities between these methods and the methods reviewed previously, when the former are applied to linear Datalog queries. We also provide a quantitative analysis that shows the impact of the initial data distribution on the performance of methods
Algebraic optimization of recursive queries
Over the past few years, much attention has been paid to deductive databases. They offer a logic-based interface, and allow formulation of complex recursive queries. However, they do not offer appropriate update facilities, and do not support existing applications. To overcome these problems an SQL-like interface is required besides a logic-based interface.\ud
\ud
In the PRISMA project we have developed a tightly-coupled distributed database, on a multiprocessor machine, with two user interfaces: SQL and PRISMAlog. Query optimization is localized in one component: the relational query optimizer. Therefore, we have defined an eXtended Relational Algebra that allows recursive query formulation and can also be used for expressing executable schedules, and we have developed algebraic optimization strategies for recursive queries. In this paper we describe an optimization strategy that rewrites regular (in the context of formal grammars) mutually recursive queries into standard Relational Algebra and transitive closure operations. We also describe how to push selections into the resulting transitive closure operations.\ud
\ud
The reason we focus on algebraic optimization is that, in our opinion, the new generation of advanced database systems will be built starting from existing state-of-the-art relational technology, instead of building a completely new class of systems
A Genetic Programming Framework for Two Data Mining Tasks: Classification and Generalized Rule Induction
This paper proposes a genetic programming (GP) framework for two major data mining tasks, namely classification and generalized rule induction. The framework emphasizes the integration between a GP algorithm and relational database systems. In particular, the fitness of individuals is computed by submitting SQL queries to a (parallel) database server. Some advantages of this integration from a data mining viewpoint are scalability, data-privacy control and automatic parallelization
A Survey on Array Storage, Query Languages, and Systems
Since scientific investigation is one of the most important providers of
massive amounts of ordered data, there is a renewed interest in array data
processing in the context of Big Data. To the best of our knowledge, a unified
resource that summarizes and analyzes array processing research over its long
existence is currently missing. In this survey, we provide a guide for past,
present, and future research in array processing. The survey is organized along
three main topics. Array storage discusses all the aspects related to array
partitioning into chunks. The identification of a reduced set of array
operators to form the foundation for an array query language is analyzed across
multiple such proposals. Lastly, we survey real systems for array processing.
The result is a thorough survey on array data storage and processing that
should be consulted by anyone interested in this research topic, independent of
experience level. The survey is not complete though. We greatly appreciate
pointers towards any work we might have forgotten to mention.Comment: 44 page
Experimental Study of the Cloud Architecture Selection for Effective Big Data Processing
Big data dictate their requirements to the hardware and software. Simple
migration to the cloud data processing, while solving the problem of increasing
computational capabilities, however creates some issues: the need to ensure the
safety, the need to control the quality during data transmission, the need to
optimize requests. Computational cloud does not simply provide scalable
resources but also requires network infrastructure, unknown routes and the
number of user requests. In addition, during functioning situation can occur,
in which you need to change the architecture of the application - part of the
data needs to be placed in a private cloud, part in a public cloud, part stays
on the client
A Brief Study of Open Source Graph Databases
With the proliferation of large irregular sparse relational datasets, new
storage and analysis platforms have arisen to fill gaps in performance and
capability left by conventional approaches built on traditional database
technologies and query languages. Many of these platforms apply graph
structures and analysis techniques to enable users to ingest, update, query and
compute on the topological structure of these relationships represented as
set(s) of edges between set(s) of vertices. To store and process Facebook-scale
datasets, they must be able to support data sources with billions of edges,
update rates of millions of updates per second, and complex analysis kernels.
These platforms must provide intuitive interfaces that enable graph experts and
novice programmers to write implementations of common graph algorithms. In this
paper, we explore a variety of graph analysis and storage platforms. We compare
their capabil- ities, interfaces, and performance by implementing and computing
a set of real-world graph algorithms on synthetic graphs with up to 256 million
edges. In the spirit of full disclosure, several authors are affiliated with
the development of STINGER.Comment: WSSSPE13, 4 Pages, 18 Pages with Appendix, 25 figure
- …