Fan, Wenfei

Geerts, Floris

Ma, Shuai

Mueller, Heiko

English

One of the central problems for data quality is inconsistency detection. Given a database D and a set Sigma of dependencies as data quality rules, we want to identify tuples in D that violate some rules in Sigma. When D is a centralized database, there have been effective SQL-based techniques for finding violations. It is, however, far more challenging when data in D is distributed, in which inconsistency detection often necessarily requires shipping data from one site to another.This paper develops techniques for detecting violations of conditional functional dependencies (CFDs) in relations that are fragmented and distributed across different sites. (1) We formulate the detection problem in various distributed settings as optimization problems, measured by either network traffic or response time. (2) We show that it is beyond reach in practice to find optimal detection methods: the detection problem is NP-complete when the data is partitioned either horizontally or vertically, and when we aim to minimize either data shipment or response time. (3) For data that is horizontally partitioned, we provide several algorithms to find violations of a set of CFDs, leveraging the structure of CFDs to reduce data shipment or increase parallelism. (4) We verify experimentally that our algorithms are scalable on large relations and complex CFDs. (5) For data that is vertically partitioned, we provide a characterization for CFDs to be checked locally without requiring data shipment, in terms of dependency preservation. We show that it is intractable to minimally refine a partition and make it dependency preserving.</p

Edinburgh Research Explorer

An overview of query optimization in relational systems,”

C-store: A column-oriented DBMS,” in VLDB,

Computers and Intractability: A Guide to the Theory of NP-Completeness.

Conditional functional dependencies for capturing data inconsistencies,”

Constraint checking with partial information,”

Discovering data quality rules,” in VLDB,

Distributed enforcement of integrity constraints,”

Dynamic programming strikes back,” in SIGMOD,

Efﬁcient and extensible algorithms for multi query optimization,” in SIGMOD,

Efﬁcient detection of distributed constraint violations,” in ICDE,

Forecast: Data quality tools, worldwide,

Foundations of Databases.

Improving data quality: Consistency and accuracy,” in VLDB,

Increasing the expressivity of conditional functional dependencies without extra complexity,” in ICDE,

Local veriﬁcation of global integrity constraints in distributed databases,”

Maintaining global integrity constraints in distributed databases,”

Minimizing communication cost in distributed multi-query processing,” in ICDE,

Multiple-query optimization,”

Network-aware join processing in global-scale database federations,” in ICDE,

On generating near-optimal tableaux for conditional functional dependencies,” in VLDB,

On the complexity of distributed query

Optimal top-down join enumeration,” in SIGMOD,

Propagating functional dependencies with conditions,” in VLDB,

R* optimizer validation and performance evaluation for distributed queries,” in VLDB,

The state of the art in distributed query processing,”

Using semi-joins to solve relational queries,”

Detecting Inconsistencies in Distributed Data

Detecting Inconsistencies in Distributed Data

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer