Database Integration: the Key to Data Interoperability

Most of new databases are no more built from scratch, but re-use existing data from several autonomous data stores. To facilitate application development, the data to be re-used should preferably be redefined as a virtual database, providing for the logical unification of the underlying data sets. This unification process is called database integration. This chapter provides a global picture of the issues raised and the approaches that have been proposed to tackle the problem

Inter-database instance identification in composite information systems

Bibliography: p. 20-21.Y. Richard Wang, Stuart E. Madnick, David C. Horton

A Scalable Blocking Framework for Multidatabase Privacy-preserving Record Linkage

Today many application domains, such as national statistics, healthcare, business analytic, fraud detection, and national security, require data to be integrated from multiple databases. Record linkage (RL) is a process used in data integration which links multiple databases to identify matching records that belong to the same entity. RL enriches the usefulness of data by removing duplicates, errors, and inconsistencies which improves the effectiveness of decision making in data analytic applications. Often, organisations are not willing or authorised to share the sensitive information in their databases with any other party due to privacy and confidentiality regulations. The linkage of databases of different organisations is an emerging research area known as privacy-preserving record linkage (PPRL). PPRL facilitates the linkage of databases by ensuring the privacy of the entities in these databases. In multidatabase (MD) context, PPRL is significantly challenged by the intrinsic exponential growth in the number of potential record pair comparisons. Such linkage often requires significant time and computational resources to produce the resulting matching sets of records. Due to increased risk of collusion, preserving the privacy of the data is more problematic with an increase of number of parties involved in the linkage process. Blocking is commonly used to scale the linkage of large databases. The aim of blocking is to remove those record pairs that correspond to non-matches (refer to different entities). Many techniques have been proposed for RL and PPRL for blocking two databases. However, many of these techniques are not suitable for blocking multiple databases. This creates a need to develop blocking technique for the multidatabase linkage context as real-world applications increasingly require more than two databases. This thesis is the first to conduct extensive research on blocking for multidatabase privacy-preserved record linkage (MD-PPRL). We consider several research problems in blocking of MD-PPRL. First, we start with a broad background literature on PPRL. This allow us to identify the main research gaps that need to be investigated in MD-PPRL. Second, we introduce a blocking framework for MD-PPRL which provides more flexibility and control to database owners in the block generation process. Third, we propose different techniques that are used in our framework for (1) blocking of multiple databases, (2) identifying blocks that need to be compared across subgroups of these databases, and (3) filtering redundant record pair comparisons by the efficient scheduling of block comparisons to improve the scalability of MD-PPRL. Each of these techniques covers an important aspect of blocking in real-world MD-PPRL applications. Finally, this thesis reports on an extensive evaluation of the combined application of these methods with real datasets, which illustrates that they outperform existing approaches in term of scalability, accuracy, and privacy

A Rational Scheme for Conflict Detection and Resolution in Distributed Collaborative Environments for Enterprise Integration

A typical enterprise may have large numbers of information sources such as data stores, expert systems, knowledge-based systems, or standard software systems. These may need to be integrated so that, for example, an application program or a decision maker can access information from all these sources. Such architectures are generally called 'Distributed Collaborative Environments for Enterprise Integration'. A general problem in these enterprise integration architectures is that information from heterogeneous, pre-existing sources may be obsolete, incomplete, incorrect or, for many other reasons, contradictory. Thus, conflicting results may occur when the same information is requested from semantically related sources. A mechanism is required to detect and resolve these conflicts in a way that is rational to any potential client of the integration environment. This thesis lays open the design of a general mechanism for conflict detection and resolution that enables intelligent information agents to reason about contradictory information from pre-existing, heterogeneous and autonomous sources. The mechanism's theoretical basis is a framework that is drawn from evidence law, which shares some fundamental commonalities with conflict detection and resolution in enterprise integration environments. Conflict detection opens with gathering the results collected by the information retrieval process. These results may have justifications or certainty assessments attached to them. Furthermore, it identifies whether and how these results are conflicting. The design of a conflict resolution mechanism is based on a rational scheme for judging the weight of conflicting results. First, the agents assess the reliability or credibility of an information source. Judgement based on the weight of conflicting results is first applied to any available, domain-specific, resolution strategies. Second, the agent applies any 'general scientific' resolution strategies that are not specific to one domain. When no domain-related expertise can solve the conflict then the agent can only judge on domain independent evaluation criteria such as the results' reliability. A scheme is sketched out for judgement based on the reliability of conflicting results, involving three steps: Ranking the conflicting results according to their reliability; Ways to redefine conflicting results; and Heuristic decision-making. The evaluation includes a computational implementation of an enterprise integration environment incorporating a model of an information agent. An example is realised in this environment. The conflict detection and resolution mechanism, and interfaces to each integrated source, are implemented in Visual C++. A case study is conducted on this scenario to evaluate each conflict detection and resolution step. Furthermore, this illustrates both the advantages over existing approaches and the limitations