4,218 research outputs found
Improving data quality : data consistency, deduplication, currency and accuracy
Data quality is one of the key problems in data management. An unprecedented amount of data has been accumulated and has become a valuable asset of an organization. The value of the data relies greatly on its quality. However, data is often dirty in real life. It may be inconsistent, duplicated, stale, inaccurate or incomplete, which can reduce its usability and increase the cost of businesses. Consequently the need for improving data quality arises, which comprises of five central issues of improving data quality, namely, data consistency, data deduplication, data currency, data accuracy and information completeness. This thesis presents the results of our work on the first four issues with regards to data consistency, deduplication, currency and accuracy. The first part of the thesis investigates incremental verifications of data consistencies in distributed data. Given a distributed database D, a set S of conditional functional dependencies (CFDs), the set V of violations of the CFDs in D, and updates ΔD to D, it is to find, with minimum data shipment, changes ΔV to V in response to ΔD. Although the problems are intractable, we show that they are bounded: there exist algorithms to detect errors such that their computational cost and data shipment are both linear in the size of ΔD and ΔV, independent of the size of the database D. Such incremental algorithms are provided for both vertically and horizontally partitioned data, and we show that the algorithms are optimal. The second part of the thesis studies the interaction between record matching and data repairing. Record matching, the main technique underlying data deduplication, aims to identify tuples that refer to the same real-world object, and repairing is to make a database consistent by fixing errors in the data using constraints. These are treated as separate processes in most data cleaning systems, based on heuristic solutions. However, our studies show that repairing can effectively help us identify matches, and vice versa. To capture the interaction, a uniform framework that seamlessly unifies repairing and matching operations is proposed to clean a database based on integrity constraints, matching rules and master data. The third part of the thesis presents our study of finding certain fixes that are absolutely correct for data repairing. Data repairing methods based on integrity constraints are normally heuristic, and they may not find certain fixes. Worse still, they may even introduce new errors when attempting to repair the data, which may not work well when repairing critical data such as medical records, in which a seemingly minor error often has disastrous consequences. We propose a framework and an algorithm to find certain fixes, based on master data, a class of editing rules and user interactions. A prototype system is also developed. The fourth part of the thesis introduces inferring data currency and consistency for conflict resolution, where data currency aims to identify the current values of entities, and conflict resolution is to combine tuples that pertain to the same real-world entity into a single tuple and resolve conflicts, which is also an important issue for data deduplication. We show that data currency and consistency help each other in resolving conflicts. We study a number of associated fundamental problems, and develop an approach for conflict resolution by inferring data currency and consistency. The last part of the thesis reports our study of data accuracy on the longstanding relative accuracy problem which is to determine, given tuples t1 and t2 that refer to the same entity e, whether t1[A] is more accurate than t2[A], i.e., t1[A] is closer to the true value of the A attribute of e than t2[A]. We introduce a class of accuracy rules and an inference system with a chase procedure to deduce relative accuracy, and the related fundamental problems are studied. We also propose a framework and algorithms for inferring accurate values with users’ interaction
Geospatial Narratives and their Spatio-Temporal Dynamics: Commonsense Reasoning for High-level Analyses in Geographic Information Systems
The modelling, analysis, and visualisation of dynamic geospatial phenomena
has been identified as a key developmental challenge for next-generation
Geographic Information Systems (GIS). In this context, the envisaged
paradigmatic extensions to contemporary foundational GIS technology raises
fundamental questions concerning the ontological, formal representational, and
(analytical) computational methods that would underlie their spatial
information theoretic underpinnings.
We present the conceptual overview and architecture for the development of
high-level semantic and qualitative analytical capabilities for dynamic
geospatial domains. Building on formal methods in the areas of commonsense
reasoning, qualitative reasoning, spatial and temporal representation and
reasoning, reasoning about actions and change, and computational models of
narrative, we identify concrete theoretical and practical challenges that
accrue in the context of formal reasoning about `space, events, actions, and
change'. With this as a basis, and within the backdrop of an illustrated
scenario involving the spatio-temporal dynamics of urban narratives, we address
specific problems and solutions techniques chiefly involving `qualitative
abstraction', `data integration and spatial consistency', and `practical
geospatial abduction'. From a broad topical viewpoint, we propose that
next-generation dynamic GIS technology demands a transdisciplinary scientific
perspective that brings together Geography, Artificial Intelligence, and
Cognitive Science.
Keywords: artificial intelligence; cognitive systems; human-computer
interaction; geographic information systems; spatio-temporal dynamics;
computational models of narrative; geospatial analysis; geospatial modelling;
ontology; qualitative spatial modelling and reasoning; spatial assistance
systemsComment: ISPRS International Journal of Geo-Information (ISSN 2220-9964);
Special Issue on: Geospatial Monitoring and Modelling of Environmental
Change}. IJGI. Editor: Duccio Rocchini. (pre-print of article in press
Loan maturity aggregation in interbank lending networks obscures mesoscale structure and economic functions
Since the 2007-2009 financial crisis, substantial academic effort has been dedicated to improving our understanding of interbank lending networks (ILNs). Because of data limitations or by choice, the literature largely lacks multiple loan maturities. We employ a complete interbank loan contract dataset to investigate whether maturity details are informative of the network structure. Applying the layered stochastic block model of Peixoto (2015) and other tools from network science on a time series of bilateral loans with multiple maturity layers in the Russian ILN, we find that collapsing all such layers consistently obscures mesoscale structure. The optimal maturity granularity lies between completely collapsing and completely separating the maturity layers and depends on the development phase of the interbank market, with a more developed market requiring more layers for optimal description. Closer inspection of the inferred maturity bins associated with the optimal maturity granularity reveals specific economic functions, from liquidity intermediation to financing. Collapsing a network with multiple underlying maturity layers or extracting one such layer, common in economic research, is therefore not only an incomplete representation of the ILN's mesoscale structure, but also conceals existing economic functions. This holds important insights and opportunities for theoretical and empirical studies on interbank market functioning, contagion, stability, and on the desirable level of regulatory data disclosure
Unsupervised String Transformation Learning for Entity Consolidation
Data integration has been a long-standing challenge in data management with
many applications. A key step in data integration is entity consolidation. It
takes a collection of clusters of duplicate records as input and produces a
single "golden record" for each cluster, which contains the canonical value for
each attribute. Truth discovery and data fusion methods, as well as Master Data
Management (MDM) systems, can be used for entity consolidation. However, to
achieve better results, the variant values (i.e., values that are logically the
same with different formats) in the clusters need to be consolidated before
applying these methods.
For this purpose, we propose a data-driven method to standardize the variant
values based on two observations: (1) the variant values usually can be
transformed to the same representation (e.g., "Mary Lee" and "Lee, Mary") and
(2) the same transformation often appears repeatedly across different clusters
(e.g., transpose the first and last name). Our approach first uses an
unsupervised method to generate groups of value pairs that can be transformed
in the same way (i.e., they share a transformation). Then the groups are
presented to a human for verification and the approved ones are used to
standardize the data. In a real-world dataset with 17,497 records, our method
achieved 75% recall and 99.5% precision in standardizing variant values by
asking a human 100 yes/no questions, which completely outperformed a state of
the art data wrangling tool
Executive control and decision-making in the prefrontal cortex
The prefrontal cortex (PFC) subserves decision-making and executive control. Here we review recent empirical and modeling works with a focus on neuroimaging studies, which start unifying these two conceptual approaches of PFC function. We propose that the PFC comprises two arbitration systems: (1) a peripheral system comprising premotor/caudal PFC regions and orbitofrontal regions involved in the selection of actions based on perceptual cues and reward values, respectively, and embedded in behavioral sets associated with external contingencies inferred as being stable; (2) a core system comprising ventromedial, dorsomedial, lateral and polar PFC regions involved in superordinate probabilistic reasoning for arbitrating online between exploiting/adjusting previously learned behavioral sets and exploring/creating new ones for efficient adaptive behavior in variable and open-ended environments
Antecedents of trust across foci: a comparative study of Turkey and China
Instead of importing Western models of interpersonal trust, we adopted a qualitative approach to understand trust relationships from indigenous cultures' perspectives. We examined trust relationships directed at different foci in the organization (supervisor, peer, and subordinate) in two different countries, Turkey and China. Semi-structured interviews were conducted with 30 Turkish and 30 Chinese employees working for a variety of large-scale organizations located in Istanbul, Turkey and Shenzhen, China. We report the content analysis of trust-building critical incidents narrated by the respondents. While the general antecedents of Ability, Benevolence, and Integrity were found to exist in both countries, Benevolence, with its culture-specific manifestations, played a very important role in trust-building across multiple foci in both countries. We also found that trust relationships in these two contexts tended to go beyond the professional domain, and to involve sharing of personal time, information, and space. Drawing on this evidence, we propose a trust-building process that is more affective in nature and which straddles both work and non-work domains
Improving National and Homeland Security through a proposed Laboratory for Information Globalization and Harmonization Technologies (LIGHT)
A recent National Research Council study found that: "Although there are many private and public databases that
contain information potentially relevant to counter terrorism programs, they lack the necessary context definitions
(i.e., metadata) and access tools to enable interoperation with other databases and the extraction of meaningful and
timely information" [NRC02, p.304, emphasis added] That sentence succinctly describes the objectives of this
project. Improved access and use of information are essential to better identify and anticipate threats, protect
against and respond to threats, and enhance national and homeland security (NHS), as well as other national
priority areas, such as Economic Prosperity and a Vibrant Civil Society (ECS) and Advances in Science and
Engineering (ASE). This project focuses on the creation and contributions of a Laboratory for Information
Globalization and Harmonization Technologies (LIGHT) with two interrelated goals:
(1) Theory and Technologies: To research, design, develop, test, and implement theory and technologies for
improving the reliability, quality, and responsiveness of automated mechanisms for reasoning and resolving semantic
differences that hinder the rapid and effective integration (int) of systems and data (dmc) across multiple
autonomous sources, and the use of that information by public and private agencies involved in national and
homeland security and the other national priority areas involving complex and interdependent social systems (soc).
This work builds on our research on the COntext INterchange (COIN) project, which focused on the integration
of diverse distributed heterogeneous information sources using ontologies, databases, context mediation algorithms,
and wrapper technologies to overcome information representational conflicts. The COIN approach makes it
substantially easier and more transparent for individual receivers (e.g., applications, users) to access and exploit
distributed sources. Receivers specify their desired context to reduce ambiguities in the interpretation of information
coming from heterogeneous sources. This approach significantly reduces the overhead involved in the integration of
multiple sources, improves data quality, increases the speed of integration, and simplifies maintenance in an
environment of changing source and receiver context - which will lead to an effective and novel distributed
information grid infrastructure. This research also builds on our Global System for Sustainable Development
(GSSD), an Internet platform for information generation, provision, and integration of multiple domains, regions,
languages, and epistemologies relevant to international relations and national security.
(2) National Priority Studies: To experiment with and test the developed theory and technologies on practical
problems of data integration in national priority areas. Particular focus will be on national and homeland security,
including data sources about conflict and war, modes of instability and threat, international and regional
demographic, economic, and military statistics, money flows, and contextualizing terrorism defense and response.
Although LIGHT will leverage the results of our successful prior research projects, this will be the first research
effort to simultaneously and effectively address ontological and temporal information conflicts as well as
dramatically enhance information quality. Addressing problems of national priorities in such rapidly changing
complex environments requires extraction of observations from disparate sources, using different interpretations, at
different points in times, for different purposes, with different biases, and for a wide range of different uses and
users. This research will focus on integrating information both over individual domains and across multiple domains.
Another innovation is the concept and implementation of Collaborative Domain Spaces (CDS), within which
applications in a common domain can share, analyze, modify, and develop information. Applications also can span
multiple domains via Linked CDSs. The PIs have considerable experience with these research areas and the
organization and management of such large scale international and diverse research projects.
The PIs come from three different Schools at MIT: Management, Engineering, and Humanities, Arts & Social
Sciences. The faculty and graduate students come from about a dozen nationalities and diverse ethnic, racial, and
religious backgrounds. The currently identified external collaborators come from over 20 different organizations
and many different countries, industrial as well as developing. Specific efforts are proposed to engage even more
women, underrepresented minorities, and persons with disabilities.
The anticipated results apply to any complex domain that relies on heterogeneous distributed data to address and
resolve compelling problems. This initiative is supported by international collaborators from (a) scientific and
research institutions, (b) business and industry, and (c) national and international agencies. Research products
include: a System for Harmonized Information Processing (SHIP), a software platform, and diverse applications in
research and education which are anticipated to significantly impact the way complex organizations, and society in
general, understand and manage critical challenges in NHS, ECS, and ASE
- …