Application of aboutness to functional benchmarking in information retrieval

Experimental approaches are widely employed to benchmark the performance of an information retrieval (IR) system. Measurements in terms of recall and precision are computed as performance indicators. Although they are good at assessing the retrieval effectiveness of an IR system, they fail to explore deeper aspects such as its underlying functionality and explain why the system shows such performance. Recently, inductive (i.e., theoretical) evaluation of IR systems has been proposed to circumvent the controversies of the experimental methods. Several studies have adopted the inductive approach, but they mostly focus on theoretical modeling of IR properties by using some metalogic. In this article, we propose to use inductive evaluation for functional benchmarking of IR models as a complement of the traditional experiment-based performance benchmarking. We define a functional benchmark suite in two stages: the evaluation criteria based on the notion of "aboutness," and the formal evaluation methodology using the criteria. The proposed benchmark has been successfully applied to evaluate various well-known classical and logic-based IR models. The functional benchmarking results allow us to compare and analyze the functionality of the different IR models

Theoretical evaluation of XML retrieval

This thesis develops a theoretical framework to evaluate XML retrieval. XML retrieval deals with retrieving those document parts that specifically answer a query. It is concerned with using the document structure to improve the retrieval of information from documents by only delivering those parts of a document an information need is about. We define a theoretical evaluation methodology based on the idea of `aboutness' and apply it to XML retrieval models. Situation Theory is used to express the aboutness proprieties of XML retrieval models. We develop a dedicated methodology for the evaluation of XML retrieval and apply this methodology to five XML retrieval models and other XML retrieval topics such as evaluation methodologies, filters and experimental results

Towards a belief revision based adaptive and context sensitive information retrieval system

In an adaptive information retrieval (IR) setting, the information seekers' beliefs about which terms are relevant or nonrelevant will naturally fluctuate. This article investigates how the theory of belief revision can be used to model adaptive IR. More specifically, belief revision logic provides a rich representation scheme to formalize retrieval contexts so as to disambiguate vague user queries. In addition, belief revision theory underpins the development of an effective mechanism to revise user profiles in accordance with information seekers' changing information needs. It is argued that information retrieval contexts can be extracted by means of the information-flow text mining method so as to realize a highly autonomous adaptive IR system. The extra bonus of a belief-based IR model is that its retrieval behavior is more predictable and explanatory. Our initial experiments show that the belief-based adaptive IR system is as effective as a classical adaptive IR system. To our best knowledge, this is the first successful implementation and evaluation of a logic-based adaptive IR model which can efficiently process large IR collections

Relevance, Rhetoric, and Argumentation: A Cross-Disciplinary Inquiry into Patterns of Thinking and Information Structuring

This dissertation research is a multidisciplinary inquiry into topicality, involving an in-depth examination of literatures and empirical data and an inductive development of a faceted typology (containing 227 fine-grained topical relevance relationships and 33 types of presentation relationship). This inquiry investigates a large variety of topical connections beyond topic matching, renders a closer look into the structure of a topic, achieves an enriched understanding of topicality and relevance, and induces a cohesive topic-oriented information architecture that is meaningful across topics and domains. The findings from the analysis contribute to the foundation work of information organization, intellectual access / information retrieval, and knowledge discovery. Using qualitative content analysis, the inquiry focuses on meaning and deep structure: Phase 1 : develop a unified theory-grounded typology of topical relevance relationships through close reading of literature and synthesis of thinking from communication, rhetoric, cognitive psychology, education, information science, argumentation, logic, law, medicine, and art history; Phase 2 : in-depth qualitative analysis of empirical relevance datasets in oral history, clinical question answering, and art image tagging, to examine manifestations of the theory-grounded typology in various contexts and to further refine the typology; the three relevance datasets were used for analysis to achieve variation in form, domain, and context. The typology of topical relevance relationships is structured with three major facets: Functional role of a piece of information plays in the overall structure of a topic or an argument; Mode of reasoning: How information contributes to the user's reasoning about a topic; Semantic relationship: How information connects to a topic semantically. This inquiry demonstrated that topical relevance with its close linkage to thinking and reasoning is central to many disciplines. The multidisciplinary approach allows synthesis and examination from new angles, leading to an integrated scheme of relevance relationships or a system of thinking that informs each individual discipline. The scheme resolving from the synthesis can be used to improve text and image understanding, knowledge organization and retrieval, reasoning, argumentation, and thinking in general, by people and machines

Semantic Model Alignment for Business Process Integration

Business process models describe an enterprise’s way of conducting business and in this form the basis for shaping the organization and engineering the appropriate supporting or even enabling IT. Thereby, a major task in working with models is their analysis and comparison for the purpose of aligning them. As models can differ semantically not only concerning the modeling languages used, but even more so in the way in which the natural language for labeling the model elements has been applied, the correct identification of the intended meaning of a legacy model is a non-trivial task that thus far has only been solved by humans. In particular at the time of reorganizations, the set-up of B2B-collaborations or mergers and acquisitions the semantic analysis of models of different origin that need to be consolidated is a manual effort that is not only tedious and error-prone but also time consuming and costly and often even repetitive. For facilitating automation of this task by means of IT, in this thesis the new method of Semantic Model Alignment is presented. Its application enables to extract and formalize the semantics of models for relating them based on the modeling language used and determining similarities based on the natural language used in model element labels. The resulting alignment supports model-based semantic business process integration. The research conducted is based on a design-science oriented approach and the method developed has been created together with all its enabling artifacts. These results have been published as the research progressed and are presented here in this thesis based on a selection of peer reviewed publications comprehensively describing the various aspects

Explainable Information Retrieval: A Survey

Explainable information retrieval is an emerging research area aiming to make transparent and trustworthy information retrieval systems. Given the increasing use of complex machine learning models in search systems, explainability is essential in building and auditing responsible information retrieval models. This survey fills a vital gap in the otherwise topically diverse literature of explainable information retrieval. It categorizes and discusses recent explainability methods developed for different application domains in information retrieval, providing a common framework and unifying perspectives. In addition, it reflects on the common concern of evaluating explanations and highlights open challenges and opportunities.Comment: 35 pages, 10 figures. Under revie

LAW SEARCH IN THE AGE OF THE ALGORITHM

The process of searching for relevant legal materials is fundamental to legal reasoning. However, despite its enormous practical and theoretical importance, law search has not been given significant attention by scholars. In this Article, we define the problem of law search and examine the consequences of new technologies capable of automating this core lawyerly task. We introduce a theory of law search in which legal relevance is a sociological phenomenon that leads to convergence over a shared set of legal materials and explore the normative stakes of law search. We examine ways in which law scholars can understand empirically the phenomenon of law search, argue that computational modeling is a valuable epistemic tool in this domain, and report the results from a multi-year, interdisciplinary effort to develop an advanced law search algorithm based on human-generated data. Finally, we explore how policymakers can manage the challenges posed by new machine learning-based search technologies

Articulating Digital Archival Practice Within Writing Program Administration: A Theoretical Framework

Throughout Writing Program Administration scholarship there has been a clear call for archivization and archival work. This dissertation project takes an interdisciplinary approach to digital archival practices for Writing Program Administrators to consider and employ in their home institutions. While I recognize that WPAs are not typically identified as “archivists,” I situate the digital archive within the digital humanities as an interdisciplinary, collaborative project and offer suggestions that lead to recommendations for making an institutional archive. I review archival practice in order to justify the digital archive as an appropriate vehicle for WPAs’ work. Further, I argue that the digital archive must be useable and, therefore, consider other commonly used composition studies archives for their usability. Overall, my dissertation seeks to define digital archival practice for WPAs in order to inspire other educators to take up this meaningful, historical work