Native Language Identification with Big Bird Embeddings

Native Language Identification (NLI) intends to classify an author's native language based on their writing in another language. Historically, the task has heavily relied on time-consuming linguistic feature engineering, and transformer-based NLI models have thus far failed to offer effective, practical alternatives. The current work investigates if input size is a limiting factor, and shows that classifiers trained using Big Bird embeddings outperform linguistic feature engineering models by a large margin on the Reddit-L2 dataset. Additionally, we provide further insight into input length dependencies, show consistent out-of-sample performance, and qualitatively analyze the embedding space. Given the effectiveness and computational efficiency of this method, we believe it offers a promising avenue for future NLI work

Retrieving Supporting Evidence for Generative Question Answering

Current large language models (LLMs) can exhibit near-human levels of performance on many natural language-based tasks, including open-domain question answering. Unfortunately, at this time, they also convincingly hallucinate incorrect answers, so that responses to questions must be verified against external sources before they can be accepted at face value. In this paper, we report two simple experiments to automatically validate generated answers against a corpus. We base our experiments on questions and passages from the MS MARCO (V1) test collection, and a retrieval pipeline consisting of sparse retrieval, dense retrieval and neural rerankers. In the first experiment, we validate the generated answer in its entirety. After presenting a question to an LLM and receiving a generated answer, we query the corpus with the combination of the question + generated answer. We then present the LLM with the combination of the question + generated answer + retrieved answer, prompting it to indicate if the generated answer can be supported by the retrieved answer. In the second experiment, we consider the generated answer at a more granular level, prompting the LLM to extract a list of factual statements from the answer and verifying each statement separately. We query the corpus with each factual statement and then present the LLM with the statement and the corresponding retrieved evidence. The LLM is prompted to indicate if the statement can be supported and make necessary edits using the retrieved material. With an accuracy of over 80%, we find that an LLM is capable of verifying its generated answer when a corpus of supporting material is provided. However, manual assessment of a random sample of questions reveals that incorrect generated answers are missed by this verification process. While this verification process can reduce hallucinations, it can not entirely eliminate them.Comment: arXiv admin note: text overlap with arXiv:2306.1378

Personalized information retrieval based on context and ontological knowledge

The article has been accepted for publication and appeared in a revised form, subsequent to peer review and/or editorial input by Cambridge University PressExtended papers from C&O-2006, the second International Workshop on Contexts and Ontologies, Theory, Practice and Applications1 collocated with the seventeenth European Conference on Artificial Intelligence (ECAI)Context modeling has been long acknowledged as a key aspect in a wide variety of problem domains. In this paper we focus on the combination of contextualization and personalization methods to improve the performance of personalized information retrieval. The key aspects in our proposed approach are a) the explicit distinction between historic user context and live user context, b) the use of ontology-driven representations of the domain of discourse, as a common, enriched representational ground for content meaning, user interests, and contextual conditions, enabling the definition of effective means to relate the three of them, and c) the introduction of fuzzy representations as an instrument to properly handle the uncertainty and imprecision involved in the automatic interpretation of meanings, user attention, and user wishes. Based on a formal grounding at the representational level, we propose methods for the automatic extraction of persistent semantic user preferences, and live, ad-hoc user interests, which are combined in order to improve the accuracy and reliability of personalization for retrieval.This research was partially supported by the European Commission under contracts FP6-001765 aceMedia and FP6-027685 MESH. The expressed content is the view of the authors but not necessarily the view of the aceMedia or MESH projects as a whole

Extrinsic Summarization Evaluation: A Decision Audit Task

Abstract. In this work we describe a large-scale extrinsic evaluation of automatic speech summarization technologies for meeting speech. The particular task is a decision audit, wherein a user must satisfy a complex information need, navigating several meetings in order to gain an understanding of how and why a given decision was made. We compare the usefulness of extractive and abstractive technologies in satisfying this information need, and assess the impact of automatic speech recognition (ASR) errors on user performance. We employ several evaluation methods for participant performance, including post-questionnaire data, human subjective and objective judgments, and an analysis of participant browsing behaviour.

Retrieving Supporting Evidence for Generative Question Answering

Current large language models (LLMs) can exhibit near-human levels of performance on many natural language-based tasks, including open-domain question answering. Unfortunately, at this time, they also convincingly hallucinate incorrect answers, so that responses to questions must be verified against external sources before they can be accepted at face value. In the thesis, I report two simple experiments to automatically validate generated answers against a corpus. We base our experiments on questions and passages from the MS MARCO (V1) test collection, and a retrieval pipeline consisting of sparse retrieval, dense retrieval and neural rerankers. In the first experiment, we validate the generated answer in its entirety. After presenting a question to an LLM and receiving a generated answer, we query the corpus with the combination of the question + generated answer. We then present the LLM with the combination of the question + generated answer + retrieved answer, prompting it to indicate if the generated answer can be supported by the retrieved answer. In the second experiment, we consider the generated answer at a more granular level, prompting the LLM to extract a list of factual statements from the answer and verifying each statement separately. We query the corpus with each factual statement and then present the LLM with the statement and the corresponding retrieved evidence. The LLM is prompted to indicate if the statement can be supported and make necessary edits using the retrieved material. With an accuracy of over 80%, we find that an LLM is capable of verifying its generated answer when a corpus of supporting material is provided. However, manual assessment of a random sample of questions reveals that incorrect generated answers are missed by this verification process. While this verification process can reduce hallucinations, it can not entirely eliminate them

Unveiling AI Aversion: Understanding Antecedents and Task Complexity Effects

Artificial Intelligence (AI) has generated significant interest due to its potential to augment human intelligence. However, user attitudes towards AI are diverse, with some individuals embracing it enthusiastically while others harbor concerns and actively avoid its use. This two essays\u27 dissertation explores the reasons behind user aversion to AI. In the first essay, I develop a concise research model to explain users\u27 AI aversion based on the theory of effective use and the adaptive structuration theory. I then employ an online experiment to test my hypotheses empirically. The multigroup analysis by Structural Equation Modeling shows that users\u27 perceptions of human dissimilarity, AI bias, and social influence strongly drive AI aversion. Moreover, I find a significant difference between the simple and the complex task groups. This study reveals why users avert using AI by systematically examining the factors related to technology, user, task, and environment, thus making a significant contribution to the emerging field of AI aversion research. Next, while trust and distrust have been recognized as influential factors shaping users\u27 attitudes towards IT artifacts, their intricate relationship with task characteristics and their impact on AI aversion remains largely unexplored. In my second essay, I conduct an online randomized controlled experiment on Amazon Mechanical Turk to bridge this critical research gap. My comprehensive analytic approach, including structural equation modeling (SEM), ANOVA, and PROCESS conditional analysis, allowed me to shed light on the intricate web of factors influencing users\u27 AI aversion. I discovered that distrust and trust mediate between task complexity and AI aversion. Moreover, this study unveiled intriguing differences in these mediated relationships between subjective and objective task groups. Specifically, my findings demonstrate that, for objective tasks, task complexity can significantly increase aversion by reducing trust and significantly decrease aversion by reducing distrust. In contrast, for subjective tasks, task complexity only significantly increases aversion by enhancing distrust. By considering various task characteristics and recognizing trust and distrust as vital mediators, my research not only pushes the boundaries of the human-AI literature but also significantly contributes to the field of AI aversion

Internet search techniques: using word count, links and directory structure as internet search tools

A thesis submitted for the degree of Doctor of Philosophy ofthe University of LutonAs the Web grows in size it becomes increasingly important that ways are developed to maximise the efficiency of the search process and index its contents with minimal human intervention. An evaluation is undertaken of current popular search engines which use a centralised index approach. Using a number of search terms and metrics that measure similarity between sets of results, it was found that there is very little commonality between the outcome of the same search performed using different search engines. A semi-automated system for searching the web is presented, the Internet Search Agent (ISA), this employs a method for indexing based upon the idea of "fingerprint types". These fingerprint types are based upon the text and links contained in the web pages being indexed. Three examples of fingerprint type are developed, the first concentrating upon the textual content of the indexed files, the other two augment this with the use of links to and from these files. By looking at the results returned as a search progresses in terms of numbers and measures of content of results for effort expended, comparisons can be made between the three fingerprint types. The ISA model allows the searcher to be presented with results in context and potentially allows for distributed searching to be implemented

Structuration sématique de documents XML centres-documents

La numérisation des documents et le développement des technologies Internet ont engendré une augmentation permanente du nombre de documents et de types de documents disponibles. Face à cette masse documentaire, XML (eXtensible Markup Language) s’est imposé comme format standard de structuration et d’échange de documents. Ainsi, un nombre de plus en plus important de documents devient disponible sous ce format. Ces documents XML peuvent être classés en deux types : les documents XML orienté-données et les documents XML orienté-textes. Les documents XML orienté-données sont constitués d’un ensemble d’éléments généralement courts et précis et sont similaires aux données relationnelles. Nous constatons que les balises utilisées pour ce type de documents décrivent généralement d’une manière précise le contenu, et offrent la sémantique basique nécessaire à la description de l’information (Exemples de balises : Article, Client, Quantité, Prix). A contrario, les documents XML orienté-textes sont riches en texte et utilisent des balises qui reflètent la plupart du temps un découpage (structurel) logique (exemples de balises : Contenu, Section, Paragraphe). Malheureusement, ces balises n’ont qu’une très pauvre vocation sémantique. Partant de cette constatation, le développement d’approches supportées par des outils automatisés permettant de décrire la sémantique des documents XML orientés-textes devient un besoin urgent, voire une nécessité pour certains usages. Dans ce contexte, nous proposons une approche de structuration sémantique des documents XML à partir de leurs structures logiques et de leurs contenus. Elle construit une arborescence de concepts. Cette approche de structuration sémantique passe par quatre phases : 1) Extraction des termes des contenus des documents en utilisant des techniques de recherche d’information ; 2) Détermination d’une taxonomie1 qui sera affectée au document, c’est-à-dire celle qui correspond au mieux à sa sémantique (cette étape se base sur une démarche de pondération d’un ensemble de taxonomies candidates) ; 3) Affectation, à chaque élément feuille de la structure logique du document, du concept le plus significatif à partir de la taxonomie retenue ; 4) Inférence de concepts aux éléments non feuilles du document. Notre approche de structuration sémantique des documents se base sur l’indexation sémantique et diffère des autres travaux par : 1) Le choix d’une taxonomie appropriée pour chaque document, il s’agit de déterminer la taxonomie qui décrit au mieux la sémantique du document, et 2) La pondération des concepts extraits de manière à donner plus d’importance aux concepts les plus spécifiques car nous partons du constat suivant : plus le niveau auquel se situe le concept est bas dans la hiérarchie, plus l’information qu’il apporte est fine et ciblée. Pour exploiter ces structures sémantiques, nous avons étendu le méta-modèle d’entrepôts de documents pour assurer leur stockage. De plus, nous avons introduit le concept de métadocument afin de permettre l’interrogation de ces structures sémantiques. Enfin, pour évaluer nos propositions, nous avons mené un ensemble d’expérimentations sur la collection de documents XML ImageCLEFMed 2010 en utilisant la ressource sémantique MeSH (NML's Medical Subject Headings). Les résultats obtenus montrent que l’algorithme de pondération des concepts des taxonomies qui a été proposé permet de sélectionner avec précision la taxonomie pertinente pour un document donné et, en conséquence, les concepts pertinents à affecter aux éléments feuilles de la structure sémantique de ce document.Le résumé en anglais n'a pas été communiqué par l'auteur

Structuration sématique de documents XML centres-documents

La numérisation des documents et le développement des technologies Internet ont engendré une augmentation permanente du nombre de documents et de types de documents disponibles. Face à cette masse documentaire, XML (eXtensible Markup Language) s’est imposé comme format standard de structuration et d’échange de documents. Ainsi, un nombre de plus en plus important de documents devient disponible sous ce format. Ces documents XML peuvent être classés en deux types : les documents XML orienté-données et les documents XML orienté-textes. Les documents XML orienté-données sont constitués d’un ensemble d’éléments généralement courts et précis et sont similaires aux données relationnelles. Nous constatons que les balises utilisées pour ce type de documents décrivent généralement d’une manière précise le contenu, et offrent la sémantique basique nécessaire à la description de l’information (Exemples de balises : Article, Client, Quantité, Prix). A contrario, les documents XML orienté-textes sont riches en texte et utilisent des balises qui reflètent la plupart du temps un découpage (structurel) logique (exemples de balises : Contenu, Section, Paragraphe). Malheureusement, ces balises n’ont qu’une très pauvre vocation sémantique. Partant de cette constatation, le développement d’approches supportées par des outils automatisés permettant de décrire la sémantique des documents XML orientés-textes devient un besoin urgent, voire une nécessité pour certains usages. Dans ce contexte, nous proposons une approche de structuration sémantique des documents XML à partir de leurs structures logiques et de leurs contenus. Elle construit une arborescence de concepts. Cette approche de structuration sémantique passe par quatre phases : 1) Extraction des termes des contenus des documents en utilisant des techniques de recherche d’information ; 2) Détermination d’une taxonomie1 qui sera affectée au document, c’est-à-dire celle qui correspond au mieux à sa sémantique (cette étape se base sur une démarche de pondération d’un ensemble de taxonomies candidates) ; 3) Affectation, à chaque élément feuille de la structure logique du document, du concept le plus significatif à partir de la taxonomie retenue ; 4) Inférence de concepts aux éléments non feuilles du document. Notre approche de structuration sémantique des documents se base sur l’indexation sémantique et diffère des autres travaux par : 1) Le choix d’une taxonomie appropriée pour chaque document, il s’agit de déterminer la taxonomie qui décrit au mieux la sémantique du document, et 2) La pondération des concepts extraits de manière à donner plus d’importance aux concepts les plus spécifiques car nous partons du constat suivant : plus le niveau auquel se situe le concept est bas dans la hiérarchie, plus l’information qu’il apporte est fine et ciblée. Pour exploiter ces structures sémantiques, nous avons étendu le méta-modèle d’entrepôts de documents pour assurer leur stockage. De plus, nous avons introduit le concept de métadocument afin de permettre l’interrogation de ces structures sémantiques. Enfin, pour évaluer nos propositions, nous avons mené un ensemble d’expérimentations sur la collection de documents XML ImageCLEFMed 2010 en utilisant la ressource sémantique MeSH (NML's Medical Subject Headings). Les résultats obtenus montrent que l’algorithme de pondération des concepts des taxonomies qui a été proposé permet de sélectionner avec précision la taxonomie pertinente pour un document donné et, en conséquence, les concepts pertinents à affecter aux éléments feuilles de la structure sémantique de ce document.Le résumé en anglais n'a pas été communiqué par l'auteur

Interactive video retrieval

Video storage, analysis, and retrieval has become an important research topic recently due to the advancements in the creation and distribution of video data. In this thesis, an investigation into interactive video retrieval is presented. Advanced feedback techniques have been investigated in the retrieval of textual data. Novel interactive schemes, mainly based on the concept of relevance feedback, have been developed and experimented. However, such approaches have not been applied in the video retrieval domain. In this thesis, we investigate the use of advanced interactive retrieval schemes for the retrieval of video data. To understand the role of various features for the video retrieval, we experimented with various retrieval strategies. We benchmarked the role of visual features, the textual features and their combination. To explore this further, we categorized query into various classes and investigated the retrieval effectiveness of various features and their combination. Based on the results, we developed a retrieval scheme for video retrieval. We developed an interactive retrieval technique based on the concept of implicit feedback. A number of retrieval models are developed based on this concept and benchmarked with a simulation- based evaluation strategy. A Binary Voting Model performed well and has been reformed for user-based experiments. We experimented with the users and compared the performance of an interactive retrieval system, using a combination of implicit and explicit feedback techniques, with that of a system using explicit feedback techniques