Reason Maintenance - Conceptual Framework

This paper describes the conceptual framework for reason maintenance developed as part of WP2

Can Large Language Models Infer and Disagree Like Humans?

Large Language Models (LLMs) have shown stellar achievements in solving a broad range of tasks. When generating text, it is common to sample tokens from these models: whether LLMs closely align with the human disagreement distribution has not been well-studied, especially within the scope of Natural Language Inference (NLI). In this paper, we evaluate the performance and alignment of LLM distribution with humans using two different techniques: Monte Carlo Reconstruction (MCR) and Log Probability Reconstruction (LPR). As a result, we show LLMs exhibit limited ability in solving NLI tasks and simultaneously fail to capture human disagreement distribution, raising concerns about their natural language understanding (NLU) ability and their representativeness of human users

The Ontological Foundations of the Debate over Originalism

Because the participants in the debate over constitutional originalism generally understand the controversy to be over a matter of the objective truth of competing interpretations of the Constitution, they do not believe that their mission is to persuade the other side. When what is at stake is a matter of objective truth, subjective opinions are of less moment. This Article begins the long overdue transcendence of our increasingly fruitless and acrimonious debate over originalism by articulating the tacit philosophical premises that make the debate possible. It demonstrates that originalism, despite its pretensions to common sense and its disavowal of abstruse philosophical analysis, is tacitly committed to three key ontological and linguistic premises. First, language represents the world. Second, propositions or statements are true if they accurately (truly) represent that world. Thus, propositions of constitutional law represent the constitutional world. As a consequence, propositions or statements of constitutional law are true if they accurately (truly) represent that constitutional world. Third, there is an ontologically independent Constitution that our constitutional interpretation describes. For the originalist, that objective Constitution is the semantic understanding of the constitutional provisions when they were originally adopted or amended. Moreover, surprisingly, originalism’s critics are also committed to these same premises about the nature of language, the nature of truth and the existence of an objective Constitution. Originalism’s critics assert that the objective Constitution has sources beyond the original understanding of its provisions. These shared premises about the nature of language and the nature of the Constitution permit the debate over originalism to proceed as a debate about the objective truth of constitutional interpretations and the accuracy of each side’s description of the objective facts about the Constitution. Because both sides of the debate believe there to be an objective answer to the questions they address, the debate can focus upon defending the account of the relevant interpretation rather than on persuading the other side. Understanding that fundamental dynamic to the debate helps explain why it has been so unproductive. Moreover, understanding that the debate over originalism is only possible if these premises are true highlights the underlying question whether such premises are indeed correct

A reformed division of labor for the science of well-being

This paper provides a philosophical assessment of leading theory-based, evidence-based and coherentist approaches to the definition and the measurement of wellbeing. It then builds on this assessment to articulate a reformed division of labor for the science of well-being and argues that this reformed division of labor can improve on the proffered approaches by combining the most plausible tenets of theory-based approaches with the most plausible tenets of coherentist approaches. This result does not per se exclude the possibility that theory-based and coherentist approaches may be independently improved or amended in the years to come. Still, together with the challenges that affect these approaches, it strengthens the case for combining the most plausible tenets of those approaches

Uncertainty-Aware Natural Language Inference with Stochastic Weight Averaging

This paper introduces Bayesian uncertainty modeling using Stochastic Weight Averaging-Gaussian (SWAG) in Natural Language Understanding (NLU) tasks. We apply the approach to standard tasks in natural language inference (NLI) and demonstrate the effectiveness of the method in terms of prediction accuracy and correlation with human annotation disagreements. We argue that the uncertainty representations in SWAG better reflect subjective interpretation and the natural variation that is also present in human language understanding. The results reveal the importance of uncertainty modeling, an often neglected aspect of neural language modeling, in NLU tasks.Comment: NoDaLiDa 2023 camera read

Conformalized Credal Set Predictors

Credal sets are sets of probability distributions that are considered as candidates for an imprecisely known ground-truth distribution. In machine learning, they have recently attracted attention as an appealing formalism for uncertainty representation, in particular due to their ability to represent both the aleatoric and epistemic uncertainty in a prediction. However, the design of methods for learning credal set predictors remains a challenging problem. In this paper, we make use of conformal prediction for this purpose. More specifically, we propose a method for predicting credal sets in the classification task, given training data labeled by probability distributions. Since our method inherits the coverage guarantees of conformal prediction, our conformal credal sets are guaranteed to be valid with high probability (without any assumptions on model or distribution). We demonstrate the applicability of our method to natural language inference, a highly ambiguous natural language task where it is common to obtain multiple annotations per example

Etiquetador automático de Marcadores Discursivos mediante Transformers

We present an automatic discourse particle (DM) tagger developed using manual annotation and machine learning. The tagger has been developed on a dataset of financial letters, where human annotators have reached an 0.897 agreement rate (IAA) on the indications of a specific annotation guide. With the annotated dataset, a prototype has been developed using the pre-trained Transformers, adapting it to the task (fine-tunning), reaching an F1-score of 0.933. An evaluation of the results obtained by the tagger is included.Presentamos un etiquetador automático de partículas discursivas (DM) desarrollado mediante etiquetado manual y aprendizaje automático. El etiquetador se ha desarrollado en un dataset de cartas financieras. Las anotadoras humanas han alcanzado un 0,897 de tasa de acuerdo (IAA) sobre las indicaciones de una guía de anotación específica. Con el dataset anotado se ha desarrollado un prototipo usando modelos de Transformers pre-entrenados adaptándolos a la tarea (fine-tuning) con un F1 de 0,933. Al final se da una evaluación de los resultados obtenidos por el tagger.The research has been carried out within the CLARA-FINT project (PID2020-116001RB-C31), funded by the Spanish Ministry of Science and Innovation