REVISITING RECOGNIZING TEXTUAL ENTAILMENT FOR EVALUATING NATURAL LANGUAGE PROCESSING SYSTEMS

Recognizing Textual Entailment (RTE) began as a unified framework to evaluate the reasoning capabilities of Natural Language Processing (NLP) models. In recent years, RTE has evolved in the NLP community into a task that researchers focus on developing models for. This thesis revisits the tradition of RTE as an evaluation framework for NLP models, especially in the era of deep learning. Chapter 2 provides an overview of different approaches to evaluating NLP sys- tems, discusses prior RTE datasets, and argues why many of them do not serve as satisfactory tests to evaluate the reasoning capabilities of NLP systems. Chapter 3 presents a new large-scale diverse collection of RTE datasets (DNC) that tests how well NLP systems capture a range of semantic phenomena that are integral to un- derstanding human language. Chapter 4 demonstrates how the DNC can be used to evaluate reasoning capabilities of NLP models. Chapter 5 discusses the limits of RTE as an evaluation framework by illuminating how existing datasets contain biases that may enable crude modeling approaches to perform surprisingly well. The remaining aspects of the thesis focus on issues raised in Chapter 5. Chapter 6 addresses issues in prior RTE datasets focused on paraphrasing and presents a high-quality test set that can be used to analyze how robust RTE systems are to paraphrases. Chapter 7 demonstrates how modeling approaches on biases, e.g. adversarial learning, can enable RTE models overcome biases discussed in Chapter 5. Chapter 8 applies these methods to the task of discovering emergency needs during disaster events

The directive function of the English modals

The aim of this thesis is to provide a detailed account, within a 'systemic' framework of those properties of English sentences containing modal verbs, which will allow us to make predictions about the potential directiveness of some such sentences but not others, about ambiguity of communicative function, and about certain social properties of directives. Part I develops a model suitable for describing all the relevant aspects of modalised directives. We argue that no systemic model so far proposed Is, by itself, adequate for this task. We also show that the communicative function of an utterance is to be accounted for, not at the semantic level, but in terms of discourse function. Illocutionary properties are seen as relevant to the Interpretation of discourse function from the meanings of sentences uttered in contexts. A multi-level model, based on the principles of Hudson's 'daughter dependency' grammar, is proposed. Part II provides descriptions of three areas crucial to an account of modalised directives, using the framework set up in Part I. A network and realisation rules for the discourse level are proposed, and the role of directives in discourse discussed. There follows a formalised account of the semantic properties underlying mood, and the meanings of the modals. In Part III we predict, from the semantics of mood and modalisatlon, which modalised sentences will be acceptable as directives, and which of the acceptable sentences will be classified as orders, requests and suggestions, when used directively In a given social context. We also predict that, again in a given social context, certain forms of directive will be regarded as more polite than others. The results of an informant programme designed to test these hypotheses are then presented, and found to corroborate very strongly the predictions made