Collecting Diverse Natural Language Inference Problems for Sentence Representation Evaluation

We present a large-scale collection of diverse natural language inference (NLI) datasets that help provide insight into how well a sentence representation captures distinct types of reasoning. The collection results from recasting 13 existing datasets from 7 semantic phenomena into a common NLI structure, resulting in over half a million labeled context-hypothesis pairs in total. We refer to our collection as the DNC: Diverse Natural Language Inference Collection. The DNC is available online at https://www.decomp.net, and will grow over time as additional resources are recast and added from novel sources.Comment: To be presented at EMNLP 2018. 15 page

Position statement: Inference in Question Answering

One can only exploit inference in Question-Answering (QA) and assess its contribution systematically, if one knows what inference is contributing to. Thus we identify a set of tasks specific to QA and discuss what inference could contribute to their achievement. We conclude with a proposal for graduated test suites as a tool for assessing the performance and impact of inference

Type B Reflexivization as an Unambiguous Testbed for Multilingual Multi-Task Gender Bias

The one-sided focus on English in previous studies of gender bias in NLP misses out on opportunities in other languages: English challenge datasets such as GAP and WinoGender highlight model preferences that are "hallucinatory", e.g., disambiguating gender-ambiguous occurrences of 'doctor' as male doctors. We show that for languages with type B reflexivization, e.g., Swedish and Russian, we can construct multi-task challenge datasets for detecting gender bias that lead to unambiguously wrong model predictions: In these languages, the direct translation of 'the doctor removed his mask' is not ambiguous between a coreferential reading and a disjoint reading. Instead, the coreferential reading requires a non-gendered pronoun, and the gendered, possessive pronouns are anti-reflexive. We present a multilingual, multi-task challenge dataset, which spans four languages and four NLP tasks and focuses only on this phenomenon. We find evidence for gender bias across all task-language combinations and correlate model bias with national labor market statistics.Comment: To appear in EMNLP 202

Ethos and Pragmatics

Ethos, the speaker’s image in speech is one of the three means of persuasion e stablished by Aristotle’s Rhetoric and is often studied in a loose way. Many scholars develop lists of self-images (ethos of a leader, modesty ethos, etc.), but few explain how one arrives at these types of ethos. This is precisely what the inferential approach described here intends to do. Considering, like many discourse analysts, that ethos is consubstantial with speech, this paper provides an overview of various types and subtypes of ethos and highlights how these can be inferred from the discourse. Mainly, we would like to point out that what the speaker says about him or herself is only a part of what has been called “said ethos”: inferential processes triggered by what the speaker says about collectivities, opponents, or the audience also help construct an ethos. This tool will be applied to analyze a corpus of Donald Trump’s tweets of 6 January 2021, the day of the assault on the Capitol. As the notion of inference is essential in creating ethos, the paper pleads for the integration of the study of this rhetorical notion in the field of pragmatics