Vagueness and referential ambiguity in a large-scale annotated corpus

In this paper, we argue that difficulties in the definition of coreference itself contribute to lower inter-annotator agreement in certain cases. Data from a large referentially annotated corpus serves to corroborate this point, using a quantitative investigation to assess which effects or problems are likely to be the most prominent. Several examples where such problems occur are discussed in more detail, and we then propose a generalisation of Poesio, Reyle and Stevenson’s Justified Sloppiness Hypothesis to provide a unified model for these cases of disagreement and argue that a deeper understanding of the phenomena involved allows to tackle problematic cases in a more principled fashion than would be possible using only pre-theoretic intuitions

A database of semantic clusters of verb usages

We are presenting VPS-30-En, a small lexical resource that contains the following 30 English verbs: access, ally, arrive, breathe, claim, cool, crush, cry, deny, enlarge, enlist, forge, furnish, hail, halt, part, plough, plug, pour, say, smash, smell, steer, submit, swell, tell, throw, trouble, wake and yield. We have created and have been using VPS-30-En to explore the interannotator agreement potential of the Corpus Pattern Analysis. VPS-30-En is a small snapshot of the Pattern Dictionary of English Verbs (Hanks and Pustejovsky, 2005), which we revised (both the entries and the annotated concordances) and enhanced with additional annotations. It is freely available at http://ufal.mff.cuni.cz/spr. In this paper, we compare the annotation scheme of VPS-30-En with the original PDEV. We also describe the adjustments we have made and their motivation, as well as the most pervasive causes of interannotator disagreements

Communicating with Culture: How Humans and Machines Detect Narrative Elements

To understand how people communicate, we must understand how they leverage shared stories and all the knowledge, information, and associations contained within those stories. I examine three classes of narrative elements that convey a wealth of cultural knowledge: Propp\u27s morphology, motifs, and discourse structure. Propp\u27s morphology communicates how roles and actions drive a narrative forward; motifs fill those roles and actions with specific, remarkable events; discourse groups these into a coherent structure to convey a point. My thesis has three aims: first, to demonstrate that people can reliably detect and identify all three of these narrative elements; second, to develop automatic detectors for discourse and motifs; third, to demonstrate the deep relation between these narrative elements and other theories of narrative structure and knowledge representation that I refer to as the \textit{continuum of communication}. The first step of my work answers two key questions about Propp\u27s morphology by demonstrating the reliability of annotators applying Propp\u27s scheme across a variety of experiments, in a double-blind annotation study. Additionally, I demonstrate a shortcoming in Propp\u27s scheme, demonstrating areas in which there are elements present in the folktales he analyzed that are not part of his morphology. The second step of my work, showing that people familiar with motifs can reliably detect when they are being used to share information and associations, approaches this problem by performing a large-scale annotation study of 21,000 examples into four categories performed by three pairs of annotators over a period of 11 weeks. I show that, in a double-blind annotation study, people familiar with the motifs had a moderate to high degree of agreement, demonstrating the reliability of humans at this task. The third step demonstrates the reliability of applying a theory of news discourse structure to news articles via a double-blind annotation study and, using the results of this annotation, demonstrate a preliminary detector of the news discourse function of paragraphs in news articles. The fourth step of my work, detecting motific usage automatically, consists of a large-scale pipeline that achieves moderate performance. This pipeline is the first work towards automatically detecting motific usage of motifs and beats out simple baselines while comparing favorably too and generalizing better than a simple neural network baseline system. Additionally, the pipeline uses explainable features that can be used in future work to further develop our understanding of how humans automatically detect motifs. Finally, I describe an exploration of the broader scope of narrative elements that communicate information between individuals who share a cultural or sub-cultural background. This work is based off of a small-scale, in-lab annotation of posts from the “incel” subculture, a niche internet community with extremist elements and, at times, disturbing content. This small annotation has revealed a complex landscape encompassing fourteen categories, more than three times the number of elements as the large-scale annotation, many of which resemble the moving parts of other theories on narrative structure and cognition, including Vladimir Propp\u27s morphology of folktales and Silvan Tomkins\u27 script theory. I describe these relations and provide a rough continuum of the landscape of narrative communication

For Women, Life, Freedom: A Participatory AI-Based Social Web Analysis of a Watershed Moment in Iran's Gender Struggles

In this paper, we present a computational analysis of the Persian language Twitter discourse with the aim to estimate the shift in stance toward gender equality following the death of Mahsa Amini in police custody. We present an ensemble active learning pipeline to train a stance classifier. Our novelty lies in the involvement of Iranian women in an active role as annotators in building this AI system. Our annotators not only provide labels, but they also suggest valuable keywords for more meaningful corpus creation as well as provide short example documents for a guided sampling step. Our analyses indicate that Mahsa Amini's death triggered polarized Persian language discourse where both fractions of negative and positive tweets toward gender equality increased. The increase in positive tweets was slightly greater than the increase in negative tweets. We also observe that with respect to account creation time, between the state-aligned Twitter accounts and pro-protest Twitter accounts, pro-protest accounts are more similar to baseline Persian Twitter activity.Comment: Accepted at IJCAI 2023 (AI for good track

Doctor of Philosophy

dissertationManual annotation of clinical texts is often used as a method of generating reference standards that provide data for training and evaluation of Natural Language Processing (NLP) systems. Manually annotating clinical texts is time consuming, expensive, and requires considerable cognitive effort on the part of human reviewers. Furthermore, reference standards must be generated in ways that produce consistent and reliable data but must also be valid in order to adequately evaluate the performance of those systems. The amount of labeled data necessary varies depending on the level of analysis, the complexity of the clinical use case, and the methods that will be used to develop automated machine systems for information extraction and classification. Evaluating methods that potentially reduce cost, manual human workload, introduce task efficiencies, and reduce the amount of labeled data necessary to train NLP tools for specific clinical use cases are active areas of research inquiry in the clinical NLP domain. This dissertation integrates a mixed methods approach using methodologies from cognitive science and artificial intelligence with manual annotation of clinical texts. Aim 1 of this dissertation identifies factors that affect manual annotation of clinical texts. These factors are further explored by evaluating approaches that may introduce efficiencies into manual review tasks applied to two different NLP development areas - semantic annotation of clinical concepts and identification of information representing Protected Health Information (PHI) as defined by HIPAA. Both experiments integrate iv different priming mechanisms using noninteractive and machine-assisted methods. The main hypothesis for this research is that integrating pre-annotation or other machineassisted methods within manual annotation workflows will improve efficiency of manual annotation tasks without diminishing the quality of generated reference standards

Grounding event references in news

Events are frequently discussed in natural language, and their accurate identification is central to language understanding. Yet they are diverse and complex in ontology and reference; computational processing hence proves challenging. News provides a shared basis for communication by reporting events. We perform several studies into news event reference. One annotation study characterises each news report in terms of its update and topic events, but finds that topic is better consider through explicit references to background events. In this context, we propose the event linking task which—analogous to named entity linking or disambiguation—models the grounding of references to notable events. It defines the disambiguation of an event reference as a link to the archival article that first reports it. When two references are linked to the same article, they need not be references to the same event. Event linking hopes to provide an intuitive approximation to coreference, erring on the side of over-generation in contrast with the literature. The task is also distinguished in considering event references from multiple perspectives over time. We diagnostically evaluate the task by first linking references to past, newsworthy events in news and opinion pieces to an archive of the Sydney Morning Herald. The intensive annotation results in only a small corpus of 229 distinct links. However, we observe that a number of hyperlinks targeting online news correspond to event links. We thus acquire two large corpora of hyperlinks at very low cost. From these we learn weights for temporal and term overlap features in a retrieval system. These noisy data lead to significant performance gains over a bag-of-words baseline. While our initial system can accurately predict many event links, most will require deep linguistic processing for their disambiguation

Factual or Believable? Negotiating the Boundaries of Confirmation Bias in Online News Stories

We examine the fake news phenomenon from a fresh perspective. Instead of assessing the factuality of news claims, our work explores the impact of these claims on reader beliefs. With the 2017 Alabama senate race as the empirical context, we examine how readers on both sides of the political spectrum evaluate online news stories considering their preconceived beliefs and values. Our analysis builds on concepts from argument and social representations theories to explore the role of argumentation in this process. We focus on detecting arguments in reader comments to depict challenges involved in reader consideration of newsworthy events and news stories. A key finding of the paper is that readers from both sides of the political spectrum appear to engage in similar strategies to confirm or negotiate acceptance or rejection of claims. The paper contributes to theory by depicting social representation as a process that mediates conflict in belief structures. We conclude by speculating about possibilities for future work, such as designing behavioral and technological interventions that can supplement fact-checking. An important goal here is to improve how we, in the presence of our biases, collectively consume online news stories and engage in the discourse that surrounds them