Capturing Ambiguity in Crowdsourcing Frame Disambiguation

FrameNet is a computational linguistics resource composed of semantic frames, high-level concepts that represent the meanings of words. In this paper, we present an approach to gather frame disambiguation annotations in sentences using a crowdsourcing approach with multiple workers per sentence to capture inter-annotator disagreement. We perform an experiment over a set of 433 sentences annotated with frames from the FrameNet corpus, and show that the aggregated crowd annotations achieve an F1 score greater than 0.67 as compared to expert linguists. We highlight cases where the crowd annotation was correct even though the expert is in disagreement, arguing for the need to have multiple annotators per sentence. Most importantly, we examine cases in which crowd workers could not agree, and demonstrate that these cases exhibit ambiguity, either in the sentence, frame, or the task itself, and argue that collapsing such cases to a single, discrete truth value (i.e. correct or incorrect) is inappropriate, creating arbitrary targets for machine learning.Comment: in publication at the sixth AAAI Conference on Human Computation and Crowdsourcing (HCOMP) 201

Incomplete Innovation and the Premature Disruption of Legal Services

Article published in the Michigan State Law Review

Frame Shifts und Frame-Vergleichbarkeit bei Englisch-Deutscher Übersetzung am Beispiel einer Volltextannotation mit FrameNet

In dieser Masterarbeit wird eine selbst erstelle Volltextannotation einer deutschen Übersetzung aus dem Englischen untersucht und die beiden Textversionen und ihre Annotation miteinander verglichen. Im Mittelpunkt stehen die Fragen, inwieweit sich die in FrameNet für das Englische formulierten Frames für die Volltextannotation eines deutschen Textes nutzen lassen und welche Abweichungen zwischen englischem Original und deutscher Übersetzung auf Ebene der Frames und ihrer Lexikalisierung bestehen. Zu Beginn der Arbeit wird ein Überblick über den relevanten theoretischen Hintergrund und Forschungsstand der Frame-Semantik einschließlich ihrer Anwendung im FrameNet Project, in der Annotation und in der sprachübergreifenden Forschung gegeben. Im dritten Kapitel wird die Untersuchung einschließlich des verwendeten Materials und der Methoden vorgestellt. Darauf aufbauend folgt die Präsentation der Ergebnisse, die in Kapitel fünf mit Bezug auf die Forschungsfragen ausgewertet werden. Den Abschluss bilden Schlussfolgerungen zur sprachübergreifenden Nutzung von FrameNet und zur Volltextannotation sowie ein Ausblick auf weitere Forschungsfelder.:1 Einleitung 2 Hintergrund und Forschungsstand 2.1 Ansätze der kognitiven Linguistik 2.2 Frame-Semantik 2.3 FrameNet Project 2.4 Framesemantische Annotation 2.5 Sprachübergreifende Annotation und Frame-Semantik 3 Vorstellung der Untersuchung 3.1 Textmaterial 3.2 Methodisches Vorgehen 3.2.1 Volltextannotation 3.2.2 Vergleich der annotierten Textversionen 4 Ergebnisse und Diskussion 4.1 Volltextannotation der deutschen Übersetzung 4.1.1 Übertragbarkeit von FrameNet-Frames auf den deutschen Text 4.2 Vergleich der annotierten Textversionen 4.2.1 Frame Shifts 4.3 Evaluation der Methoden 5 Fazit und Ausblick 5.1 Volltextannotation 5.2 Sprachübergreifende Nutzung von FrameNet 5.3 Verbindung von Frame-Semantik und Konstruktionsgrammatik 5.4 Abschließendes Fazit Literaturverzeichni

Human-AI Interaction in the Presence of Ambiguity: From Deliberation-based Labeling to Ambiguity-aware AI

Ambiguity, the quality of being open to more than one interpretation, permeates our lives. It comes in different forms including linguistic and visual ambiguity, arises for various reasons and gives rise to disagreements among human observers that can be hard or impossible to resolve. As artificial intelligence (AI) is increasingly infused into complex domains of human decision making it is crucial that the underlying AI mechanisms also support a notion of ambiguity. Yet, existing AI approaches typically assume that there is a single correct answer for any given input, lacking mechanisms to incorporate diverse human perspectives in various parts of the AI pipeline, including data labeling, model development and user interface design. This dissertation aims to shed light on the question of how humans and AI can be effective partners in the presence of ambiguous problems. To address this question, we begin by studying group deliberation as a tool to detect and analyze ambiguous cases in data labeling. We present three case studies that investigate group deliberation in the context of different labeling tasks, data modalities and types of human labeling expertise. First, we present CrowdDeliberation, an online platform for synchronous group deliberation in novice crowd work, and show how worker deliberation affects resolvability and accuracy in text classification tasks of varying subjectivity. We then translate our findings to the expert domain of medical image classification to demonstrate how imposing additional structure on deliberation arguments can improve the efficiency of the deliberation process without compromising its reliability. Finally, we present CrowdEEG, an online platform for collaborative annotation and deliberation of medical time series data, implementing an asynchronous and highly structured deliberation process. Our findings from an observational study with 36 sleep health professionals help explain how disagreements arise and when they can be resolved through group deliberation. Beyond investigating group deliberation within data labeling, we also demonstrate how the resulting deliberation data can be used to support both human and artificial intelligence. To this end, we first present results from a controlled experiment with ten medical generalists, suggesting that reading deliberation data from medical specialists significantly improves generalists' comprehension and diagnostic accuracy on difficult patient cases. Second, we leverage deliberation data to simulate and investigate AI assistants that not only highlight ambiguous cases, but also explain the underlying sources of ambiguity to end users in human-interpretable terms. We provide evidence suggesting that this form of ambiguity-aware AI can help end users to triage and trust AI-provided data classifications. We conclude by outlining the main contributions of this dissertation and directions for future research