Automatic Extraction of Narrative Structure from Long Form Text

Automatic understanding of stories is a long-time goal of artificial intelligence and natural language processing research communities. Stories literally explain the human experience. Understanding our stories promotes the understanding of both individuals and groups of people; various cultures, societies, families, organizations, governments, and corporations, to name a few. People use stories to share information. Stories are told –by narrators– in linguistic bundles of words called narratives. My work has given computers awareness of narrative structure. Specifically, where are the boundaries of a narrative in a text. This is the task of determining where a narrative begins and ends, a non-trivial task, because people rarely tell one story at a time. People don’t specifically announce when we are starting or stopping our stories: We interrupt each other. We tell stories within stories. Before my work, computers had no awareness of narrative boundaries, essentially where stories begin and end. My programs can extract narrative boundaries from novels and short stories with an F1 of 0.65. Before this I worked on teaching computers to identify which paragraphs of text have story content, with an F1 of 0.75 (which is state of the art). Additionally, I have taught computers to identify the narrative point of view (POV; how the narrator identifies themselves) and diegesis (how involved in the story’s action is the narrator) with F1 of over 0.90 for both narrative characteristics. For the narrative POV, diegesis, and narrative level extractors I ran annotation studies, with high agreement, that allowed me to teach computational models to identify structural elements of narrative through supervised machine learning. My work has given computers the ability to find where stories begin and end in raw text. This allows for further, automatic analysis, like extraction of plot, intent, event causality, and event coreference. These tasks are impossible when the computer can’t distinguish between which stories are told in what spans of text. There are two key contributions in my work: 1) my identification of features that accurately extract elements of narrative structure and 2) the gold-standard data and reports generated from running annotation studies on identifying narrative structure

Inducing Stereotypical Character Roles from Plot Structure

If we are to understand stories, we must understand characters: characters are central to every narrative and drive the action forward. Critically, many stories (especially cultural ones) employ stereotypical character roles in their stories for different purposes, including efficient communication among bundles of default characteristics and associations, ease understanding of those characters\u27 role in the overall narrative, and many more. These roles include ideas such as hero, villain, or victim, as well as culturally-specific roles such as, for example, the donor (in Russian tales) or the trickster (in Native American tales). My thesis aims to learn these roles automatically, inducing them from data using a clustering technique. The first step of learning character roles, however, is to identify which coreference chains correspond to characters, which are defined by narratologists as animate entities that drive the plot forward. The first part of my work has focused on this character identification problem, specifically focusing on the problem of animacy detection. Prior work treated animacy as a word-level property, and researchers developed statistical models to classify words as either animate or inanimate. I claimed this approach to the problem is ill-posed and presented a new hybrid approach for classifying the animacy of coreference chains that achieved state-of-the-art performance. The next step of my work is to develop approaches first to identify the characters and then a new unsupervised clustering approach to learn stereotypical roles. My character identification system consists of two stages: first, I detect animate chains from the coreference chains using my existing animacy detector; second, I apply a supervised machine learning model that identifies which of those chains qualify as characters. I proposed a narratologically grounded definition of character and built a supervised machine learning model with a small set of features that achieved state-of-the-art performance. In the last step, I successfully implemented a clustering approach with plot and thematic information to cluster the archetypes. This work resulted in a completely new approach to understanding the structure of stories, greatly advancing the state-of-the-art of story understanding

Weakly-supervised Learning Approaches for Event Knowledge Acquisition and Event Detection

Capabilities of detecting events and recognizing temporal, subevent, or causality relations among events can facilitate many applications in natural language understanding. However, supervised learning approaches that previous research mainly uses have two problems. First, due to the limited size of annotated data, supervised systems cannot sufficiently capture diverse contexts to distill universal event knowledge. Second, under certain application circumstances such as event recognition during emergent natural disasters, it is infeasible to spend days or weeks to annotate enough data to train a system. My research aims to use weakly-supervised learning to address these problems and to achieve automatic event knowledge acquisition and event recognition. In this dissertation, I first introduce three weakly-supervised learning approaches that have been shown effective in acquiring event relational knowledge. Firstly, I explore the observation that regular event pairs show a consistent temporal relation despite of their various contexts, and these rich contexts can be used to train a contextual temporal relation classifier to further recognize new temporal relation knowledge. Secondly, inspired by the double temporality characteristic of narrative texts, I propose a weakly supervised approach that identifies 287k narrative paragraphs using narratology principles and then extract rich temporal event knowledge from identified narratives. Lastly, I develop a subevent knowledge acquisition approach by exploiting two observations that 1) subevents are temporally contained by the parent event and 2) the definitions of the parent event can be used to guide the identification of subevents. I collect rich weak supervision to train a contextual BERT classifier and apply the classifier to identify new subevent knowledge. Recognizing texts that describe specific categories of events is also challenging due to language ambiguity and diverse descriptions of events. So I also propose a novel method to rapidly build a fine-grained event recognition system on social media texts for disaster management. My method creates high-quality weak supervision based on clustering-assisted word sense disambiguation and enriches tweet message representations using preceding context tweets and reply tweets in building event recognition classifiers

Multimodal knowledge integration for object detection and visual reasoning

We humans still perceive and reason in a different way than artificial intelligence models. We witness, we listen, we touch, we understand the world via multi-modal sensing, while machine models rely only on a single or a few modalities and ignore abundant information. In this thesis, we explore techniques for reducing the perception gap between machines and humans and focus on two families of tasks, reasoning and detection. First, we incorporate information from text, audio, motion, external knowledge bases, for training computer vision models. We find that data inputs from more extensive channels provide complementary information to improve models. Second, we study how multimodal inputs can be fully utilized. We argue that most existing deep learning methods are prone to pay too large attention to shallow patterns in the input features, which causes the resulting models to be biased. We propose robust training to overcome the issue. Third, we extend the benefits of multi-modal information to the supervision signals instead of the inputs, by learning a weakly supervised detection model from the natural supervision of textual captions or audio narrations. With the help of NLP constituency parsing, it is possible to extract structural knowledges from the captions and narrations, hence determines the entities and relations of visual objects

An Actor-Centric Approach to Facial Animation Control by Neural Networks For Non-Player Characters in Video Games

Game developers increasingly consider the degree to which character animation emulates facial expressions found in cinema. Employing animators and actors to produce cinematic facial animation by mixing motion capture and hand-crafted animation is labor intensive and therefore expensive. Emotion corpora and neural network controllers have shown promise toward developing autonomous animation that does not rely on motion capture. Previous research and practice in disciplines of Computer Science, Psychology and the Performing Arts have provided frameworks on which to build a workflow toward creating an emotion AI system that can animate the facial mesh of a 3d non-player character deploying a combination of related theories and methods. However, past investigations and their resulting production methods largely ignore the emotion generation systems that have evolved in the performing arts for more than a century. We find very little research that embraces the intellectual process of trained actors as complex collaborators from which to understand and model the training of a neural network for character animation. This investigation demonstrates a workflow design that integrates knowledge from the performing arts and the affective branches of the social and biological sciences. Our workflow begins at the stage of developing and annotating a fictional scenario with actors, to producing a video emotion corpus, to designing training and validating a neural network, to analyzing the emotion data annotation of the corpus and neural network, and finally to determining resemblant behavior of its autonomous animation control of a 3d character facial mesh. The resulting workflow includes a method for the development of a neural network architecture whose initial efficacy as a facial emotion expression simulator has been tested and validated as substantially resemblant to the character behavior developed by a human actor

Electrophysiological Correlates of Natural Language Processing in Children and Adults

To understand the causes of differences in language ability we must measure the specific and separable processes that contribute to natural language comprehension. Specifically, we need measures of the three language subsystems – semantics, syntax, and phonology – as they are used during the comprehension of real speech. Event-Related Potentials (ERPs) are a promising approach to reaching this level of specificity. Previous research has identified distinct ERP effects for each of the subsystems – the N400 to semantic anomalies, the Anterior Negativity and P600 to syntactic anomalies, and the Phonological Mapping Negativity to unexpected speech sounds. However, these studies typically use stimuli and tasks that encourage processing that differs from real-world language comprehension. Further, previous ERP studies indexing language processing in young children not only use unfamiliar tasks, but also typically exclude data from the large proportion of children. We need to measure language-related ERPs in a context as close as possible to real-world processing, and in a manner that includes data from representative rather than highly-selected samples of children. The experiments described in this dissertation achieve that goal. Adults and five-year-old children listened to a child-directed story while answering comprehension questions. Infrequent violations were included to independently probe the three language subsystems. In children and adults, the canonical N400 response was evident in response to semantic violations. Morphosyntactic violations elicited a long-duration Anterior Negativity without a later P600. Phonological violations on suffixes elicited a Phonological Mapping Negativity in adults. This is the first report of this phonological effect outside of highly-predictable lexical contexts. Popular normed behavioral assessments were also administered to the children who participated in this study. Results from these assessments confirmed that performance on tasks claiming to measure categorically different abilities are correlated with one another, and that language measures correlate with so-called nonverbal measures. ERPs indexing different language subsystem did not correlate with each other or with measures of nonverbal cognitive ability. Using multiple ERP measures during natural language comprehension, we are able to isolate specific aspects of language processing, increasing the possibility of making meaningful connections between biology, experience, and resulting language ability

Uncovering the Hidden Cognitive Processes and Underlying Dynamics of Deception

This dissertation examines the processing demands associated with motor responding and verbal statements during deceptive (or deceptive-like) behavior. In the first set of studies presented in Chapter 2, participants motor movements in a false response paradigm revealed signatures of competition with the truth. In a second set of studies presented in Chapter 3, deceptive participants used language that reflected cognitive and social demands inherent to various types of deception. In evaluating both motor and verbal cues, this dissertation provides a comprehensive, multi-modal approach to better understanding the cognitive processes underlying deception. in conducting the motor responding studies, participants\u27 arm movements were analyzed as they navigated a motor tracking device (computer-mouse, Nintendo Wiimote). To visually co-present response options, where the true option acts as a competitor to a false target. In an initial study, competition during deceptive responding was shown to be much greater than during truthful responding. In two follow-up studies, the introduction of various task-based cognitive demands was shown to systematically modulate response performance. Specifically, these studies suggest that an intention to false respond early in question presentation will amplify competition effects, and that false responding to information in autobiographical memory is much more difficult than responding to information in general semantic memory. In the studies analyzing verbal statements, the focus is turned to large-scale linguistic analyses using automated natural language processing tools. In the first study, changes in language use were identifed between deceptive and truthful narratives using six psychologically relevant categories. A major finding was that the language of deception is adapted to faciliate ease of cognitive processing. In a second study, the indicative phrasing and semantic content of deceptive texts was extracted using a contrastive corpus analysis, whereby indicative features are defined by frequent use in one corpus while being infrequent in a comparative corpus. Two contexts of deception were evaluated. In the first context of computer-mediated conversations, decievers used a range of unique thematic elements, as in avoiding personal involvement in their narrative accounts. In the second context of attitudes towards abortion, unique thematic elements once again emerged; for example, participants tended to position their arguments in terms of formal law

Sentiment Analysis for Fake News Detection

[Abstract] In recent years, we have witnessed a rise in fake news, i.e., provably false pieces of information created with the intention of deception. The dissemination of this type of news poses a serious threat to cohesion and social well-being, since it fosters political polarization and the distrust of people with respect to their leaders. The huge amount of news that is disseminated through social media makes manual verification unfeasible, which has promoted the design and implementation of automatic systems for fake news detection. The creators of fake news use various stylistic tricks to promote the success of their creations, with one of them being to excite the sentiments of the recipients. This has led to sentiment analysis, the part of text analytics in charge of determining the polarity and strength of sentiments expressed in a text, to be used in fake news detection approaches, either as a basis of the system or as a complementary element. In this article, we study the different uses of sentiment analysis in the detection of fake news, with a discussion of the most relevant elements and shortcomings, and the requirements that should be met in the near future, such as multilingualism, explainability, mitigation of biases, or treatment of multimedia elements.Xunta de Galicia; ED431G 2019/01Xunta de Galicia; ED431C 2020/11This work has been funded by FEDER/Ministerio de Ciencia, Innovación y Universidades — Agencia Estatal de Investigación through the ANSWERASAP project (TIN2017-85160-C2-1-R); and by Xunta de Galicia through a Competitive Reference Group grant (ED431C 2020/11). CITIC, as Research Center of the Galician University System, is funded by the Consellería de Educación, Universidade e Formación Profesional of the Xunta de Galicia through the European Regional Development Fund (ERDF/FEDER) with 80%, the Galicia ERDF 2014-20 Operational Programme, and the remaining 20% from the Secretaría Xeral de Universidades (ref. ED431G 2019/01). David Vilares is also supported by a 2020 Leonardo Grant for Researchers and Cultural Creators from the BBVA Foundation. Carlos Gómez-Rodríguez has also received funding from the European Research Council (ERC), under the European Union’s Horizon 2020 research and innovation programme (FASTPARSE, grant No. 714150