Gesture in Automatic Discourse Processing

Computers cannot fully understand spoken language without access to the wide range of modalities that accompany speech. This thesis addresses the particularly expressive modality of hand gesture, and focuses on building structured statistical models at the intersection of speech, vision, and meaning.My approach is distinguished in two key respects. First, gestural patterns are leveraged to discover parallel structures in the meaning of the associated speech. This differs from prior work that attempted to interpret individual gestures directly, an approach that was prone to a lack of generality across speakers. Second, I present novel, structured statistical models for multimodal language processing, which enable learning about gesture in its linguistic context, rather than in the abstract.These ideas find successful application in a variety of language processing tasks: resolving ambiguous noun phrases, segmenting speech into topics, and producing keyframe summaries of spoken language. In all three cases, the addition of gestural features -- extracted automatically from video -- yields significantly improved performance over a state-of-the-art text-only alternative. This marks the first demonstration that hand gesture improves automatic discourse processing

Resolving pronominal anaphora using commonsense knowledge

Coreference resolution is the task of resolving all expressions in a text that refer to the same entity. Such expressions are often used in writing and speech as shortcuts to avoid repetition. The most frequent form of coreference is the anaphor. To resolve anaphora not only grammatical and syntactical strategies are required, but also semantic approaches should be taken into consideration. This dissertation presents a framework for automatically resolving pronominal anaphora by integrating recent findings from the field of linguistics with new semantic features. Commonsense knowledge is the routine knowledge people have of the everyday world. Because such knowledge is widely used it is frequently omitted from social communications such as texts. It is understandable that without this knowledge computers will have difficulty making sense of textual information. In this dissertation a new set of computational and linguistic features are used in a supervised learning approach to resolve the pronominal anaphora in document. Commonsense knowledge sources such as ConceptNet and WordNet are used and similarity measures are extracted to uncover the elaborative information embedded in the words that can help in the process of anaphora resolution. The anaphoric system is tested on 350 Wall Street Journal articles from the BBN corpus. When compared with other systems available such as BART (Versley et al. 2008) and Charniak and Elsner 2009, our system performed better and also resolved a much wider range of anaphora. We were able to achieve a 92% F-measure on the BBN corpus and an average of 85% F-measure when tested on other genres of documents such as children stories and short stories selected from the web

A review of temporal aspects of hand gesture analysis applied to discourse analysis and natural conversation

Lately, there has been a\ud n increasing\ud interest in hand gesture analysis systems. Recent works have employed\ud pat\ud tern recognition techniques and have focused on the development of systems with more natural user\ud interfaces. These systems may use gestures to control interfaces or recognize sign language gestures\ud , which\ud can provide systems with multimodal interaction; o\ud r consist in multimodal tools to help psycholinguists to\ud understand new aspects of discourse analysis and to automate laborious tasks.\ud Gestures are characterized\ud by several aspects, mainly by movements\ud and sequence of postures\ud . Since data referring to move\ud ments\ud or\ud sequences\ud carry temporal information\ud , t\ud his paper presents a\ud literature\ud review\ud about\ud temporal aspects of\ud hand gesture analysis, focusing on applications related to natural conversation and psycholinguistic\ud analysis, using Systematic Literature Revi\ud ew methodology. In our results, we organized works according to\ud type of analysis, methods, highlighting the use of Machine Learning techniques, and applications.FAPESP 2011/04608-