Text–to–Video: Image Semantics and NLP

When aiming at automatically translating an arbitrary text into a visual story, the main challenge consists in finding a semantically close visual representation whereby the displayed meaning should remain the same as in the given text. Besides, the appearance of an image itself largely influences how its meaningful information is transported towards an observer. This thesis now demonstrates that investigating in both, image semantics as well as the semantic relatedness between visual and textual sources enables us to tackle the challenging semantic gap and to find a semantically close translation from natural language to a corresponding visual representation. Within the last years, social networking became of high interest leading to an enormous and still increasing amount of online available data. Photo sharing sites like Flickr allow users to associate textual information with their uploaded imagery. Thus, this thesis exploits this huge knowledge source of user generated data providing initial links between images and words, and other meaningful data. In order to approach visual semantics, this work presents various methods to analyze the visual structure as well as the appearance of images in terms of meaningful similarities, aesthetic appeal, and emotional effect towards an observer. In detail, our GPU-based approach efficiently finds visual similarities between images in large datasets across visual domains and identifies various meanings for ambiguous words exploring similarity in online search results. Further, we investigate in the highly subjective aesthetic appeal of images and make use of deep learning to directly learn aesthetic rankings from a broad diversity of user reactions in social online behavior. To gain even deeper insights into the influence of visual appearance towards an observer, we explore how simple image processing is capable of actually changing the emotional perception and derive a simple but effective image filter. To identify meaningful connections between written text and visual representations, we employ methods from Natural Language Processing (NLP). Extensive textual processing allows us to create semantically relevant illustrations for simple text elements as well as complete storylines. More precisely, we present an approach that resolves dependencies in textual descriptions to arrange 3D models correctly. Further, we develop a method that finds semantically relevant illustrations to texts of different types based on a novel hierarchical querying algorithm. Finally, we present an optimization based framework that is capable of not only generating semantically relevant but also visually coherent picture stories in different styles.Bei der automatischen Umwandlung eines beliebigen Textes in eine visuelle Geschichte, besteht die größte Herausforderung darin eine semantisch passende visuelle Darstellung zu finden. Dabei sollte die Bedeutung der Darstellung dem vorgegebenen Text entsprechen. Darüber hinaus hat die Erscheinung eines Bildes einen großen Einfluß darauf, wie seine bedeutungsvollen Inhalte auf einen Betrachter übertragen werden. Diese Dissertation zeigt, dass die Erforschung sowohl der Bildsemantik als auch der semantischen Verbindung zwischen visuellen und textuellen Quellen es ermöglicht, die anspruchsvolle semantische Lücke zu schließen und eine semantisch nahe Übersetzung von natürlicher Sprache in eine entsprechend sinngemäße visuelle Darstellung zu finden. Des Weiteren gewann die soziale Vernetzung in den letzten Jahren zunehmend an Bedeutung, was zu einer enormen und immer noch wachsenden Menge an online verfügbaren Daten geführt hat. Foto-Sharing-Websites wie Flickr ermöglichen es Benutzern, Textinformationen mit ihren hochgeladenen Bildern zu verknüpfen. Die vorliegende Arbeit nutzt die enorme Wissensquelle von benutzergenerierten Daten welche erste Verbindungen zwischen Bildern und Wörtern sowie anderen aussagekräftigen Daten zur Verfügung stellt. Zur Erforschung der visuellen Semantik stellt diese Arbeit unterschiedliche Methoden vor, um die visuelle Struktur sowie die Wirkung von Bildern in Bezug auf bedeutungsvolle Ähnlichkeiten, ästhetische Erscheinung und emotionalem Einfluss auf einen Beobachter zu analysieren. Genauer gesagt, findet unser GPU-basierter Ansatz effizient visuelle Ähnlichkeiten zwischen Bildern in großen Datenmengen quer über visuelle Domänen hinweg und identifiziert verschiedene Bedeutungen für mehrdeutige Wörter durch die Erforschung von Ähnlichkeiten in Online-Suchergebnissen. Des Weiteren wird die höchst subjektive ästhetische Anziehungskraft von Bildern untersucht und "deep learning" genutzt, um direkt ästhetische Einordnungen aus einer breiten Vielfalt von Benutzerreaktionen im sozialen Online-Verhalten zu lernen. Um noch tiefere Erkenntnisse über den Einfluss des visuellen Erscheinungsbildes auf einen Betrachter zu gewinnen, wird erforscht, wie alleinig einfache Bildverarbeitung in der Lage ist, tatsächlich die emotionale Wahrnehmung zu verändern und ein einfacher aber wirkungsvoller Bildfilter davon abgeleitet werden kann. Um bedeutungserhaltende Verbindungen zwischen geschriebenem Text und visueller Darstellung zu ermitteln, werden Methoden des "Natural Language Processing (NLP)" verwendet, die der Verarbeitung natürlicher Sprache dienen. Der Einsatz umfangreicher Textverarbeitung ermöglicht es, semantisch relevante Illustrationen für einfache Textteile sowie für komplette Handlungsstränge zu erzeugen. Im Detail wird ein Ansatz vorgestellt, der Abhängigkeiten in Textbeschreibungen auflöst, um 3D-Modelle korrekt anzuordnen. Des Weiteren wird eine Methode entwickelt die, basierend auf einem neuen hierarchischen Such-Anfrage Algorithmus, semantisch relevante Illustrationen zu Texten verschiedener Art findet. Schließlich wird ein optimierungsbasiertes Framework vorgestellt, das nicht nur semantisch relevante, sondern auch visuell kohärente Bildgeschichten in verschiedenen Bildstilen erzeugen kann

The effectiveness of written corrective feedback on French as a second language accuracy

Cette étude quasi-expérimentale a pour objectifs de (1) comparer les effets de trois techniques de rétroaction corrective à l’écrit (RCÉ), à savoir la RCÉ directe (reformulation) et deux RCÉs indirectes (soulignement et soulignement plus indices métalinguistiques) sur la précision langagière à l’écrit, (2) d’examiner les effets modérateurs de la catégorie d’erreur et (3) d’examiner les effets modérateurs du niveau de compétence langagière des apprenants ainsi que leur capacité d’analyse verbale. Quatre groupes de français langue seconde (n = 65) inscrits dans des programmes de français enrichi dans des écoles secondaires anglophones à Montréal ont participé à cette étude. L’intervention expérimentale a comporté trois cycles d’écriture, de rétroaction et de révision qui ont été complétés sur des intervalles de deux semaines, et a ciblé l’accord dans le groupe nominal (GN), l’accord dans le prédicat, la structure du GN, la structure du groupe verbal (GV) ainsi que les homophones grammaticaux. L’intervention a été réalisée à travers trois conditions expérimentales (RCÉ directe, RCÉ indirecte sous forme de soulignement, RCÉ indirecte sous forme de soulignement plus indices métalinguistiques) et une condition de comparaison (pas de RCÉ). Chaque groupe expérimental a complété six tâches d’écriture (rappel de texte) : une tâche durant le pré-test et le post-test immédiat, une autre tâche pour le post-test différé et trois tâches différentes durant l’intervention expérimentale. Les quatre groupes ont révisé les textes complétés durant l’intervention expérimentale. Tous les participants ont aussi complété un test d’analyse verbale. Les résultats indiquent que (1) la provision de la RCÉ est plus bénéfique que son absence, (2) la RCÉ indirecte est meilleure que la RCÉ directe, (3) la RCÉ combinée avec des indices métalinguistiques (i.e., RCÉ indirecte métalinguistique) est plus efficace que le soulignement sans indices métalinguistiques. Concernant la deuxième question, les résultats suggèrent que certaines catégories d'erreurs étaient plus sensibles à la RCÉ que d'autres. Bien qu'aucune amélioration n'ait été constatée pour l'accord dans le GN ou pour l'accord dans le prédicat, l'utilisation précise des homophones grammaticaux s’est améliorée pour tous les groupes. Une telle amélioration était plus durable pour le groupe ayant reçu la RCÉ indirecte métalinguistique. Quant à la troisième question, les résultats indiquent que le niveau de compétence langagière n'a joué un rôle modérateur que pour le groupe recevant la RCÉ indirecte métalinguistique, mais aucun effet de ce type n'a été observé pour la capacité d'analyse verbale, et ce, quelle que soit la condition expérimentale.This quasi-experimental study aims (1) to compare the effects of three written corrective feedback (WCF) techniques on linguistic accuracy in writing. The three techniques consist of direct WCF, a technique in which the teacher provides the correct form, and two indirect WCF types, underlining only versus underlining plus metalinguistic clues, techniques in which the teacher elicits the correct form from the student. This study also aims (2) to examine the moderating effects of error category, and (3) to explore the moderating effects of individual variables such as language proficiency and language analytical ability. Four groups of French as a second language students (n = 65) enrolled in enriched French language programs in English secondary schools in Montreal participated in this study. They were divided into three experimental conditions, namely 1) direct WCF, 2) underlining only and 3) underlining plus metalinguistic clues, and a comparison condition - no WCF. The experimental intervention, which consisted of three cycles of writing, feedback and revision, was completed over two-week intervals. It targeted five grammatical categories: agreement in the nominal phrase (NP), agreement in the verb phrase (VP), the structure of the NP, the structure of the VP as well as grammatical homophones. Each group completed six writing tasks (text-reconstruction tasks): one task during the pre-test and the immediate post-test, another task for the delayed post-test and three different tasks during the experimental intervention. All four groups revised the texts they completed during the intervention. All participants also completed a test of language analytical ability. Results indicate that (1) providing WCF was more beneficial than withholding it, (2) indirect WCF was better than direct WCF, (3) that indirect WCF plus metalinguistic clues was more effective than indirect WCF only. Regarding the second question, results suggest that some error categories were more responsive to WCF, regardless of its type, than others. While no improvement was found for agreement in the NP or for agreement in the VP, the accurate use of grammatical homophones increased for all groups. Such an increase was more lasting for the group that received indirect WCF that is combined with metalinguistic clues. As for the third question, results indicate that proficiency played a moderating role only for the group receiving indirect WCF that is combined with metalinguistic clues, but no such effects were observed for the language analytical abilities, irrespective of the experimental condition

TinkRBooks : tinkerable story elements for emergent literacy

Thesis (Ph. D.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 92-102).Printed words are an abstract representation of concepts. Today parents teach children how to read by demonstrating how text is related to imagery. I present textual tinkerability, an idea for demonstrating reading by using multisensory gestures to expose and alter the text-graphic relationship within the story. Tinkerability allows readers to physically express words as they read, giving them some degree of control over the narrative. Two interactive storybooks called TinkRBooks demonstrate how tinkerability supports parent-child emergent literacy. Design guidelines were developed to showcase how tinkerability can be used for creating educationally meaningful interactivity. TinkRBooks allows parents to gesturally modify and discuss how text relates to concepts within a narrative. TinkRBooks allows children to actively explore the abstract relationship between printed words and their meanings, even before this relationship is properly understood. This ability to explore textual representation changes the way parents read to their children during emergent literacy. When using a TinkRBook, parents spend more time talking, discussing more comprehensive ideas with their children and provoking more meta dialogue than with regular books. TinkRBook also encourages children to drive their reading inquiry, by actively demonstrating the concepts relating to vocabulary schema within the narrative. The result is a new story sharing experience that benefits both parents and children by allowing them to understand how the choice of words impacts the story experience.by Angela Chang.Ph.D

Dialogic feedback and its effects on English language learners’ writing development: a case study

This exploratory case study aimed to investigate the meanings that second language (L2) learners derive from their writing purposes revising processes and their engagement with dialogic feedback. The goal was to grasp the meaning that revision experiences had for L2 learners (n=17) and to understand the context within which they were operating and its influence on their revision behavior. In order to gain a better understanding of L2 learner engagement and investment in the revision processes which were bound to time and context the case study was built upon an ecological-semiotic approach to learning. The case study aimed to construct a clearer reality within a bounded system – a culture of a classroom feedback practices learner and teacher histories – and bounded by time (4 9 and 30 months) to understand why some L2 learners benefited from feedback and others did not. Drawing on dialogic interactions interviews writing samples and intensive revisions qualitative interpretations were used to discover factors that explained L2 learner variation in receiving processing and applying dialogic feedback. The findings suggest that (1) dialogic feedback acted as a communicative sign-making process that developed L2 learners’ writing and revisions and encompassed writing as a personal act that occurred in a classroom setting; (2) L2 learners encountered unique opportunities and constraints that afforded and limited their L2 writing learning experiences in a variety of ways in which individual differences played a significant role; (3) the way L2 learners responded to a writing task differed according to how they interpreted the goal of the task or the goal they set for themselves in relation to the task and task conditions; (4) L2 learners may hold different even contradictory beliefs about L2 writing and revisions at different times influenced by diverse personal and contextual factors which shaped their approaches to revisions; and (5) L2 learners varied enormously with regard to revisions as well as affective factors such as writing anxiety attitude and investment. The results of the study present a better understanding of L2 learners’ perspectives and the value of feedback as a revising tool

Information-theoretic causal inference of lexical flow

This volume seeks to infer large phylogenetic networks from phonetically encoded lexical data and contribute in this way to the historical study of language varieties. The technical step that enables progress in this case is the use of causal inference algorithms. Sample sets of words from language varieties are preprocessed into automatically inferred cognate sets, and then modeled as information-theoretic variables based on an intuitive measure of cognate overlap. Causal inference is then applied to these variables in order to determine the existence and direction of influence among the varieties. The directed arcs in the resulting graph structures can be interpreted as reflecting the existence and directionality of lexical flow, a unified model which subsumes inheritance and borrowing as the two main ways of transmission that shape the basic lexicon of languages. A flow-based separation criterion and domain-specific directionality detection criteria are developed to make existing causal inference algorithms more robust against imperfect cognacy data, giving rise to two new algorithms. The Phylogenetic Lexical Flow Inference (PLFI) algorithm requires lexical features of proto-languages to be reconstructed in advance, but yields fully general phylogenetic networks, whereas the more complex Contact Lexical Flow Inference (CLFI) algorithm treats proto-languages as hidden common causes, and only returns hypotheses of historical contact situations between attested languages. The algorithms are evaluated both against a large lexical database of Northern Eurasia spanning many language families, and against simulated data generated by a new model of language contact that builds on the opening and closing of directional contact channels as primary evolutionary events. The algorithms are found to infer the existence of contacts very reliably, whereas the inference of directionality remains difficult. This currently limits the new algorithms to a role as exploratory tools for quickly detecting salient patterns in large lexical datasets, but it should soon be possible for the framework to be enhanced e.g. by confidence values for each directionality decision

Hesitations in Spoken Dialogue Systems

Betz S. Hesitations in Spoken Dialogue Systems. Bielefeld: Universität Bielefeld; 2020