Toward Communicating Simple Sentences Using Pictorial Representations

This article discusses communicating simple sentences using pictorial representations

Text-to-Pictogram Summarization for Augmentative and Alternative Communication

Many people suffer from language disorders that affect their communicative capabilities. Augmentative and alternative communication devices assist learning process through graphical representation of common words. In this article, we present a complete text-to-pictogram system able to simplify complex texts and ease its comprehension with pictograms

Text-to-picture tools, systems, and approaches: a survey

Text-to-picture systems attempt to facilitate high-level, user-friendly communication between humans and computers while promoting understanding of natural language. These systems interpret a natural language text and transform it into a visual format as pictures or images that are either static or dynamic. In this paper, we aim to identify current difficulties and the main problems faced by prior systems, and in particular, we seek to investigate the feasibility of automatic visualization of Arabic story text through multimedia. Hence, we analyzed a number of well-known text-to-picture systems, tools, and approaches. We showed their constituent steps, such as knowledge extraction, mapping, and image layout, as well as their performance and limitations. We also compared these systems based on a set of criteria, mainly natural language processing, natural language understanding, and input/output modalities. Our survey showed that currently emerging techniques in natural language processing tools and computer vision have made promising advances in analyzing general text and understanding images and videos. Furthermore, important remarks and findings have been deduced from these prior works, which would help in developing an effective text-to-picture system for learning and educational purposes. - 2019, The Author(s).This work was made possible by NPRP grant #10-0205-170346 from the Qatar National Research Fund (a member of Qatar Foundation). The statements made herein are solely the responsibility of the authors

An Ontology based Text-to-Picture Multimedia m-Learning System

Multimedia Text-to-Picture is the process of building mental representation from words associated with images. From the research aspect, multimedia instructional message items are illustrations of material using words and pictures that are designed to promote user realization. Illustrations can be presented in a static form such as images, symbols, icons, figures, tables, charts, and maps; or in a dynamic form such as animation, or video clips. Due to the intuitiveness and vividness of visual illustration, many text to picture systems have been proposed in the literature like, Word2Image, Chat with Illustrations, and many others as discussed in the literature review chapter of this thesis. However, we found that some common limitations exist in these systems, especially for the presented images. In fact, the retrieved materials are not fully suitable for educational purposes. Many of them are not context-based and didn’t take into consideration the need of learners (i.e., general purpose images). Manually finding the required pedagogic images to illustrate educational content for learners is inefficient and requires huge efforts, which is a very challenging task. In addition, the available learning systems that mine text based on keywords or sentences selection provide incomplete pedagogic illustrations. This is because words and their semantically related terms are not considered during the process of finding illustrations. In this dissertation, we propose new approaches based on the semantic conceptual graph and semantically distributed weights to mine optimal illustrations that match Arabic text in the children’s story domain. We combine these approaches with best keywords and sentences selection algorithms, in order to improve the retrieval of images matching the Arabic text. Our findings show significant improvements in modelling Arabic vocabulary with the most meaningful images and best coverage of the domain in discourse. We also develop a mobile Text-to-Picture System that has two novel features, which are (1) a conceptual graph visualization (CGV) and (2) a visual illustrative assessment. The CGV shows the relationship between terms associated with a picture. It enables the learners to discover the semantic links between Arabic terms and improve their understanding of Arabic vocabulary. The assessment component allows the instructor to automatically follow up the performance of learners. Our experiments demonstrate the efficiency of our multimedia text-to-picture system in enhancing the learners’ knowledge and boost their comprehension of Arabic vocabulary

Toward Communicating Simple Sentences Using Pictorial Representations

This paper evaluates the hypothesis that pictorial representations can be used to effectively convey simple sentences across language barriers. Comparative evaluations show that a considerable amount of understanding can be achieved using visual descriptions of information, with evaluation figures within a comparable range of those obtained with linguistic representations produced by an automatic machine translation system.

From Cultural Diversity To Group Creativity: Using Language-Retrieved Pictures To Support Computer-Mediated Intercultural Brainstorming

International and intercultural groups increasingly perform various kinds of knowledge work that require groups to brainstorm or generate new ideas, such as problem solving, intelligence analysis and design. One observation based on the understanding of cultural differences and group idea generation suggests that cultures, or socially shared systems of concepts and practices among communities of people, introduce both benefits and obstacles to intercultural brainstorming. Cultural diversity in concepts and ways of thinking is in general beneficial, while cultural discrepancy in social norms, communication styles and language can be detrimental to idea sharing and brainstorming outcomes. The major goal of this dissertation is to reconcile the tension between the benefits and obstacles of intercultural collaboration. In this dissertation, I investigate how people with different cultural backgrounds communicate to perform brainstorming. I further propose brainstorming support tools accordingly, and evaluate the designs in the contexts of cross-cultural and cross-lingual brainstorming. The dissertation considers that using computers to retrieve and display language-retrieved pictures, which are pictures relevant to the ongoing conversation, can effectively support intercultural brainstorming. As individuals from different cultures vary in terms of how they perceive and interpret image content, the design attempts to present pictures to elicit diverse thoughts from members of intercultural groups. A study confirms the usefulness of this design for American-Chinese intercultural groups. The dissertation further considers to bridge cultures at the language level, using machine translation (MT) to allow group members to produce and read ideas in their native languages. Another study shows that MT supports the production of ideas but not the comprehension of ideas. The results point to the need to further investigate the detailed processes for producing and comprehending ideas in intercultural groups to inform future designs. The dissertation contributes to the understanding of computer-mediated intercultural brainstorming with behavioral studies and design work, and shows the need for technical designs to take understanding of various aspects of culture, such as social and communicative norms, cognition and languages spoken, into consideration