Generating multimedia presentations: from plain text to screenplay

In many Natural Language Generation (NLG) applications, the output is limited to plain text – i.e., a string of words with punctuation and paragraph breaks, but no indications for layout, or pictures, or dialogue. In several projects, we have begun to explore NLG applications in which these extra media are brought into play. This paper gives an informal account of what we have learned. For coherence, we focus on the domain of patient information leaflets, and follow an example in which the same content is expressed first in plain text, then in formatted text, then in text with pictures, and finally in a dialogue script that can be performed by two animated agents. We show how the same meaning can be mapped to realisation patterns in different media, and how the expanded options for expressing meaning are related to the perceived style and tone of the presentation. Throughout, we stress that the extra media are not simple added to plain text, but integrated with it: thus the use of formatting, or pictures, or dialogue, may require radical rewording of the text itself

Desiderata for an Every Citizen Interface to the National Information Infrastructure: Challenges for NLP

In this paper, I provide desiderata for an interface that would enable ordinary people to properly access the capabilities of the NII. I identify some of the technologies that will be needed to achieve these desiderata, and discuss current and future research directions that could lead to the development of such technologies. In particular, I focus on the ways in which theory and techniques from natural language processing could contribute to future interfaces to the NII. Introduction The evolving national information infrastructure (NII) has made available a vast array of on-line services and networked information resources in a variety of forms (text, speech, graphics, images, video). At the same time, advances in computing and telecommunications technology have made it possible for an increasing number of households to own (or lease or use) powerful personal computers that are connected to this resource. Accompanying this progress is the expectation that people will be able to more..

Integrated content presentation for multilingual and multimedia information access

For multilingual and multimedia information retrieval from multiple potentially distributed collections generating the output in the form of standard ranked lists may often mean that a user has to explore the contents of many lists before finding sufficient relevant or linguistically accessible material to satisfy their information need. In some situations delivering an integrated multilingual multimedia presentation could enable the user to explore a topic allowing them to select from among a range of available content based on suitably chosen displayed metadata. A presentation of this type has similarities with the outputs of existing adaptive hypermedia systems. However, such systems are generated based on “closed” content with sophisticated user and domain models. Extending them to “open” domain information retrieval applications would raise many issues. We present an outline exploration of what will form a challenging new direction for research in multilingual information access

Generating Explanatory Captions for Information Graphics

Graphical presentations can be used to communicate information in relational data sets succinctly and effectively. However, novel graphical presentations about numerous attributes and their relationships are often difficult to understand completely until explained. Automatically generated graphical presentations must therefore either be limited to simple, conventional ones, or risk incomprehensibility. One way of alleviating this problem is to design graphical presentation systems that can work in conjunction with a natural language generator to produce "explanatory captions." This paper presents three strategies for generating explanatory captions to accompany information graphics based on: (1) a representation of the structure of the graphical presentation (2) a framework for identifyingthe perceptual complexity of graphical elements, and (3) the structure of the data expressed in the graphic. We describe an implemented system and illustrate how it is used to generate explanatory cap..

Interactive Search and Exploration in Online Discussion Forums Using Multimodal Embeddings

In this paper we present a novel interactive multimodal learning system, which facilitates search and exploration in large networks of social multimedia users. It allows the analyst to identify and select users of interest, and to find similar users in an interactive learning setting. Our approach is based on novel multimodal representations of users, words and concepts, which we simultaneously learn by deploying a general-purpose neural embedding model. We show these representations to be useful not only for categorizing users, but also for automatically generating user and community profiles. Inspired by traditional summarization approaches, we create the profiles by selecting diverse and representative content from all available modalities, i.e. the text, image and user modality. The usefulness of the approach is evaluated using artificial actors, which simulate user behavior in a relevance feedback scenario. Multiple experiments were conducted in order to evaluate the quality of our multimodal representations, to compare different embedding strategies, and to determine the importance of different modalities. We demonstrate the capabilities of the proposed approach on two different multimedia collections originating from the violent online extremism forum Stormfront and the microblogging platform Twitter, which are particularly interesting due to the high semantic level of the discussions they feature

Applying science of learning in education: Infusing psychological science into the curriculum

The field of specialization known as the science of learning is not, in fact, one field. Science of learning is a term that serves as an umbrella for many lines of research, theory, and application. A term with an even wider reach is Learning Sciences (Sawyer, 2006). The present book represents a sliver, albeit a substantial one, of the scholarship on the science of learning and its application in educational settings (Science of Instruction, Mayer 2011). Although much, but not all, of what is presented in this book is focused on learning in college and university settings, teachers of all academic levels may find the recommendations made by chapter authors of service. The overarching theme of this book is on the interplay between the science of learning, the science of instruction, and the science of assessment (Mayer, 2011). The science of learning is a systematic and empirical approach to understanding how people learn. More formally, Mayer (2011) defined the science of learning as the “scientific study of how people learn” (p. 3). The science of instruction (Mayer 2011), informed in part by the science of learning, is also on display throughout the book. Mayer defined the science of instruction as the “scientific study of how to help people learn” (p. 3). Finally, the assessment of student learning (e.g., learning, remembering, transferring knowledge) during and after instruction helps us determine the effectiveness of our instructional methods. Mayer defined the science of assessment as the “scientific study of how to determine what people know” (p.3). Most of the research and applications presented in this book are completed within a science of learning framework. Researchers first conducted research to understand how people learn in certain controlled contexts (i.e., in the laboratory) and then they, or others, began to consider how these understandings could be applied in educational settings. Work on the cognitive load theory of learning, which is discussed in depth in several chapters of this book (e.g., Chew; Lee and Kalyuga; Mayer; Renkl), provides an excellent example that documents how science of learning has led to valuable work on the science of instruction. Most of the work described in this book is based on theory and research in cognitive psychology. We might have selected other topics (and, thus, other authors) that have their research base in behavior analysis, computational modeling and computer science, neuroscience, etc. We made the selections we did because the work of our authors ties together nicely and seemed to us to have direct applicability in academic settings

Generating multimedia briefings: coordinating language and illustration

AbstractCommunication can be more effective when several media (such as text, speech, or graphics) are integrated and coordinated to present information. This changes the nature of media-specific generation (e.g., language or graphics generation), which must take into account the multimedia context in which it occurs. This paper presents work on coordinating and integrating speech, text, static and animated three-dimensional graphics, and stored images, as part of several systems we have developed at Columbia University. A particular focus of our work has been on the generation of presentations that brief a user on information of interes

New metric products, movies and 3D models from old stereopairs and their application to the in situ palaeontological site of Ambrona

[ES] Este artículo está basado en la información del siguiente proyecto:● LDGP_mem_006-1: "[S_Ambrona_Insitu] Levantamiento fotogramétrico del yacimiento paleontológico “Museo in situ” de Ambrona (Soria)", http://hdl.handle.net/10810/7353● LDGP_mem_006-1: "[S_Ambrona_Insitu] Levantamiento fotogramétrico del yacimiento paleontológico “Museo in situ” de Ambrona (Soria)", http://hdl.handle.net/10810/7353[EN] This paper is based on the information gathered in the following project:[EN] 3D modelling tools from photographic pictures have experienced significant improvements in the last years. One of the most outstanding changes is the spread of the photogrammetric systems based on algorithms referred to as Structure from Motion (SfM) in contrast with the traditional stereoscopic pairs. Nevertheless, the availability of important collections of stereoscopic registers collected during past decades invites us to explore the possibilities for re-using these photographs in order to generate new multimedia products, especially due to the fact that many of the documented elements have been largely altered or even disappeared. This article analyses an example of application to the re-use of a collection of photographs from the palaeontological site of Ambrona (Soria, Spain). More specifically, different pieces of software based on Structure from Motion (SfM) algorithms for the generation of 3D models with photographic textures are tested and some derived products such as orthoimages, video or applications of Augmented Reality (AR) are presented.[ES] Las herramientas de modelado 3D a partir de imágenes fotográficas han experimentado avances muy significativos en los últimos años. Uno de los más destacados corresponde a la generalización de los sistemas fotogramétricos basados en los algoritmos denominados Structure from Motion (SfM) sobre los proyectos de documentación tradicional basados en pares estereoscópicos. La existencia de importantes colecciones de registros estereoscópicos realizados durante las décadas anteriores invita a explorar las posibilidades de reutilización de estos registros para la obtención de productos multimedia actuales, máxime cuando algunos de los elementos documentados han sufrido grandes modificaciones o incluso desaparecido. En el presente artículo se analiza la reutilización de colecciones fotográficas de yacimientos paleontológicos mediante un ejemplo centrado en el yacimiento de Ambrona (Soria, España). En concreto, se contrastan varios programas basados en los algoritmos denominados Structure from Motion (SfM) para la generación del modelo 3D con textura y otros productos derivados como ortoimágenes, vídeos o aplicaciones de Realidad Aumentada (RA)

Moving outside the box: Researching e-learning in disruptive times

Indexación: Scopus.The rise of technology’s influence in a cross-section of fields within formal education, not to mention in the broader social world, has given rise to new forms in the way we view learning, i.e. what constitutes valid knowledge and how we arrive at that knowledge. Some scholars have claimed that technology is but a tool to support the meaning-making that lies at the root of knowledge production while others argue that technology is increasingly and inextricably intertwined not just with knowledge construction but with changes to knowledge makers themselves. Regardless which side one stands in this growing debate, it is difficult to deny that the processes we use to research learning supported by technology in order to understand these growing intricacies, have profound implications. In this paper, my aim is to argue and defend a call in the research on ICT for a critical reflective approach to researching technology use. Using examples from qualitative research in e-learning I have conducted on three continents over 15 years, and in diverse educational contexts, I seek to unravel the means and justification for research approaches that can lead to closing the gap between research and practice. These studies combined with those from a cross-disciplinary array of fields support the promotion of a research paradigm that examines the socio-cultural contexts of learning with ICT, at a time that coincides with technology becoming a social networking facilitator. Beyond the examples and justification of the merits and power of qualitative research to uncover the stories that matter in these socially embodied e-learning contexts, I discuss the methodologically and ethically charged decisions using emerging affordances of technology for analyzing and representing results, including visual ethnography. The implications both for the consumers and producers of research of moving outside the box of established research practices are yet unfathomable but excitinghttp://www.ejel.org/volume15/issue1/p5