Evaluating Information Retrieval and Access Tasks

This open access book summarizes the first two decades of the NII Testbeds and Community for Information access Research (NTCIR). NTCIR is a series of evaluation forums run by a global team of researchers and hosted by the National Institute of Informatics (NII), Japan. The book is unique in that it discusses not just what was done at NTCIR, but also how it was done and the impact it has achieved. For example, in some chapters the reader sees the early seeds of what eventually grew to be the search engines that provide access to content on the World Wide Web, today’s smartphones that can tailor what they show to the needs of their owners, and the smart speakers that enrich our lives at home and on the move. We also get glimpses into how new search engines can be built for mathematical formulae, or for the digital record of a lived human life. Key to the success of the NTCIR endeavor was early recognition that information access research is an empirical discipline and that evaluation therefore lay at the core of the enterprise. Evaluation is thus at the heart of each chapter in this book. They show, for example, how the recognition that some documents are more important than others has shaped thinking about evaluation design. The thirty-three contributors to this volume speak for the many hundreds of researchers from dozens of countries around the world who together shaped NTCIR as organizers and participants. This book is suitable for researchers, practitioners, and students—anyone who wants to learn about past and present evaluation efforts in information retrieval, information access, and natural language processing, as well as those who want to participate in an evaluation task or even to design and organize one

Knowledge Expansion of a Statistical Machine Translation System using Morphological Resources

Translation capability of a Phrase-Based Statistical Machine Translation (PBSMT) system mostly depends on parallel data and phrases that are not present in the training data are not correctly translated. This paper describes a method that efficiently expands the existing knowledge of a PBSMT system without adding more parallel data but using external morphological resources. A set of new phrase associations is added to translation and reordering models; each of them corresponds to a morphological variation of the source/target/both phrases of an existing association. New associations are generated using a string similarity score based on morphosyntactic information. We tested our approach on En-Fr and Fr-En translations and results showed improvements of the performance in terms of automatic scores (BLEU and Meteor) and reduction of out-of-vocabulary (OOV) words. We believe that our knowledge expansion framework is generic and could be used to add different types of information to the model.JRC.G.2-Global security and crisis managemen

Semantic consistency in text generation

Automatic input-grounded text generation tasks process input texts and generate human-understandable natural language text for the processed information. The development of neural sequence-to-sequence (seq2seq) models, which are usually trained in an end-to-end fashion, pushed the frontier of the performance on text generation tasks expeditiously. However, they are claimed to be defective in semantic consistency w.r.t. their corresponding input texts. Also, not only the models are to blame. The corpora themselves always include examples whose output is semantically inconsistent to its input. Any model that is agnostic to such data divergence issues will be prone to semantic inconsistency. Meanwhile, the most widely-used overlap-based evaluation metrics comparing the generated texts to their corresponding references do not evaluate the input-output semantic consistency explicitly, which makes this problem hard to detect. In this thesis, we focus on studying semantic consistency in three automatic text generation scenarios: Data-to-text Generation, Single Document Abstractive Summarization, and Chit-chat Dialogue Generation, by seeking for the answers to the following research questions: (1) how to define input-output semantic consistency in different text generation tasks? (2) how to quantitatively evaluate the input-output semantic consistency? (3) how to achieve better semantic consistency in individual tasks? We systematically define the semantic inconsistency phenomena in these three tasks as omission, intrinsic hallucination, and extrinsic hallucination. For Data-to-text Generation, we jointly learn a sentence planner that tightly controls which part of input source gets generated in what sequence, with a neural seq2seq text generator, to decrease all three types of semantic inconsistency in model-generated texts. The evaluation results confirm that the texts generated by our model contain much less omissions while maintaining low level of extrinsic hallucinations without sacrificing fluency compared to seq2seq models. For Single Document Abstractive Summarization, we reduce the level of extrinsic hallucinations in training data by automatically introducing assisting articles to each document-summary instance to provide the supplemental world-knowledge that is present in the summary but missing from the doc ument. With the help of a novel metric, we show that seq2seq models trained with as sisting articles demonstrate less extrinsic hallucinations than the ones trained without them. For Chit-chat Dialogue Generation, by filtering out the omitted and hallucinated examples from training set using a newly introduced evaluation metric, and encoding it into the neural seq2seq response generation models as a control factor, we diminish the level of omissions and extrinsic hallucinations in the generated dialogue responses

Graph-based Approaches to Text Generation

Deep Learning advances have enabled more fluent and flexible text generation. However, while these neural generative approaches were initially successful in tasks such as machine translation, they face problems – such as unfaithfulness to the source, repetition and incoherence – when applied to generation tasks where the input is structured data, such as graphs. Generating text from graph-based data, including Abstract Meaning Representation (AMR) or Knowledge Graphs (KG), is a challenging task due to the inherent difficulty of properly encoding the input graph while maintaining its original semantic structure. Previous work requires linearizing the input graph, which makes it complicated to properly capture the graph structure since the linearized representation weakens structural information by diluting the explicit connectivity, particularly when the graph structure is complex. This thesis makes an attempt to tackle these issues focusing on two major challenges: first, the creation and improvement of neural text generation systems that can better operate when consuming graph-based input data. Second, we examine text-to-text pretrained language models for graph-to-text generation, including multilingual generation, and present possible methods to adapt these models pretrained on natural language to graph-structured data. In the first part of this thesis, we investigate how to directly exploit graph structures for text generation. We develop novel graph-to-text methods with the capability of incorporating the input graph structure into the learned representations, enhancing the quality of the generated text. For AMR-to-text generation, we present a dual encoder, which incorporates different graph neural network methods, to capture complementary perspectives of the AMR graph. Next, we propose a new KG-to-text framework that learns richer contextualized node embeddings, combining global and local node contexts. We thus introduce a parameter-efficient mechanism for inserting the node connections into the Transformer architecture operating with shortest path lengths between nodes, showing strong performance while using considerably fewer parameters. The second part of this thesis focuses on pretrained language models for text generation from graph-based input data. We first examine how encoder-decoder text-to-text pretrained language models perform in various graph-to-text tasks and propose different task-adaptive pretraining strategies for improving their downstream performance. We then propose a novel structure-aware adapter method that allows to directly inject the input graph structure into pretrained models, without updating their parameters and reducing their reliance on specific representations of the graph structure. Finally, we investigate multilingual text generation from AMR structures, developing approaches that can operate in languages beyond English

Relevance, Rhetoric, and Argumentation: A Cross-Disciplinary Inquiry into Patterns of Thinking and Information Structuring

This dissertation research is a multidisciplinary inquiry into topicality, involving an in-depth examination of literatures and empirical data and an inductive development of a faceted typology (containing 227 fine-grained topical relevance relationships and 33 types of presentation relationship). This inquiry investigates a large variety of topical connections beyond topic matching, renders a closer look into the structure of a topic, achieves an enriched understanding of topicality and relevance, and induces a cohesive topic-oriented information architecture that is meaningful across topics and domains. The findings from the analysis contribute to the foundation work of information organization, intellectual access / information retrieval, and knowledge discovery. Using qualitative content analysis, the inquiry focuses on meaning and deep structure: Phase 1 : develop a unified theory-grounded typology of topical relevance relationships through close reading of literature and synthesis of thinking from communication, rhetoric, cognitive psychology, education, information science, argumentation, logic, law, medicine, and art history; Phase 2 : in-depth qualitative analysis of empirical relevance datasets in oral history, clinical question answering, and art image tagging, to examine manifestations of the theory-grounded typology in various contexts and to further refine the typology; the three relevance datasets were used for analysis to achieve variation in form, domain, and context. The typology of topical relevance relationships is structured with three major facets: Functional role of a piece of information plays in the overall structure of a topic or an argument; Mode of reasoning: How information contributes to the user's reasoning about a topic; Semantic relationship: How information connects to a topic semantically. This inquiry demonstrated that topical relevance with its close linkage to thinking and reasoning is central to many disciplines. The multidisciplinary approach allows synthesis and examination from new angles, leading to an integrated scheme of relevance relationships or a system of thinking that informs each individual discipline. The scheme resolving from the synthesis can be used to improve text and image understanding, knowledge organization and retrieval, reasoning, argumentation, and thinking in general, by people and machines

Quantifying diversity in user experience

Evaluation should be integral to any design activity. Evaluation in innovative product development practices however is highly complicated. It often needs to be applied to immature prototypes, while at the same time users’ responses may greatly vary across different individuals and situations. This thesis has focused on methods and tools for inquiring into users’ experiences with interactive products. More specifically, it had three objectives: a) to conceptualize the notion of diversity in subjective judgments of users’ experiences with interactive products, b) to establish empirical evidence for the prevalence of diversity, and c) to provide a number of methodological tools for the study of diversity in the context of product development. Two critical sources of diversity in the context of users’ experiences with interactive products were identified and respective methodological solutions were proposed: a) understanding interpersonal diversity through personal attribute judgments, and b) understanding the dynamics of experience through experience narratives. Personal Attribute Judgments, and in particular, the Repertory Grid Technique, is proposed as an alternative to standardized psychometric scales, in measuring users’ responses to artifacts in the context of parallel design. It is argued that traditional approaches that rely on the a-priori definition of the measures by the researchers have at least two limitations. First, such approaches are inherently limited as researchers might fail to consider a given dimension as relevant for the given product and context, or they might simply lack validated measurement scales for a relevant dimension. Secondly, such approaches assume that participants are able to interpret and position a given statement that is defined by the researcher to their own context. Recent literature has challenged this assumption, suggesting that in certain cases participants are unable to interpret the personal relevance of the statement in their own context, and might instead employ shallow processing, that is respond to surface features of the language rather than attaching personal relevance to the question. In contrast, personal attributes are elicited from each individual respondent, instead of being a-priori imposed by the experimenter, and thus are supposed to be highly relevant to the individual. However, personal attributes require substantially more complex quantitative analysis procedures. It is illustrated that traditional analysis procedures fail to bring out the richness of the personal attribute judgments and two new Multi-Dimensional Scaling procedures that extract multiple complementary views from such datasets are proposed. An alternative approach for the measurement of the dynamics of experience over time is proposed that relies on a) the retrospective elicitation of idiosyncratic selfreports of one’s experiences with a product, the so-called experience narratives, and b) the extraction of generalized knowledge from these narratives through computational content analysis techniques. iScale, a tool that aims at increasing users’ accuracy and effectiveness in recalling their experiences with a product is proposed. iScale uses sketching in imposing a structured process in the reconstruction of one’s experiences from memory. Two different versions of iScale, each grounded in a distinct theory of how people reconstruct emotional experiences from memory, were developed and empirically tested. A computational approach for the extraction of generalized knowledge from experience narratives, that combines traditional coding procedures with computational approaches for assessing the semantic similarity between documents, is proposed and compared with traditional content analysis. Through these two methodological contributions, this thesis argues against averaging in the subjective evaluation of interactive products. It proposes the development of interactive tools that can assist designers in moving across multiple levels of abstractions of empirical data, as design-relevant knowledge might be found on all these levels

Flying together towards EFL teacher development as language learners and professionals through genre writing

Tese (doutorado) - Universidade Federal de Santa Catarina, Centro de Comunicação e Expressão, Programa de Pós-Graduação em Letras/Inglês e Literatura Correspondente, Florianópolis, 2009This qualitative study on the inter-relation between English as a Foreign Language Teacher Education (EFLTE) and the teaching of writing follows, mainly, the theoretical-methodological approach of Socio-discursive Interactionism (Bronckart, 2003, 2006, 2008a and his followers). Its general objective is to investigate in what aspects and to what extent an interventionist practice concerning the teaching of writing can contribute to EFL teachers development as language learners and professionals. Specifically, the aims of this study are: a) to identify which elements related to the language capacities of action, discursive and linguistic-discursive can be taught for the writing of an academic summary (AS); b) to investigate in what aspects and to what extent the process of writing an AS by means of a didactic sequence (DS) can contribute to teachers development; c) to investigate in what aspects and to what extent the process of planning a DS to the teaching of writing of specific genres can contribute to the teachers' development; and d) to investigate which individual representations were constructed during the processes of writing an AS and planning DS for the teaching of writing specific genres. Four main sets of data are analyzed: a corpus of ten ASs produced by the participant-teachers of an EFLTE course, participant-teachers DS plans, and participant-teachers direct self-confrontation texts. The results of the analysis of each set of data reveals that: 1) the elements related to the three language capacities to be studied for writing an AS should be: the thematic content and context of production of the academic article, descriptive type of sequence and theoretical type of discourse, affirmative sentences, present simple tense and present passive, nominal group, nominal and pronominal anaphora, logic modalization, connectors and reporting verbs; 2) the participants substantially developed as language learners, since they made considerable progress in the three language capacities from the first to the last AS versions; 3) the participants knowledge development related to the three language capacities in the task of planning DSs was partially adequate. Crossing the results from the second and third sets of data with the teaching knowledge base dimensions (Richards, 1998), named Theories of Teaching, Teaching Skills, Communication Skills, Pedagogical Knowledge Skills and Decision Making and Contextual Knowledge reveals that the participants developed in the six dimensions; 4) the participants also developed in the reflective dimensions named Epistemological, Ontological, Pedagogical, Linguistic and Axiological. The overall findings, therefore, reveal that a genre-based perspective for the teaching of writing through the use of the procedure of DS can be seen as an adequate theoretical, methodological and reflexive mechanism for EFLTE.Este estudo qualitativo sobre a inter-relação entre formação de professores de inglês como língua estrangeira e ensino de escrita segue, principalmente, a abordagem teórico-metodológica do Interacionismo Sociodiscursivo (Bronckart, 2003; 2006; 2008a e seus seguidores). Seu objetivo geral é investigar em que aspectos e até que ponto uma prática intervencionista relacionada ao ensino de escrita pode contribuir no desenvolvimento de professores de Inglês como língua estrangeira como aprendizes e profissionais. Especificamente, este estudo objetiva: a) identificar quais elementos relacionados às capacidades de linguagem de ação, discursiva e lingüístico-discursiva podem ser ensinados na escrita de um resumo acadêmico (RA); b) investigar de que forma e até que ponto o processo de escrita de um RA através do procedimento de sequência didática (SD) pode contribuir para o desenvolvimento de professores; c) investigar de que forma e até que ponto o processo de planejamento de uma SD para o ensino de escrita pode contribuir para o desenvolvimento de professores; e d) investigar que representações individuais foram construídas durante o processo de escrita de um RA e planejamento de uma SD para ensino de escrita de gêneros específicos. Quatro conjuntos de dados são analisados: um corpus de dez RAs, resumos acadêmicos produzidos por professores de inglês participantes de um curso de formação continuada, planos de SD dos professores participantes, e textos de auto-confrontação simples dos participantes. Os resultados da análise de cada conjunto de dados mostram que: 1) os elementos, relacionados às três capacidades de linguagem, a serem estudados na escrita de RAs devem ser: o conteúdo temático e o contexto de produção do artigo acadêmico, tipo de sequência descritiva e tipo de discurso teórico, orações afirmativas, presente simples e voz passiva no presente, grupo nominal e anáfora nominal e pronominal, modalização lógica, conectores e verbos de dizer; 2) os participantes se desenvolveram substancialmente como aprendizes da língua inglesa, pois progrediram consideravelmente nas três capacidades de linguagem da primeira à última versão dos RAs; 3) o desenvolvimento dos participantes com relação às três capacidades de linguagem na tarefa de planejamento de SDs foi parcialmente adequado. O cruzamento dos dados do segundo e terceiro conjuntos com as seis dimensões básicas de conhecimento de ensino (Richards, 1998), nomeadamente, Teorias de Ensino, Habilidades de Ensino, Habilidades de Comunicação, Habilidades Pedagógicas e Poder de Decisão e Conhecimento do Contexto revela que os participantes se desenvolveram nas seis dimensões; 4) os participantes também se desenvolveram nas dimensões reflexivas Epistemológica, Ontológica, Pedagógica, Linguística e Axiológica. Os resultados gerais deste estudo revelam que uma perspectiva de ensino de escrita baseada em gêneros textuais através do procedimento de SD pode ser um mecanismo teórico, metodológico e reflexivo adequado para o desenvolvimento de professores de inglês como língua estrangeira

Research on Phraseology Across Continents

The second volume of the IDP series contains papers by phraseologists from five continents: Europe, Australia, North America, South America and Asia, which were written within the framework of the project Intercontinental Dialogue on Phraseology, prepared and coordinated by Joanna Szerszunowicz, conducted by the University of Bialystok in cooperation with Kwansei Gakuin University in Japan. The book consists of the following parts: Dialogue on Phraseology, General and Corpus Linguistics & Phraseology, Lexicography & Phraseology, Contrastive Linguistics, Translation & Phraseology, Literature, Cultural Studies, Education & Phraseology. Dialogue contains two papers written by widely recognised phraseologists: professor Anita Naciscione from Latvia and professor Irine Goshkheteliani.The volume has been financed by the Philological Department of the University of Bialysto