Multimodal Grounding for Language Processing

This survey discusses how recent developments in multimodal processing facilitate conceptual grounding of language. We categorize the information flow in multimodal processing with respect to cognitive models of human information processing and analyze different methods for combining multimodal representations. Based on this methodological inventory, we discuss the benefit of multimodal grounding for a variety of language processing tasks and the challenges that arise. We particularly focus on multimodal grounding of verbs which play a crucial role for the compositional power of language.Comment: The paper has been published in the Proceedings of the 27 Conference of Computational Linguistics. Please refer to this version for citations: https://www.aclweb.org/anthology/papers/C/C18/C18-1197

Senior Recital: Jacob Beinborn

Kemp Recital HallMarch 24, 2013Sunday Afternoon4:30 p.m

Analyzing Cognitive Plausibility of Subword Tokenization

Subword tokenization has become the de-facto standard for tokenization, although comparative evaluations of subword vocabulary quality across languages are scarce. Existing evaluation studies focus on the effect of a tokenization algorithm on the performance in downstream tasks, or on engineering criteria such as the compression rate. We present a new evaluation paradigm that focuses on the cognitive plausibility of subword tokenization. We analyze the correlation of the tokenizer output with the response time and accuracy of human performance on a lexical decision task. We compare three tokenization algorithms across several languages and vocabulary sizes. Our results indicate that the UnigramLM algorithm yields less cognitively plausible tokenization behavior and a worse coverage of derivational morphemes, in contrast with prior work

Making Prevention Work: Preventive structures and policies for children, youth and families: Comprehensive report. Materials about Prevention Vol. 15 June 2020.

This report maps preventive structures and policies for children, young people and families in 12 European countries. By examining what works in each of the countries surveyed, it aims to provide a foundation for the development of prevention policies across Europe. The report draws on a concept of prevention that is framed in universalist and integrative terms. The concept is universalist in that it addresses all children and young people, even those not seen as being “at-risk.” It is integrative because prevention should be organized from a child’s point of view, not in terms of administrative responsibilities. As such, this concept targets the establishment of prevention chains that link different institutions over the life-course. The report includes summary factsheets of the preventive concepts, structures and practices mapped in 12 EU member states (Austria, Czechia, Denmark, England (UK), Finland, France, Germany, Ireland, Lithuania, the Netherlands, Spain and Sweden). In addition, three in-depth case studies (Austria, France and the Netherlands) featuring data from interviews with experts and implementing actors are also presented

Cross-Lingual Transfer of Cognitive Processing Complexity

When humans read a text, their eye movements are influenced by the structural complexity of the input sentences. This cognitive phenomenon holds across languages and recent studies indicate that multilingual language models utilize structural similarities between languages to facilitate cross-lingual transfer. We use sentence-level eye-tracking patterns as a cognitive indicator for structural complexity and show that the multilingual model XLM-RoBERTa can successfully predict varied patterns for 13 typologically diverse languages, despite being fine-tuned only on English data. We quantify the sensitivity of the model to structural complexity and distinguish a range of complexity characteristics. Our results indicate that the model develops a meaningful bias towards sentence length but also integrates cross-lingual differences. We conduct a control experiment with randomized word order and find that the model seems to additionally capture more complex structural information

Probing Multilingual BERT for Genetic and Typological Signals

We probe the layers in multilingual BERT (mBERT) for phylogenetic and geographic language signals across 100 languages and compute language distances based on the mBERT representations. We 1) employ the language distances to infer and evaluate language trees, finding that they are close to the reference family tree in terms of quartet tree distance, 2) perform distance matrix regression analysis, finding that the language distances can be best explained by phylogenetic and worst by structural factors and 3) present a novel measure for measuring diachronic meaning stability (based on cross-lingual representation variability) which correlates significantly with published ranked lists based on linguistic approaches. Our results contribute to the nascent field of typological interpretability of cross-lingual text representations.Comment: COLING 202

Dynamic Top-k Estimation Consolidates Disagreement between Feature Attribution Methods

Feature attribution scores are used for explaining the prediction of a text classifier to users by highlighting a k number of tokens. In this work, we propose a way to determine the number of optimal k tokens that should be displayed from sequential properties of the attribution scores. Our approach is dynamic across sentences, method-agnostic, and deals with sentence length bias. We compare agreement between multiple methods and humans on an NLI task, using fixed k and dynamic k. We find that perturbation-based methods and Vanilla Gradient exhibit highest agreement on most method--method and method--human agreement metrics with a static k. Their advantage over other methods disappears with dynamic ks which mainly improve Integrated Gradient and GradientXInput. To our knowledge, this is the first evidence that sequential properties of attribution scores are informative for consolidating attribution signals for human interpretation

Making Prevention Work: Case Study Netherlands. Materials about Prevention Vol. 18 June 2020

As part of a larger project mapping preventive structures and policies for children, young people and families in 12 European countries, the Making Prevention Work study aims to provide a consistent base for developing preventive policies in Europe. It examines approaches across the EU that demonstrate success with local preventive work. The in-depth case study of the Netherlands presented in this publication is one of three published in the context of the Making Prevention Work study. Making Prevention Work draws on a concept of prevention that is framed in universalist and integrative terms. The concept is universalist in that it addresses all children and young people, even those not seen as being “at-risk.” It is integrative because prevention should be organized from a child’s point of view, not in terms of administrative responsibilities. As such, this concept targets the establishment of prevention chains that link different institutions over the life-course. Making Prevention Work includes summary factsheets of the preventive concepts, structures and practices mapped in 12 EU member states (Austria, Czech Republic, Denmark, England (UK), Finland, France, Germany, Ireland, Lithuania, the Netherlands, Spain and Sweden) as well as three case studies (Austria, France and the Netherlands) featuring data from interviews with experts and implementing actors