Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT

Pretrained contextual representation models (Peters et al., 2018; Devlin et al., 2018) have pushed forward the state-of-the-art on many NLP tasks. A new release of BERT (Devlin, 2018) includes a model simultaneously pretrained on 104 languages with impressive performance for zero-shot cross-lingual transfer on a natural language inference task. This paper explores the broader cross-lingual potential of mBERT (multilingual) as a zero shot language transfer model on 5 NLP tasks covering a total of 39 languages from various language families: NLI, document classification, NER, POS tagging, and dependency parsing. We compare mBERT with the best-published methods for zero-shot cross-lingual transfer and find mBERT competitive on each task. Additionally, we investigate the most effective strategy for utilizing mBERT in this manner, determine to what extent mBERT generalizes away from language specific features, and measure factors that influence cross-lingual transfer.Comment: EMNLP 2019 Camera Read

Memory-Based Lexical Acquisition and Processing

Current approaches to computational lexicology in language technology are knowledge-based (competence-oriented) and try to abstract away from specific formalisms, domains, and applications. This results in severe complexity, acquisition and reusability bottlenecks. As an alternative, we propose a particular performance-oriented approach to Natural Language Processing based on automatic memory-based learning of linguistic (lexical) tasks. The consequences of the approach for computational lexicology are discussed, and the application of the approach on a number of lexical acquisition and disambiguation tasks in phonology, morphology and syntax is described.Comment: 18 page

Automatic case acquisition from texts for process-oriented case-based reasoning

This paper introduces a method for the automatic acquisition of a rich case representation from free text for process-oriented case-based reasoning. Case engineering is among the most complicated and costly tasks in implementing a case-based reasoning system. This is especially so for process-oriented case-based reasoning, where more expressive case representations are generally used and, in our opinion, actually required for satisfactory case adaptation. In this context, the ability to acquire cases automatically from procedural texts is a major step forward in order to reason on processes. We therefore detail a methodology that makes case acquisition from processes described as free text possible, with special attention given to assembly instruction texts. This methodology extends the techniques we used to extract actions from cooking recipes. We argue that techniques taken from natural language processing are required for this task, and that they give satisfactory results. An evaluation based on our implemented prototype extracting workflows from recipe texts is provided.Comment: Sous presse, publication pr\'evue en 201

Analysis of delayed product differentiation under pull type policies

Delayed product differentiation (DPD) increases manufacturers\u27 competitiveness in the market by enabling them to more quickly respond to changes in customers\u27 demands. DPD has also been shown to require less Work-in-Process (WIP) than a non-DPD setup in some cases. Previous research was mainly focused on the level of semi-finished and/or finished good inventory under a base-stock policy. The control of WIP inventory was not considered. DPD may also improve response times under pull inventory control schemes, in which the amount of WIP is controlled directly. These systems can be modeled as closed queueing networks in which a fixed number of kanbans circulate as customers among each set of one or more processing stages.;In this study, we first developed models to analyze the performance of simple kanban and CONstant-WIP (CONWIP) controlled systems and set the number of kanbans to achieve a specified performance level. The models help us better understand the behavior of pull systems. The performance evaluation method uses nonlinear programming (NLP) models to bound the throughput for fixed number of kanbans or minimize the number of kanbans necessary to achieve a specified throughput. The model shows how random supplies and demands prevent equilibrium from occurring in a single-stage kanbans system.;We studied a model for a system of two products with unlimited supply and demand using three CONWIP loops to represent the common processes and the differentiated processes for each product. The same system after DPD has more common processes and fewer differentiated processes. The NLP model can determine numbers of kanbans for each loop to achieve specified throughput targets. Because the throughput bounds are not as tight as desired, we developed a heuristic algorithm that starts from the NLP solution and adjusts the kanbans using simulation to evaluate the performance. A comparison of the result of the heuristic algorithm for the systems with and without DPD indicates that DPD reduces the amount of WIP necessary to achieve a specified throughput. Furthermore, we show how models of systems with similar structure can be generalized

Machine Learning and the Cognitive Basis of Natural Language

Articl