69,330 research outputs found
Implementation architectures for natural language generation
Generic software architectures aim to support re-use of components, focusing of research and development effort, and evaluation and comparison of approaches. In the field of natural language processing, generic frameworks for understanding have been successfully deployed to meet all of these aims, but nothing comparable yet exists for generation. The nature of the task itself, and the current methodologies available to research it, seem to make it more difficult to reach the necessary level of consensus to support generic proposals. Recent work has made progress towards establishing a generic framework for generation at the functional level, but left open the issue of actual implementation. In this paper, we discuss the requirements for such an implementation layer for generation systems, drawing on two initial attempts to implement it. We argue that it is possible and useful to distinguish âfunctional architecture â from âimplementation architectureâ for generation systems. 1 The Case for a Generic Software Architecture for NLG Most natural language generation (NLG) systems have some kind of modular structure
Sequence-to-Sequence Spanish Pre-trained Language Models
In recent years, substantial advancements in pre-trained language models have
paved the way for the development of numerous non-English language versions,
with a particular focus on encoder-only and decoder-only architectures. While
Spanish language models encompassing BERT, RoBERTa, and GPT have exhibited
prowess in natural language understanding and generation, there remains a
scarcity of encoder-decoder models designed for sequence-to-sequence tasks
involving input-output pairs. This paper breaks new ground by introducing the
implementation and evaluation of renowned encoder-decoder architectures,
exclusively pre-trained on Spanish corpora. Specifically, we present Spanish
versions of BART, T5, and BERT2BERT-style models and subject them to a
comprehensive assessment across a diverse range of sequence-to-sequence tasks,
spanning summarization, rephrasing, and generative question answering. Our
findings underscore the competitive performance of all models, with BART and T5
emerging as top performers across all evaluated tasks. As an additional
contribution, we have made all models publicly available to the research
community, fostering future exploration and development in Spanish language
processing
PyCUDA and PyOpenCL: A Scripting-Based Approach to GPU Run-Time Code Generation
High-performance computing has recently seen a surge of interest in
heterogeneous systems, with an emphasis on modern Graphics Processing Units
(GPUs). These devices offer tremendous potential for performance and efficiency
in important large-scale applications of computational science. However,
exploiting this potential can be challenging, as one must adapt to the
specialized and rapidly evolving computing environment currently exhibited by
GPUs. One way of addressing this challenge is to embrace better techniques and
develop tools tailored to their needs. This article presents one simple
technique, GPU run-time code generation (RTCG), along with PyCUDA and PyOpenCL,
two open-source toolkits that support this technique.
In introducing PyCUDA and PyOpenCL, this article proposes the combination of
a dynamic, high-level scripting language with the massive performance of a GPU
as a compelling two-tiered computing platform, potentially offering significant
performance and productivity advantages over conventional single-tier, static
systems. The concept of RTCG is simple and easily implemented using existing,
robust infrastructure. Nonetheless it is powerful enough to support (and
encourage) the creation of custom application-specific tools by its users. The
premise of the paper is illustrated by a wide range of examples where the
technique has been applied with considerable success.Comment: Submitted to Parallel Computing, Elsevie
Multiword expression aware neural machine translation
Multiword Expressions (MWEs) are a frequently occurring phenomenon found in all natural languages that is of great importance to linguistic theory, natural language processing applications, and machine translation systems. Neural Machine Translation (NMT) architectures do not handle these expression well and previous studies have not explicitly addressed MWEs in this framework. In this work, we show that using external linguistic resources and data augmentation we can improve both translations of MWEs that occur in the source, and the generation of MWEs on the target, and improve performance by up to 5.09 BLEU points on MWE test sets. We also devise a MWE score to specifically assess the quality of MWE translation which agrees with human evaluation. We make available the MWEscore implementation â along with MWE-annotated training sets and corpus-based lists of MWEs â for reproduction and extension
- âŠ