Arithmetic-based pretraining improving numeracy of pretrained language models

State-of-the-art pretrained language models tend to perform below their capabilities when applied out-of-the-box on tasks that require understanding and working with numbers. Recent work suggests two main reasons for this: (1) popular tokenisation algorithms have limited expressiveness for numbers, and (2) common pretraining objectives do not target numeracy. Approaches that address these shortcomings usually require architectural changes or pretraining from scratch. In this paper, we propose a new extended pretraining approach called Arithmetic-Based Pretraining that jointly addresses both in one extended pretraining step without requiring architectural changes or pretraining from scratch. Arithmetic-Based Pretraining combines contrastive learning to improve the number representation, and a novel extended pretraining objective called Inferable Number Prediction Task to improve numeracy. Our experiments show the effectiveness of Arithmetic-Based Pretraining in three different tasks that require improved numeracy, i.e., reading comprehension in the DROP dataset, inference-on-tables in the InfoTabs dataset, and table-to-text generation in the WikiBio and Sci-Gen datasets

Modelling of the preparation of masters of professional education for activities in the information and digital environment

У статті теоретично обґрунтовано модель підготовки та формування готовності майбутніх педагогів професійного навчання до діяльності в інформаційно-освітньому середовищі на основі розвитку їх цифрових компетентностей. У процесі аналізу видів діяльності майбутнього педагога професійного навчання ступеня вищої освіти магістра, зазначених у відповідному професійному стандарті, та галузі застосування його цифрових компетентностей виявлений їх взаємозв'язок і наводяться можливості розвитку цих компетентностей випускника в межах запропонованої освітньо-професійної програми підготовки магістрів професійної освіти. Програма побудована на основі вивчення прикладного профілю цифрових компетентностей майбутнього педагога професійного навчання. Методологія дослідження спирається на аналіз цифрових компетентностей педагога і логіку побудови предметної підготовки майбутніх магістрів професійної освіти до успішної та ефективної професійної діяльності в умовах інформаційно-цифрової діяльності. У дослідженні проаналізовано зміст основних цифрових компетентностей педагога професійного навчання; запропонована модель підготовки та формування професійної готовності випускника до діяльності в інформаційно-цифровому середовищі на основі розвитку відповідних компетентностей; запропоновано зміст освітньо-професійної програми «Комп’ютерні технології в управлінні та навчанні» підготовки магістра в межах галузі знань 01 Освіта/Педагогіка за спеціальністю 015.39 Професійна освіта (Цифрові технології) на засадах розвитку цифрових компетентностей майбутнього фахівця в межах його професійної підготовки. Показано, що для підвищення ефективності діяльності педагогів у сучасних умовах інформатизації та цифровізації суспільства і формування віртуальної соціально-освітньої системи необхідно введення в зміст їх професійної підготовки навчальних дисциплін, спрямованих на розвиток цифрових компетентностей на визначеному рівні.The article theoretically substantiates the model of training and formation of future teachers' of vocational education readiness to work in the information and educational environment based on the development of their digital competencies. Upon analyzing the types of activities of a future teacher with a master's degree that are specified in the professional standard and the field of application of their digital competencies, the research paper reveals their interconnection and presents the possibilities of developing these graduate competencies within the proposed educational and professional master's program. The program is elaborated upon studying the applied profile of the future teacher's digital competencies. The research methodology is based on the analysis of the digital competencies of the teacher and the logic of elaboration of the subject preparation of future masters for successful and effective professional activity in the context of information and digital activities. The content of the teacher’s basic digital competencies is analyzed in the study; a model of training and formation of a graduate's professional readiness to work in the information and digital environment based on the development of relevant competencies is proposed; the content of the educational and professional program "Computer Technologies in Management and Education" for the master's degree in the field of knowledge 01 Education/Pedagogy, specialty 015.39 Vocational Education (Digital Technologies) on the basis of the development of digital competencies of a future specialist within his/her professional training is suggested. The research demonstrates that it is necessary to include subjects aimed at developing digital competencies at a certain level in the content of their professional training in order to increase the effectiveness of teachers' activities in the modern conditions of informatization and digitalization of society and the formation of a virtual social and educational system

A new benchmark dataset with production methodology for short text semantic similarity algorithms

This research presents a new benchmark dataset for evaluating Short Text Semantic Similarity (STSS) measurement algorithms and the methodology used for its creation. The power of the dataset is evaluated by using it to compare two established algorithms, STASIS and Latent Semantic Analysis. This dataset focuses on measures for use in Conversational Agents; other potential applications include email processing and data mining of social networks. Such applications involve integrating the STSS algorithm in a complex system, but STSS algorithms must be evaluated in their own right and compared with others for their effectiveness before systems integration. Semantic similarity is an artifact of human perception; therefore its evaluation is inherently empirical and requires benchmark datasets derived from human similarity ratings. The new dataset of 64 sentence pairs, STSS-131, has been designed to meet these requirements drawing on a range of resources from traditional grammar to cognitive neuroscience. The human ratings are obtained from a set of trials using new and improved experimental methods, with validated measures and statistics. The results illustrate the increased challenge and the potential longevity of the STSS-131 dataset as the Gold Standard for future STSS algorithm evaluation. © 2013 ACM 1550-4875/2013/12-ART17 15.00

SciGen : a dataset for reasoning-aware text generation from scientific tables

We introduce SciGen, a new challenge dataset consisting of tables from scientific articles and their corresponding descriptions, for the task of reasoning-aware data-to-text generation. Describing scientific tables goes beyond the surface realization of the table content and requires reasoning over table values. The unique properties of SciGen are that (1) tables mostly contain numerical values, and (2) the corresponding descriptions require arithmetic reasoning. SciGen is the first dataset that assesses the arithmetic reasoning capabilities of generation models on complex input structures, such as tables from scientific articles, and thus it opens new avenues for future research in reasoning-aware text generation and evaluation. The core part of SciGen, including the test data, is annotated by one of the authors of the corresponding articles. Such expert annotations do not scale to large training data sizes. To tackle this, we propose a pipeline for automatically extracting high-quality table-description pairs from the LaTeX sources of scientific articles. We study the effectiveness of state-of-the-art data-to-text generation models on SciGen and evaluate the results using common metrics and human evaluation. Our results and analyses show that adding high-quality unsupervised training data improves the correctness and reduces the hallucination in generated descriptions, however, the ability of state-of-the-art models is still severely limited on this task

Falsesum : Generating document-level NLI examples for recognizing factual inconsistency in summarization

Neural abstractive summarization models are prone to generate summaries which are factually inconsistent with their source documents. Previous work has introduced the task of recognizing such factual inconsistency as a downstream application of natural language inference (NLI). However, state-of-the-art NLI models perform poorly in this context due to their inability to generalize to the target task. In this work, we show that NLI models can be effective for this task when the training data is augmented with high-quality task-oriented examples. We introduce Falsesum, a data generation pipeline leveraging a controllable text generation model to perturb human-annotated summaries, introducing varying types of factual inconsistencies. Unlike previously introduced document-level NLI datasets, our generated dataset contains examples that are diverse and inconsistent yet plausible. We show that models trained on a Falsesum-augmented NLI dataset improve the state-of-the-art performance across four benchmarks for detecting factual inconsistency in summarization