42 research outputs found
Dialect-robust Evaluation of Generated Text
Evaluation metrics that are not robust to dialect variation make it
impossible to tell how well systems perform for many groups of users, and can
even penalize systems for producing text in lower-resource dialects. However,
currently, there exists no way to quantify how metrics respond to change in the
dialect of a generated utterance. We thus formalize dialect robustness and
dialect awareness as goals for NLG evaluation metrics. We introduce a suite of
methods and corresponding statistical tests one can use to assess metrics in
light of the two goals. Applying the suite to current state-of-the-art metrics,
we demonstrate that they are not dialect-robust and that semantic perturbations
frequently lead to smaller decreases in a metric than the introduction of
dialect features. As a first step to overcome this limitation, we propose a
training schema, NANO, which introduces regional and language information to
the pretraining process of a metric. We demonstrate that NANO provides a
size-efficient way for models to improve the dialect robustness while
simultaneously improving their performance on the standard metric benchmark
FormNetV2: Multimodal Graph Contrastive Learning for Form Document Information Extraction
The recent advent of self-supervised pre-training techniques has led to a
surge in the use of multimodal learning in form document understanding.
However, existing approaches that extend the mask language modeling to other
modalities require careful multi-task tuning, complex reconstruction target
designs, or additional pre-training data. In FormNetV2, we introduce a
centralized multimodal graph contrastive learning strategy to unify
self-supervised pre-training for all modalities in one loss. The graph
contrastive objective maximizes the agreement of multimodal representations,
providing a natural interplay for all modalities without special customization.
In addition, we extract image features within the bounding box that joins a
pair of tokens connected by a graph edge, capturing more targeted visual cues
without loading a sophisticated and separately pre-trained image embedder.
FormNetV2 establishes new state-of-the-art performance on FUNSD, CORD, SROIE
and Payment benchmarks with a more compact model size.Comment: Accepted to ACL 202
Relatório de estágio em farmácia comunitária
Relatório de estágio realizado no âmbito do Mestrado Integrado em Ciências Farmacêuticas, apresentado à Faculdade de Farmácia da Universidade de Coimbr
More Constructions, More Genres: Extending Stanford Dependencies
The Stanford dependency scheme aims to provide a simple and intuitive but linguistically sound way of annotating the dependencies between words in a sentence. In this paper, we address two limitations the scheme has suffered from: First, despite providing good coverage of core grammatical relations, the scheme has not offered explicit analyses of more difficult syntactic constructions; second, because the scheme was initially developed primarily on newswire data, it did not focus on constructions that are rare in newswire but very frequent in more informal texts, such as casual speech and current web texts. Here, we propose dependency analyses for several linguistically interesting constructions and extend the scheme to provide better coverage of modern web dat
Universal Dependencies 1.0
Universal Dependencies is a project that seeks to develop cross-linguistically consistent treebank annotation for many languages, with the goal of facilitating multilingual parser development, cross-lingual learning, and parsing research from a language typology perspective. The annotation scheme is based on (universal) Stanford dependencies (de Marneffe et al., 2006, 2008, 2014), Google universal part-of-speech tags (Petrov et al., 2012), and the Interset interlingua for morphosyntactic tagsets (Zeman, 2008)