258 research outputs found
FF2: A Feature Fusion Two-Stream Framework for Punctuation Restoration
To accomplish punctuation restoration, most existing methods focus on
introducing extra information (e.g., part-of-speech) or addressing the class
imbalance problem. Recently, large-scale transformer-based pre-trained language
models (PLMS) have been utilized widely and obtained remarkable success.
However, the PLMS are trained on the large dataset with marks, which may not
fit well with the small dataset without marks, causing the convergence to be
not ideal. In this study, we propose a Feature Fusion two-stream framework
(FF2) to bridge the gap. Specifically, one stream leverages a pre-trained
language model to capture the semantic feature, while another auxiliary module
captures the feature at hand. We also modify the computation of multi-head
attention to encourage communication among heads. Then, two features with
different perspectives are aggregated to fuse information and enhance context
awareness. Without additional data, the experimental results on the popular
benchmark IWSLT demonstrate that FF2 achieves new SOTA performance, which
verifies that our approach is effective.Comment: 5pages. arXiv admin note: substantial text overlap with
arXiv:2203.1248
Dependency Parsing with Dilated Iterated Graph CNNs
Dependency parses are an effective way to inject linguistic knowledge into
many downstream tasks, and many practitioners wish to efficiently parse
sentences at scale. Recent advances in GPU hardware have enabled neural
networks to achieve significant gains over the previous best models, these
models still fail to leverage GPUs' capability for massive parallelism due to
their requirement of sequential processing of the sentence. In response, we
propose Dilated Iterated Graph Convolutional Neural Networks (DIG-CNNs) for
graph-based dependency parsing, a graph convolutional architecture that allows
for efficient end-to-end GPU parsing. In experiments on the English Penn
TreeBank benchmark, we show that DIG-CNNs perform on par with some of the best
neural network parsers.Comment: 2nd Workshop on Structured Prediction for Natural Language Processing
(at EMNLP '17
Four-in-One: A Joint Approach to Inverse Text Normalization, Punctuation, Capitalization, and Disfluency for Automatic Speech Recognition
Features such as punctuation, capitalization, and formatting of entities are
important for readability, understanding, and natural language processing
tasks. However, Automatic Speech Recognition (ASR) systems produce spoken-form
text devoid of formatting, and tagging approaches to formatting address just
one or two features at a time. In this paper, we unify spoken-to-written text
conversion via a two-stage process: First, we use a single transformer tagging
model to jointly produce token-level tags for inverse text normalization (ITN),
punctuation, capitalization, and disfluencies. Then, we apply the tags to
generate written-form text and use weighted finite state transducer (WFST)
grammars to format tagged ITN entity spans. Despite joining four models into
one, our unified tagging approach matches or outperforms task-specific models
across all four tasks on benchmark test sets across several domains
- …