4 research outputs found
Chinese Spelling Correction as Rephrasing Language Model
This paper studies Chinese Spelling Correction (CSC), which aims to detect
and correct potential spelling errors in a given sentence. Current
state-of-the-art methods regard CSC as a sequence tagging task and fine-tune
BERT-based models on sentence pairs. However, we note a critical flaw in the
process of tagging one character to another, that the correction is excessively
conditioned on the error. This is opposite from human mindset, where
individuals rephrase the complete sentence based on its semantics, rather than
solely on the error patterns memorized before. Such a counter-intuitive
learning process results in the bottleneck of generalizability and
transferability of machine spelling correction. To address this, we propose
(ReLM), where the model is trained to rephrase
the entire sentence by infilling additional slots, instead of
character-to-character tagging. This novel training paradigm achieves the new
state-of-the-art results across fine-tuned and zero-shot CSC benchmarks,
outperforming previous counterparts by a large margin. Our method also learns
transferable language representation when CSC is jointly trained with other
tasks
Rethinking Masked Language Modeling for Chinese Spelling Correction
In this paper, we study Chinese Spelling Correction (CSC) as a joint decision
made by two separate models: a language model and an error model. Through
empirical analysis, we find that fine-tuning BERT tends to over-fit the error
model while under-fit the language model, resulting in poor generalization to
out-of-distribution error patterns. Given that BERT is the backbone of most CSC
models, this phenomenon has a significant negative impact. To address this
issue, we are releasing a multi-domain benchmark LEMON, with higher quality and
diversity than existing benchmarks, to allow a comprehensive assessment of the
open domain generalization of CSC models. Then, we demonstrate that a very
simple strategy, randomly masking 20\% non-error tokens from the input sequence
during fine-tuning is sufficient for learning a much better language model
without sacrificing the error model. This technique can be applied to any model
architecture and achieves new state-of-the-art results on SIGHAN, ECSpell, and
LEMON.Comment: Accepted by ACL'202