Domain-adaptive pre-training (or DA-training for short), also known as
post-training, aims to train a pre-trained general-purpose language model (LM)
using an unlabeled corpus of a particular domain to adapt the LM so that
end-tasks in the domain can give improved performances. However, existing
DA-training methods are in some sense blind as they do not explicitly identify
what knowledge in the LM should be preserved and what should be changed by the
domain corpus. This paper shows that the existing methods are suboptimal and
proposes a novel method to perform a more informed adaptation of the knowledge
in the LM by (1) soft-masking the attention heads based on their importance to
best preserve the general knowledge in the LM and (2) contrasting the
representations of the general and the full (both general and domain knowledge)
to learn an integrated representation with both general and domain-specific
knowledge. Experimental results will demonstrate the effectiveness of the
proposed approach.Comment: EMNLP 202