We present Pre-trained Machine Reader (PMR), a novel method to retrofit
Pre-trained Language Models (PLMs) into Machine Reading Comprehension (MRC)
models without acquiring labeled data. PMR is capable of resolving the
discrepancy between model pre-training and downstream fine-tuning of existing
PLMs, and provides a unified solver for tackling various extraction tasks. To
achieve this, we construct a large volume of general-purpose and high-quality
MRC-style training data with the help of Wikipedia hyperlinks and design a Wiki
Anchor Extraction task to guide the MRC-style pre-training process. Although
conceptually simple, PMR is particularly effective in solving extraction tasks
including Extractive Question Answering and Named Entity Recognition, where it
shows tremendous improvements over previous approaches especially under
low-resource settings. Moreover, viewing sequence classification task as a
special case of extraction task in our MRC formulation, PMR is even capable to
extract high-quality rationales to explain the classification process,
providing more explainability of the predictions