Pre-trained language models have made great progress on dialogue tasks.
However, these models are typically trained on surface dialogue text, thus are
proven to be weak in understanding the main semantic meaning of a dialogue
context. We investigate Abstract Meaning Representation (AMR) as explicit
semantic knowledge for pre-training models to capture the core semantic
information in dialogues during pre-training. In particular, we propose a
semantic-based pre-training framework that extends the standard pre-training
framework (Devlin et al., 2019) by three tasks for learning 1) core semantic
units, 2) semantic relations and 3) the overall semantic representation
according to AMR graphs. Experiments on the understanding of both chit-chats
and task-oriented dialogues show the superiority of our model. To our
knowledge, we are the first to leverage a deep semantic representation for
dialogue pre-training.Comment: Accepted as oral in COLING202