A major concern in using deep learning based generative models for
document-grounded dialogs is the potential generation of responses that are not
\textit{faithful} to the underlying document. Existing automated metrics used
for evaluating the faithfulness of response with respect to the grounding
document measure the degree of similarity between the generated response and
the document's content. However, these automated metrics are far from being
well aligned with human judgments. Therefore, to improve the measurement of
faithfulness, we propose a new metric that utilizes (Conditional) Point-wise
Mutual Information (PMI) between the generated response and the source
document, conditioned on the dialogue. PMI quantifies the extent to which the
document influences the generated response -- with a higher PMI indicating a
more faithful response. We build upon this idea to create a new decoding
technique that incorporates PMI into the response generation process to predict
more faithful responses. Our experiments on the BEGIN benchmark demonstrate an
improved correlation of our metric with human evaluation. We also show that our
decoding technique is effective in generating more faithful responses when
compared to standard decoding techniques on a set of publicly available
document-grounded dialog datasets.Comment: EMNLP 202