In this paper, we address the hallucination problem commonly found in natural
language generation tasks. Language models often generate fluent and convincing
content but can lack consistency with the provided source, resulting in
potential inaccuracies. We propose a new decoding method called
Fidelity-Enriched Contrastive Search (FECS), which augments the contrastive
search framework with context-aware regularization terms. FECS promotes tokens
that are semantically similar to the provided source while penalizing
repetitiveness in the generated text. We demonstrate its effectiveness across
two tasks prone to hallucination: abstractive summarization and dialogue
generation. Results show that FECS consistently enhances faithfulness across
various language model sizes while maintaining output diversity comparable to
well-performing decoding algorithms.Comment: Accepted as a short paper at EMNLP 202