Video captioning aims to describe the content of videos using natural
language. Although significant progress has been made, there is still much room
to improve the performance for real-world applications, mainly due to the
long-tail words challenge. In this paper, we propose a text with knowledge
graph augmented transformer (TextKG) for video captioning. Notably, TextKG is a
two-stream transformer, formed by the external stream and internal stream. The
external stream is designed to absorb additional knowledge, which models the
interactions between the additional knowledge, e.g., pre-built knowledge graph,
and the built-in information of videos, e.g., the salient object regions,
speech transcripts, and video captions, to mitigate the long-tail words
challenge. Meanwhile, the internal stream is designed to exploit the
multi-modality information in videos (e.g., the appearance of video frames,
speech transcripts, and video captions) to ensure the quality of caption
results. In addition, the cross attention mechanism is also used in between the
two streams for sharing information. In this way, the two streams can help each
other for more accurate results. Extensive experiments conducted on four
challenging video captioning datasets, i.e., YouCookII, ActivityNet Captions,
MSRVTT, and MSVD, demonstrate that the proposed method performs favorably
against the state-of-the-art methods. Specifically, the proposed TextKG method
outperforms the best published results by improving 18.7% absolute CIDEr scores
on the YouCookII dataset.Comment: Accepted by CVPR202