266 research outputs found
-: Boosting isual ialog with Cascaded Spatial-Temporal Multi-Modal aphs
We propose - - a novel visual dialog model that
combines pre-trained language models (LMs) with graph neural networks (GNNs).
Prior works mainly focused on one class of models at the expense of the other,
thus missing out on the opportunity of combining their respective benefits. At
the core of - is a novel integration mechanism that
alternates between spatial-temporal multi-modal GNNs and BERT layers, and that
covers three distinct contributions: First, we use multi-modal GNNs to process
the features of each modality (image, question, and dialog history) and exploit
their local structures before performing BERT global attention. Second, we
propose hub-nodes that link to all other nodes within one modality graph,
allowing the model to propagate information from one GNN (modality) to the
other in a cascaded manner. Third, we augment the BERT hidden states with
fine-grained multi-modal GNN features before passing them to the next
- layer. Evaluations on VisDial v1.0, VisDial v0.9,
VisDialConv, and VisPro show that - achieves new
state-of-the-art results across all four datasets.Comment: WACV 202
- …