745 research outputs found
Video Question Answering with Iterative Video-Text Co-Tokenization
Video question answering is a challenging task that requires understanding
jointly the language input, the visual information in individual video frames,
as well as the temporal information about the events occurring in the video. In
this paper, we propose a novel multi-stream video encoder for video question
answering that uses multiple video inputs and a new video-text iterative
co-tokenization approach to answer a variety of questions related to videos. We
experimentally evaluate the model on several datasets, such as MSRVTT-QA,
MSVD-QA, IVQA, outperforming the previous state-of-the-art by large margins.
Simultaneously, our model reduces the required GFLOPs from 150-360 to only 67,
producing a highly efficient video question answering model.Comment: ECCV 202
Learning Fine-Grained Visual Understanding for Video Question Answering via Decoupling Spatial-Temporal Modeling
While recent large-scale video-language pre-training made great progress in
video question answering, the design of spatial modeling of video-language
models is less fine-grained than that of image-language models; existing
practices of temporal modeling also suffer from weak and noisy alignment
between modalities. To learn fine-grained visual understanding, we decouple
spatial-temporal modeling and propose a hybrid pipeline, Decoupled
Spatial-Temporal Encoders, integrating an image- and a video-language encoder.
The former encodes spatial semantics from larger but sparsely sampled frames
independently of time, while the latter models temporal dynamics at lower
spatial but higher temporal resolution. To help the video-language model learn
temporal relations for video QA, we propose a novel pre-training objective,
Temporal Referring Modeling, which requires the model to identify temporal
positions of events in video sequences. Extensive experiments demonstrate that
our model outperforms previous work pre-trained on orders of magnitude larger
datasets.Comment: BMVC 2022. Code is available at https://github.com/shinying/des
Location-aware Graph Convolutional Networks for Video Question Answering
We addressed the challenging task of video question answering, which requires
machines to answer questions about videos in a natural language form. Previous
state-of-the-art methods attempt to apply spatio-temporal attention mechanism
on video frame features without explicitly modeling the location and relations
among object interaction occurred in videos. However, the relations between
object interaction and their location information are very critical for both
action recognition and question reasoning. In this work, we propose to
represent the contents in the video as a location-aware graph by incorporating
the location information of an object into the graph construction. Here, each
node is associated with an object represented by its appearance and location
features. Based on the constructed graph, we propose to use graph convolution
to infer both the category and temporal locations of an action. As the graph is
built on objects, our method is able to focus on the foreground action contents
for better video question answering. Lastly, we leverage an attention mechanism
to combine the output of graph convolution and encoded question features for
final answer reasoning. Extensive experiments demonstrate the effectiveness of
the proposed methods. Specifically, our method significantly outperforms
state-of-the-art methods on TGIF-QA, Youtube2Text-QA, and MSVD-QA datasets.
Code and pre-trained models are publicly available at:
https://github.com/SunDoge/L-GC
Harvest Video Foundation Models via Efficient Post-Pretraining
Building video-language foundation models is costly and difficult due to the
redundant nature of video data and the lack of high-quality video-language
datasets. In this paper, we propose an efficient framework to harvest video
foundation models from image ones. Our method is intuitively simple by randomly
dropping input video patches and masking out input text during the
post-pretraining procedure. The patch dropping boosts the training efficiency
significantly and text masking enforces the learning of cross-modal fusion. We
conduct extensive experiments to validate the effectiveness of our method on a
wide range of video-language downstream tasks including various zero-shot
tasks, video question answering, and video-text retrieval. Despite its
simplicity, our method achieves state-of-the-art performances, which are
comparable to some heavily pretrained video foundation models. Our method is
extremely efficient and can be trained in less than one day on 8 GPUs,
requiring only WebVid-10M as pretraining data. We hope our method can serve as
a simple yet strong counterpart for prevalent video foundation models, provide
useful insights when building them, and make large pretrained models more
accessible and sustainable. This is part of the InternVideo project
\url{https://github.com/OpenGVLab/InternVideo}
- …