Large-scale multi-modal training with image-text pairs imparts strong
generalization to CLIP model. Since training on a similar scale for videos is
infeasible, recent approaches focus on the effective transfer of image-based
CLIP to the video domain. In this pursuit, new parametric modules are added to
learn temporal information and inter-frame relationships which require
meticulous design efforts. Furthermore, when the resulting models are learned
on videos, they tend to overfit on the given task distribution and lack in
generalization aspect. This begs the following question: How to effectively
transfer image-level CLIP representations to videos? In this work, we show that
a simple Video Fine-tuned CLIP (ViFi-CLIP) baseline is generally sufficient to
bridge the domain gap from images to videos. Our qualitative analysis
illustrates that the frame-level processing from CLIP image-encoder followed by
feature pooling and similarity matching with corresponding text embeddings
helps in implicitly modeling the temporal cues within ViFi-CLIP. Such
fine-tuning helps the model to focus on scene dynamics, moving objects and
inter-object relationships. For low-data regimes where full fine-tuning is not
viable, we propose a `bridge and prompt' approach that first uses fine-tuning
to bridge the domain gap and then learns prompts on language and vision side to
adapt CLIP representations. We extensively evaluate this simple yet strong
baseline on zero-shot, base-to-novel generalization, few-shot and fully
supervised settings across five video benchmarks. Our code is available at
https://github.com/muzairkhattak/ViFi-CLIP.Comment: Accepted at CVPR 202