89 research outputs found
Learning Latent Super-Events to Detect Multiple Activities in Videos
In this paper, we introduce the concept of learning latent super-events from
activity videos, and present how it benefits activity detection in continuous
videos. We define a super-event as a set of multiple events occurring together
in videos with a particular temporal organization; it is the opposite concept
of sub-events. Real-world videos contain multiple activities and are rarely
segmented (e.g., surveillance videos), and learning latent super-events allows
the model to capture how the events are temporally related in videos. We design
temporal structure filters that enable the model to focus on particular
sub-intervals of the videos, and use them together with a soft attention
mechanism to learn representations of latent super-events. Super-event
representations are combined with per-frame or per-segment CNNs to provide
frame-level annotations. Our approach is designed to be fully differentiable,
enabling end-to-end learning of latent super-event representations jointly with
the activity detector using them. Our experiments with multiple public video
datasets confirm that the proposed concept of latent super-event learning
significantly benefits activity detection, advancing the state-of-the-arts.Comment: CVPR 201
Joint Adaptive Representations for Image-Language Learning
Image-language learning has made unprecedented progress in visual
understanding. These developments have come at high costs, as contemporary
vision-language models require large model scales and amounts of data. We here
propose a much easier recipe for image-language learning, which produces
effective models, outperforming bigger and more expensive ones, often trained
on orders of magnitude larger datasets. Our key finding is the joint learning
of a compact vision and language representation, which adaptively and
iteratively fuses the multi-modal features. This results in a more effective
image-language learning, greatly lowering the FLOPs by combining and reducing
the number of tokens for both text and images, e.g. a 33\% reduction in FLOPs
is achieved, compared to baseline fusion techniques used by popular
image-language models, while improving performance. This also allows the model
to scale without a large increase in FLOPs or memory. In addition, we propose
adaptive pre-training data sampling which improves the data efficiency. The
proposed approach achieves competitive performance compared to much larger
models, and does so with significantly less data and FLOPs. With only 40M
training examples and with 39 GFLOPs our lightweight model outperforms many
times larger state-of-the-art models of 2-20x more FLOPs and using bigger
datasets some of which with close to 1B training examples.Comment: T4V Worksho
- …