991 research outputs found
Slow and steady feature analysis: higher order temporal coherence in video
How can unlabeled video augment visual learning? Existing methods perform
"slow" feature analysis, encouraging the representations of temporally close
frames to exhibit only small differences. While this standard approach captures
the fact that high-level visual signals change slowly over time, it fails to
capture *how* the visual content changes. We propose to generalize slow feature
analysis to "steady" feature analysis. The key idea is to impose a prior that
higher order derivatives in the learned feature space must be small. To this
end, we train a convolutional neural network with a regularizer on tuples of
sequential frames from unlabeled video. It encourages feature changes over time
to be smooth, i.e., similar to the most recent changes. Using five diverse
datasets, including unlabeled YouTube and KITTI videos, we demonstrate our
method's impact on object, scene, and action recognition tasks. We further show
that our features learned from unlabeled video can even surpass a standard
heavily supervised pretraining approach.Comment: in Computer Vision and Pattern Recognition (CVPR) 2016, Las Vegas,
NV, June 201
Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization
We propose a self-supervised method for learning motion-focused video
representations. Existing approaches minimize distances between temporally
augmented videos, which maintain high spatial similarity. We instead propose to
learn similarities between videos with identical local motion dynamics but an
otherwise different appearance. We do so by adding synthetic motion
trajectories to videos which we refer to as tubelets. By simulating different
tubelet motions and applying transformations, such as scaling and rotation, we
introduce motion patterns beyond what is present in the pretraining data. This
allows us to learn a video representation that is remarkably data efficient:
our approach maintains performance when using only 25\% of the pretraining
videos. Experiments on 10 diverse downstream settings demonstrate our
competitive performance and generalizability to new domains and fine-grained
actions.Comment: Accepted in ICCV 202
Tubelet-Contrastive Self-Supervision for Video-Efficient Generalization
We propose a self-supervised method for learning motion-focused video representations. Existing approaches minimize distances between temporally augmented videos, which maintain high spatial similarity. We instead propose to learn similarities between videos with identical local motion dynamics but an otherwise different appearance. We do so by adding synthetic motion trajectories to videos which we refer to as tubelets. By simulating different tubelet motions and applying transformations, such as scaling and rotation, we introduce motion patterns beyond what is present in the pretraining data. This allows us to learn a video representation that is remarkably data efficient: our approach maintains performance when using only 25% of the pretraining videos. Experiments on 10 diverse downstream settings demonstrate our competitive performance and generalizability to new domains and fine-grained actions. Code is available at https://github.com/fmthoker/tubelet-contrast
Video-efficient foundation models
The thesis strives to endow video-efficiency in video understanding by addressing the research question ''What enables video-efficient video foundation models?'' Video-efficiency encompasses developing video foundation models that are not only accurate but also exhibit label-efficiency i.e. require fewer labels, domain-efficiency i.e. applicable to a variety of video learning scenarios, and data-efficiency i.e. reduce the amount of video data needed for learning. The research question is addressed for RGB and non-RGB video modalities. In Chapter 2, we focus on improving the label- and domain-efficiency of non-RGB action recognition and detection. Chapter 3 introduces a new self-supervised approach for learning feature representations for 3D-skeleton video sequences. In Chapter 4, we conduct a large-scale study of existing RGB-based self-supervised video models to assess their performance across different facets of video-efficiency. Chapter 5 presents a new method for video self-supervision that explicitly aims to learn motion focused video-representations. To summarize, this thesis presents several novel approaches to improve the video-efficiency of video foundation models. Our research highlights the importance of transferring knowledge between RGB and non-RGB video modalities, exploring self-supervision for non-RGB video modeling, analyzing self-supervised models beyond canonical setups and carefully designing new self-supervised tasks to develop video foundation models that can exhibit different facets of video-efficiency. We hope that our work will inspire further research and development in this area, leading to even more video-efficient foundation models
COMEDIAN: Self-Supervised Learning and Knowledge Distillation for Action Spotting using Transformers
We present COMEDIAN, a novel pipeline to initialize spatio-temporal
transformers for action spotting, which involves self-supervised learning and
knowledge distillation. Action spotting is a timestamp-level temporal action
detection task. Our pipeline consists of three steps, with two initialization
stages. First, we perform self-supervised initialization of a spatial
transformer using short videos as input. Additionally, we initialize a temporal
transformer that enhances the spatial transformer's outputs with global context
through knowledge distillation from a pre-computed feature bank aligned with
each short video segment. In the final step, we fine-tune the transformers to
the action spotting task. The experiments, conducted on the SoccerNet-v2
dataset, demonstrate state-of-the-art performance and validate the
effectiveness of COMEDIAN's pretraining paradigm. Our results highlight several
advantages of our pretraining pipeline, including improved performance and
faster convergence compared to non-pretrained models.Comment: Source code is available here:
https://github.com/juliendenize/eztorc
Multiscale Video Pretraining for Long-Term Activity Forecasting
Long-term activity forecasting is an especially challenging research problem
because it requires understanding the temporal relationships between observed
actions, as well as the variability and complexity of human activities. Despite
relying on strong supervision via expensive human annotations, state-of-the-art
forecasting approaches often generalize poorly to unseen data. To alleviate
this issue, we propose Multiscale Video Pretraining (MVP), a novel
self-supervised pretraining approach that learns robust representations for
forecasting by learning to predict contextualized representations of future
video clips over multiple timescales. MVP is based on our observation that
actions in videos have a multiscale nature, where atomic actions typically
occur at a short timescale and more complex actions may span longer timescales.
We compare MVP to state-of-the-art self-supervised video learning approaches on
downstream long-term forecasting tasks including long-term action anticipation
and video summary prediction. Our comprehensive experiments across the Ego4D
and Epic-Kitchens-55/100 datasets demonstrate that MVP out-performs
state-of-the-art methods by significant margins. Notably, MVP obtains a
relative performance gain of over 20% accuracy in video summary forecasting
over existing methods
Self-supervised video pretraining yields human-aligned visual representations
Humans learn powerful representations of objects and scenes by observing how
they evolve over time. Yet, outside of specific tasks that require explicit
temporal understanding, static image pretraining remains the dominant paradigm
for learning visual foundation models. We question this mismatch, and ask
whether video pretraining can yield visual representations that bear the
hallmarks of human perception: generalisation across tasks, robustness to
perturbations, and consistency with human judgements. To that end we propose a
novel procedure for curating videos, and develop a contrastive framework which
learns from the complex transformations therein. This simple paradigm for
distilling knowledge from videos, called VITO, yields general representations
that far outperform prior video pretraining methods on image understanding
tasks, and image pretraining methods on video understanding tasks. Moreover,
VITO representations are significantly more robust to natural and synthetic
deformations than image-, video-, and adversarially-trained ones. Finally,
VITO's predictions are strongly aligned with human judgements, surpassing
models that were specifically trained for that purpose. Together, these results
suggest that video pretraining could be a simple way of learning unified,
robust, and human-aligned representations of the visual world.Comment: Technical repor
- …