68 research outputs found
Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect
Despite being impactful on a variety of problems and applications, the
generative adversarial nets (GANs) are remarkably difficult to train. This
issue is formally analyzed by \cite{arjovsky2017towards}, who also propose an
alternative direction to avoid the caveats in the minmax two-player training of
GANs. The corresponding algorithm, called Wasserstein GAN (WGAN), hinges on the
1-Lipschitz continuity of the discriminator. In this paper, we propose a novel
approach to enforcing the Lipschitz continuity in the training procedure of
WGANs. Our approach seamlessly connects WGAN with one of the recent
semi-supervised learning methods. As a result, it gives rise to not only better
photo-realistic samples than the previous methods but also state-of-the-art
semi-supervised learning results. In particular, our approach gives rise to the
inception score of more than 5.0 with only 1,000 CIFAR-10 images and is the
first that exceeds the accuracy of 90% on the CIFAR-10 dataset using only 4,000
labeled images, to the best of our knowledge.Comment: Accepted as a conference paper in International Conference on
Learning Representation(ICLR). Xiang Wei and Boqing Gong contributed equally
in this wor
Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect
Despite being impactful on a variety of problems and applications, the
generative adversarial nets (GANs) are remarkably difficult to train. This
issue is formally analyzed by \cite{arjovsky2017towards}, who also propose an
alternative direction to avoid the caveats in the minmax two-player training of
GANs. The corresponding algorithm, called Wasserstein GAN (WGAN), hinges on the
1-Lipschitz continuity of the discriminator. In this paper, we propose a novel
approach to enforcing the Lipschitz continuity in the training procedure of
WGANs. Our approach seamlessly connects WGAN with one of the recent
semi-supervised learning methods. As a result, it gives rise to not only better
photo-realistic samples than the previous methods but also state-of-the-art
semi-supervised learning results. In particular, our approach gives rise to the
inception score of more than 5.0 with only 1,000 CIFAR-10 images and is the
first that exceeds the accuracy of 90% on the CIFAR-10 dataset using only 4,000
labeled images, to the best of our knowledge.Comment: Accepted as a conference paper in International Conference on
Learning Representation(ICLR). Xiang Wei and Boqing Gong contributed equally
in this wor
Video Timeline Modeling For News Story Understanding
In this paper, we present a novel problem, namely video timeline modeling.
Our objective is to create a video-associated timeline from a set of videos
related to a specific topic, thereby facilitating the content and structure
understanding of the story being told. This problem has significant potential
in various real-world applications, such as news story summarization. To
bootstrap research in this area, we curate a realistic benchmark dataset,
YouTube-News-Timeline, consisting of over k timelines and k YouTube
news videos. Additionally, we propose a set of quantitative metrics as the
protocol to comprehensively evaluate and compare methodologies. With such a
testbed, we further develop and benchmark exploratory deep learning approaches
to tackle this problem. We anticipate that this exploratory work will pave the
way for further research in video timeline modeling. The assets are available
via
https://github.com/google-research/google-research/tree/master/video_timeline_modeling.Comment: Accepted as a spotlight by NeurIPS 2023, Track on Datasets and
Benchmark
Unified Visual Relationship Detection with Vision and Language Models
This work focuses on training a single visual relationship detector
predicting over the union of label spaces from multiple datasets. Merging
labels spanning different datasets could be challenging due to inconsistent
taxonomies. The issue is exacerbated in visual relationship detection when
second-order visual semantics are introduced between pairs of objects. To
address this challenge, we propose UniVRD, a novel bottom-up method for Unified
Visual Relationship Detection by leveraging vision and language models (VLMs).
VLMs provide well-aligned image and text embeddings, where similar
relationships are optimized to be close to each other for semantic unification.
Our bottom-up design enables the model to enjoy the benefit of training with
both object detection and visual relationship datasets. Empirical results on
both human-object interaction detection and scene-graph generation demonstrate
the competitive performance of our model. UniVRD achieves 38.07 mAP on
HICO-DET, outperforming the current best bottom-up HOI detector by 14.26 mAP.
More importantly, we show that our unified detector performs as well as
dataset-specific models in mAP, and achieves further improvements when we scale
up the model. Our code will be made publicly available on GitHub.Comment: Accepted to ICCV 2023. Code is available at
https://github.com/google-research/scenic/tree/main/scenic/projects/univr
Spatiotemporally Discriminative Video-Language Pre-Training with Text Grounding
Most of existing video-language pre-training methods focus on instance-level
alignment between video clips and captions via global contrastive learning but
neglect rich fine-grained local information, which is of importance to
downstream tasks requiring temporal localization and semantic reasoning. In
this work, we propose a simple yet effective video-language pre-training
framework, namely G-ViLM, to learn discriminative spatiotemporal features. Two
novel designs involving spatiotemporal grounding and temporal grouping promote
learning local region-noun alignment and temporal-aware features
simultaneously. Specifically, spatiotemporal grounding aggregates semantically
similar video tokens and aligns them with noun phrases extracted from the
caption to promote local region-noun correspondences. Moreover, temporal
grouping leverages cut-and-paste to manually create temporal scene changes and
then learns distinguishable features from different scenes. Comprehensive
evaluations demonstrate that G-ViLM performs favorably against existing
approaches on four representative downstream tasks, covering text-video
retrieval, video question answering, video action recognition and temporal
action localization. G-ViLM performs competitively on all evaluated tasks and
in particular achieves R@10 of 65.1 on zero-shot MSR-VTT retrieval, over 9%
higher than the state-of-the-art method
- …