8 research outputs found
Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA
Videos convey rich information. Dynamic spatio-temporal relationships between
people/objects, and diverse multimodal events are present in a video clip.
Hence, it is important to develop automated models that can accurately extract
such information from videos. Answering questions on videos is one of the tasks
which can evaluate such AI abilities. In this paper, we propose a video
question answering model which effectively integrates multi-modal input sources
and finds the temporally relevant information to answer questions.
Specifically, we first employ dense image captions to help identify objects and
their detailed salient regions and actions, and hence give the model useful
extra information (in explicit textual format to allow easier matching) for
answering questions. Moreover, our model is also comprised of dual-level
attention (word/object and frame level), multi-head self/cross-integration for
different sources (video and dense captions), and gates which pass more
relevant information to the classifier. Finally, we also cast the frame
selection problem as a multi-label classification task and introduce two loss
functions, In-andOut Frame Score Margin (IOFSM) and Balanced Binary
Cross-Entropy (BBCE), to better supervise the model with human importance
annotations. We evaluate our model on the challenging TVQA dataset, where each
of our model components provides significant gains, and our overall model
outperforms the state-of-the-art by a large margin (74.09% versus 70.52%). We
also present several word, object, and frame level visualization studies. Our
code is publicly available at:
https://github.com/hyounghk/VideoQADenseCapFrameGate-ACL2020Comment: ACL 2020 (11 pages
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention
We present Perceiver-VL, a vision-and-language framework that efficiently
handles high-dimensional multimodal inputs such as long videos and text.
Powered by the iterative latent cross-attention of Perceiver, our framework
scales with linear complexity, in contrast to the quadratic complexity of
self-attention used in many state-of-the-art transformer-based models. To
further improve the efficiency of our framework, we also study applying
LayerDrop on cross-attention layers and introduce a mixed-stream architecture
for cross-modal retrieval. We evaluate Perceiver-VL on diverse video-text and
image-text benchmarks, where Perceiver-VL achieves the lowest GFLOPs and
latency while maintaining competitive performance. In addition, we also provide
comprehensive analyses of various aspects of our framework, including
pretraining data, scalability of latent size and input size, dropping
cross-attention layers at inference to reduce latency, modality aggregation
strategy, positional encoding, and weight initialization strategy. Our code and
checkpoints are available at: https://github.com/zinengtang/Perceiver_VLComment: WACV 2023 (first two authors contributed equally
CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation
We present CoDi-2, a versatile and interactive Multimodal Large Language
Model (MLLM) that can follow complex multimodal interleaved instructions,
conduct in-context learning (ICL), reason, chat, edit, etc., in an any-to-any
input-output modality paradigm. By aligning modalities with language for both
encoding and generation, CoDi-2 empowers Large Language Models (LLMs) to not
only understand complex modality-interleaved instructions and in-context
examples, but also autoregressively generate grounded and coherent multimodal
outputs in the continuous feature space. To train CoDi-2, we build a
large-scale generation dataset encompassing in-context multimodal instructions
across text, vision, and audio. CoDi-2 demonstrates a wide range of zero-shot
capabilities for multimodal generation, such as in-context learning, reasoning,
and compositionality of any-to-any modality generation through multi-round
interactive conversation. CoDi-2 surpasses previous domain-specific models on
tasks such as subject-driven image generation, vision transformation, and audio
editing. CoDi-2 signifies a substantial breakthrough in developing a
comprehensive multimodal foundation model adept at interpreting in-context
language-vision-audio interleaved instructions and producing multimodal
outputs.Comment: Project Page: https://codi-2.github.io
Paxion: Patching Action Knowledge in Video-Language Foundation Models
Action knowledge involves the understanding of textual, visual, and temporal
aspects of actions. We introduce the Action Dynamics Benchmark (ActionBench)
containing two carefully designed probing tasks: Action Antonym and Video
Reversal, which targets multimodal alignment capabilities and temporal
understanding skills of the model, respectively. Despite recent video-language
models' (VidLM) impressive performance on various benchmark tasks, our
diagnostic tasks reveal their surprising deficiency (near-random performance)
in action knowledge, suggesting that current models rely on object recognition
abilities as a shortcut for action understanding. To remedy this, we propose a
novel framework, Paxion, along with a new Discriminative Video Dynamics
Modeling (DVDM) objective. The Paxion framework utilizes a Knowledge Patcher
network to encode new action knowledge and a Knowledge Fuser component to
integrate the Patcher into frozen VidLMs without compromising their existing
capabilities. Due to limitations of the widely-used Video-Text Contrastive
(VTC) loss for learning action knowledge, we introduce the DVDM objective to
train the Knowledge Patcher. DVDM forces the model to encode the correlation
between the action text and the correct ordering of video frames. Our extensive
analyses show that Paxion and DVDM together effectively fill the gap in action
knowledge understanding (~50% to 80%), while maintaining or improving
performance on a wide spectrum of both object- and action-centric downstream
tasks. The code and data will be made publicly available for research purposes
at https://github.com/MikeWangWZHL/Paxion.git.Comment: NeurIPS 2023 spotligh
Unifying Vision, Text, and Layout for Universal Document Processing
We propose Universal Document Processing (UDOP), a foundation Document AI
model which unifies text, image, and layout modalities together with varied
task formats, including document understanding and generation. UDOP leverages
the spatial correlation between textual content and document image to model
image, text, and layout modalities with one uniform representation. With a
novel Vision-Text-Layout Transformer, UDOP unifies pretraining and multi-domain
downstream tasks into a prompt-based sequence generation scheme. UDOP is
pretrained on both large-scale unlabeled document corpora using innovative
self-supervised objectives and diverse labeled data. UDOP also learns to
generate document images from text and layout modalities via masked image
reconstruction. To the best of our knowledge, this is the first time in the
field of document AI that one model simultaneously achieves high-quality neural
document editing and content customization. Our method sets the
state-of-the-art on 8 Document AI tasks, e.g., document understanding and QA,
across diverse data domains like finance reports, academic papers, and
websites. UDOP ranks first on the leaderboard of the Document Understanding
Benchmark.Comment: CVPR 202
Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA
Videos convey rich information. Dynamic spatio-temporal relationships between people/objects, and diverse multimodal events are present in a video clip. Hence, it is important to develop automated models that can accurately extract such information from videos. Answering questions on videos is one of the tasks which can evaluate such AI abilities. In this paper, we propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions. Specifically, we first employ dense image captions to help identify objects and their detailed salient regions and actions, and hence give the model useful extra information (in explicit textual format to allow easier matching) for answering questions. Moreover, our model is also comprised of dual-level attention (word/object and frame level), multi-head self/cross-integration for different sources (video and dense captions), and gates which pass more relevant information to the classifier. Finally, we also cast the frame selection problem as a multi-label classification task and introduce two loss functions, In-andOut Frame Score Margin (IOFSM) and Balanced Binary Cross-Entropy (BBCE), to better supervise the model with human importance annotations. We evaluate our model on the challenging TVQA dataset, where each of our model components provides significant gains, and our overall model outperforms the state-of-the-art by a large margin (74.09% versus 70.52%). We also present several word, object, and frame level visualization studies
Continuous Language Generative Flow
Recent years have witnessed various types of generative models for natural language generation (NLG), especially RNNs or transformer based sequence-to-sequence models, as well as variational autoencoder (VAE) and generative adversarial network (GAN) based models. However, flow-based generative models, which achieve strong performance in image generation due to their invertibility and exact density estimation properties, have been less explored for NLG. In this paper, we propose a flow-based language generation model by adapting previous flow generative models to language generation via continuous input embeddings, adapted affine coupling structures, and a novel architecture for autoregressive text generation. We also apply our framework to Sequence-to-Sequence generation, including text- and video-based Question Generation (QG) and Neural Machine Translation (NMT), and data augmentation for Question Answering (QA). We use our language flow model to provide extra input features for QG and NMT, which achieves improvements over the strong QG baselines on SQuAD and TVQA and NMT baseline on WMT16. We also augment QA data with new context by injecting noise to the latent features of the language flow and show this augmentation leads to a large performance improvement from strong baselines on SQuAD and TVQA