1,238 research outputs found
A-CAP: Anticipation Captioning with Commonsense Knowledge
Humans possess the capacity to reason about the future based on a sparse
collection of visual cues acquired over time. In order to emulate this ability,
we introduce a novel task called Anticipation Captioning, which generates a
caption for an unseen oracle image using a sparsely temporally-ordered set of
images. To tackle this new task, we propose a model called A-CAP, which
incorporates commonsense knowledge into a pre-trained vision-language model,
allowing it to anticipate the caption. Through both qualitative and
quantitative evaluations on a customized visual storytelling dataset, A-CAP
outperforms other image captioning methods and establishes a strong baseline
for anticipation captioning. We also address the challenges inherent in this
task.Comment: Accepted to CVPR 202
Exploring Transferability of Multimodal Adversarial Samples for Vision-Language Pre-training Models with Contrastive Learning
Vision-language pre-training models (VLP) are vulnerable, especially to
multimodal adversarial samples, which can be crafted by adding imperceptible
perturbations on both original images and texts. However, under the black-box
setting, there have been no works to explore the transferability of multimodal
adversarial attacks against the VLP models. In this work, we take CLIP as the
surrogate model and propose a gradient-based multimodal attack method to
generate transferable adversarial samples against the VLP models. By applying
the gradient to optimize the adversarial images and adversarial texts
simultaneously, our method can better search for and attack the vulnerable
images and text information pairs. To improve the transferability of the
attack, we utilize contrastive learning including image-text contrastive
learning and intra-modal contrastive learning to have a more generalized
understanding of the underlying data distribution and mitigate the overfitting
of the surrogate model so that the generated multimodal adversarial samples
have a higher transferability for VLP models. Extensive experiments validate
the effectiveness of the proposed method
Braid: Weaving Symbolic and Neural Knowledge into Coherent Logical Explanations
Traditional symbolic reasoning engines, while attractive for their precision
and explicability, have a few major drawbacks: the use of brittle inference
procedures that rely on exact matching (unification) of logical terms, an
inability to deal with uncertainty, and the need for a precompiled rule-base of
knowledge (the "knowledge acquisition" problem). To address these issues, we
devise a novel logical reasoner called Braid, that supports probabilistic
rules, and uses the notion of custom unification functions and dynamic rule
generation to overcome the brittle matching and knowledge-gap problem prevalent
in traditional reasoners. In this paper, we describe the reasoning algorithms
used in Braid, and their implementation in a distributed task-based framework
that builds proof/explanation graphs for an input query. We use a simple QA
example from a children's story to motivate Braid's design and explain how the
various components work together to produce a coherent logical explanation.
Finally, we evaluate Braid on the ROC Story Cloze test and achieve close to
state-of-the-art results while providing frame-based explanations.Comment: Accepted at AAAI-202
- …