8 research outputs found
AttnGrounder: Talking to Cars with Attention
We propose Attention Grounder (AttnGrounder), a single-stage end-to-end
trainable model for the task of visual grounding. Visual grounding aims to
localize a specific object in an image based on a given natural language text
query. Unlike previous methods that use the same text representation for every
image region, we use a visual-text attention module that relates each word in
the given query with every region in the corresponding image for constructing a
region dependent text representation. Furthermore, for improving the
localization ability of our model, we use our visual-text attention module to
generate an attention mask around the referred object. The attention mask is
trained as an auxiliary task using a rectangular mask generated with the
provided ground-truth coordinates. We evaluate AttnGrounder on the Talk2Car
dataset and show an improvement of 3.26% over the existing methods
Ref-NMS: Breaking Proposal Bottlenecks in Two-Stage Referring Expression Grounding
The prevailing framework for solving referring expression grounding is based
on a two-stage process: 1) detecting proposals with an object detector and 2)
grounding the referent to one of the proposals. Existing two-stage solutions
mostly focus on the grounding step, which aims to align the expressions with
the proposals. In this paper, we argue that these methods overlook an obvious
mismatch between the roles of proposals in the two stages: they generate
proposals solely based on the detection confidence (i.e., expression-agnostic),
hoping that the proposals contain all right instances in the expression (i.e.,
expression-aware). Due to this mismatch, current two-stage methods suffer from
a severe performance drop between detected and ground-truth proposals. To this
end, we propose Ref-NMS, which is the first method to yield expression-aware
proposals at the first stage. Ref-NMS regards all nouns in the expression as
critical objects, and introduces a lightweight module to predict a score for
aligning each box with a critical object. These scores can guide the NMS
operation to filter out the boxes irrelevant to the expression, increasing the
recall of critical objects, resulting in a significantly improved grounding
performance. Since Ref- NMS is agnostic to the grounding step, it can be easily
integrated into any state-of-the-art two-stage method. Extensive ablation
studies on several backbones, benchmarks, and tasks consistently demonstrate
the superiority of Ref-NMS. Codes are available at:
https://github.com/ChopinSharp/ref-nms.Comment: Appear in AAAI 2021, Codes are available at:
https://github.com/ChopinSharp/ref-nm
Knowledge-guided Pairwise Reconstruction Network for Weakly Supervised Referring Expression Grounding
Weakly supervised referring expression grounding (REG) aims at localizing the
referential entity in an image according to linguistic query, where the mapping
between the image region (proposal) and the query is unknown in the training
stage. In referring expressions, people usually describe a target entity in
terms of its relationship with other contextual entities as well as visual
attributes. However, previous weakly supervised REG methods rarely pay
attention to the relationship between the entities. In this paper, we propose a
knowledge-guided pairwise reconstruction network (KPRN), which models the
relationship between the target entity (subject) and contextual entity (object)
as well as grounds these two entities. Specifically, we first design a
knowledge extraction module to guide the proposal selection of subject and
object. The prior knowledge is obtained in a specific form of semantic
similarities between each proposal and the subject/object. Second, guided by
such knowledge, we design the subject and object attention module to construct
the subject-object proposal pairs. The subject attention excludes the unrelated
proposals from the candidate proposals. The object attention selects the most
suitable proposal as the contextual proposal. Third, we introduce a pairwise
attention and an adaptive weighting scheme to learn the correspondence between
these proposal pairs and the query. Finally, a pairwise reconstruction module
is used to measure the grounding for weakly supervised learning. Extensive
experiments on four large-scale datasets show our method outperforms existing
state-of-the-art methods by a large margin.Comment: Accepted by ACMMM 2019. arXiv admin note: text overlap with
arXiv:1908.1056