Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models

Pretrained vision-language models, such as CLIP, have demonstrated strong generalization capabilities, making them promising tools in the realm of zero-shot visual recognition. Visual relation detection (VRD) is a typical task that identifies relationship (or interaction) types between object pairs within an image. However, naively utilizing CLIP with prevalent class-based prompts for zero-shot VRD has several weaknesses, e.g., it struggles to distinguish between different fine-grained relation types and it neglects essential spatial information of two objects. To this end, we propose a novel method for zero-shot VRD: RECODE, which solves RElation detection via COmposite DEscription prompts. Specifically, RECODE first decomposes each predicate category into subject, object, and spatial components. Then, it leverages large language models (LLMs) to generate description-based prompts (or visual cues) for each component. Different visual cues enhance the discriminability of similar relation categories from different perspectives, which significantly boosts performance in VRD. To dynamically fuse different cues, we further introduce a chain-of-thought method that prompts LLMs to generate reasonable weights for different visual cues. Extensive experiments on four VRD benchmarks have demonstrated the effectiveness and interpretability of RECODE

Empower Distantly Supervised Relation Extraction with Collaborative Adversarial Training

With recent advances in distantly supervised (DS) relation extraction (RE), considerable attention is attracted to leverage multi-instance learning (MIL) to distill high-quality supervision from the noisy DS. Here, we go beyond label noise and identify the key bottleneck of DS-MIL to be its low data utilization: as high-quality supervision being refined by MIL, MIL abandons a large amount of training instances, which leads to a low data utilization and hinders model training from having abundant supervision. In this paper, we propose collaborative adversarial training to improve the data utilization, which coordinates virtual adversarial training (VAT) and adversarial training (AT) at different levels. Specifically, since VAT is label-free, we employ the instance-level VAT to recycle instances abandoned by MIL. Besides, we deploy AT at the bag-level to unleash the full potential of the high-quality supervision got by MIL. Our proposed method brings consistent improvements (~ 5 absolute AUC score) to the previous state of the art, which verifies the importance of the data utilization issue and the effectiveness of our method.Comment: Accepted by AAAI 202

Number of unigenes related to intestinal digestive enzymes.

<p>x-axis indicates the corresponding number of unigenes and y-axis indicates the specific unigenes related to intestinal digestive enzymes. Three hundred and ninety-four protease-related unigenes were identified in <i>M</i>. <i>alternatus</i> Hope transcriptome. <b>(I) Number of specific unigenes related to serine proteases in intestinal digestive enzymes</b>. The icon indicates specific unigenes. The number in brackets indicates the corresponding number of unigenes.</p

Unigenes length distribution.

<p>The y-axis number has been converted into logarithmic scale.</p

Sequence statistics of the Illumina sequencing assembly.

<p>Sequence statistics of the Illumina sequencing assembly.</p

Number of unigenes related to pesticide receptors and resistance-related genes.

<p>x-axis indicates the number of unigenes, y-axis indicates the specific unigenes related to pesticide receptors and resistance-related genes. Seven hundred and fifty-nine insecticide receptors and resistance-related unigenes were identified in <i>M</i>. <i>alternatus</i> transcriptome.</p

Number of unigenes related to RNAi.

<p>x-axis indicates the corresponding number of unigenes and y-axis indicates the specific unigenes related to RNAi. Forty-two unigenes coding for RNAi identified in the database, 26 of them were larger than 600 bp (61.90%) and 23 were more than 1 kb (54.76%).</p

Characteristics of the homology search of Illumina sequences against the NR database.

<p>(A) E-value distribution of the BLAST hits for each unique sequence with a cut-off E-value of 1.0E^-5. (B) Species distribution of the BLASTX results. The first hit of each sequence was used for further <i>in silico</i> analysis.</p