106 research outputs found
Non-Linguistic Supervision for Contrastive Learning of Sentence Embeddings
Semantic representation learning for sentences is an important and
well-studied problem in NLP. The current trend for this task involves training
a Transformer-based sentence encoder through a contrastive objective with text,
i.e., clustering sentences with semantically similar meanings and scattering
others. In this work, we find the performance of Transformer models as sentence
encoders can be improved by training with multi-modal multi-task losses, using
unpaired examples from another modality (e.g., sentences and unrelated
image/audio data). In particular, besides learning by the contrastive loss on
text, our model clusters examples from a non-linguistic domain (e.g.,
visual/audio) with a similar contrastive loss at the same time. The reliance of
our framework on unpaired non-linguistic data makes it language-agnostic,
enabling it to be widely applicable beyond English NLP. Experiments on 7
semantic textual similarity benchmarks reveal that models trained with the
additional non-linguistic (images/audio) contrastive objective lead to higher
quality sentence embeddings. This indicates that Transformer models are able to
generalize better by doing a similar task (i.e., clustering) with unpaired
examples from different modalities in a multi-task fashion.Comment: Accepted to NeurIPS 202
Bootstrapping Vision-Language Learning with Decoupled Language Pre-training
We present a novel methodology aimed at optimizing the application of frozen
large language models (LLMs) for resource-intensive vision-language (VL)
pre-training. The current paradigm uses visual features as prompts to guide
language models, with a focus on determining the most relevant visual features
for corresponding text. Our approach diverges by concentrating on the language
component, specifically identifying the optimal prompts to align with visual
features. We introduce the Prompt-Transformer (P-Former), a model that predicts
these ideal prompts, which is trained exclusively on linguistic data, bypassing
the need for image-text pairings. This strategy subtly bifurcates the
end-to-end VL training process into an additional, separate stage. Our
experiments reveal that our framework significantly enhances the performance of
a robust image-to-text baseline (BLIP-2), and effectively narrows the
performance gap between models trained with either 4M or 129M image-text pairs.
Importantly, our framework is modality-agnostic and flexible in terms of
architectural design, as validated by its successful application in a video
learning task using varied base modules. The code is available at
https://github.com/yiren-jian/BLITextComment: The code is available at https://github.com/yiren-jian/BLITex
Learning to Collocate Visual-Linguistic Neural Modules for Image Captioning
Humans tend to decompose a sentence into different parts like \textsc{sth do
sth at someplace} and then fill each part with certain content. Inspired by
this, we follow the \textit{principle of modular design} to propose a novel
image captioner: learning to Collocate Visual-Linguistic Neural Modules
(CVLNM). Unlike the \re{widely used} neural module networks in VQA, where the
language (\ie, question) is fully observable, \re{the task of collocating
visual-linguistic modules is more challenging.} This is because the language is
only partially observable, for which we need to dynamically collocate the
modules during the process of image captioning. To sum up, we make the
following technical contributions to design and train our CVLNM: 1)
\textit{distinguishable module design} -- \re{four modules in the encoder}
including one linguistic module for function words and three visual modules for
different content words (\ie, noun, adjective, and verb) and another linguistic
one in the decoder for commonsense reasoning, 2) a self-attention based
\textit{module controller} for robustifying the visual reasoning, 3) a
part-of-speech based \textit{syntax loss} imposed on the module controller for
further regularizing the training of our CVLNM. Extensive experiments on the
MS-COCO dataset show that our CVLNM is more effective, \eg, achieving a new
state-of-the-art 129.5 CIDEr-D, and more robust, \eg, being less likely to
overfit to dataset bias and suffering less when fewer training samples are
available. Codes are available at \url{https://github.com/GCYZSL/CVLMN}Comment: Accepted to IJCV. Codes are available at
https://github.com/GCYZSL/CVLM
Improving Representation Learning for Histopathologic Images with Cluster Constraints
Recent advances in whole-slide image (WSI) scanners and computational
capabilities have significantly propelled the application of artificial
intelligence in histopathology slide analysis. While these strides are
promising, current supervised learning approaches for WSI analysis come with
the challenge of exhaustively labeling high-resolution slides - a process that
is both labor-intensive and time-consuming. In contrast, self-supervised
learning (SSL) pretraining strategies are emerging as a viable alternative,
given that they don't rely on explicit data annotations. These SSL strategies
are quickly bridging the performance disparity with their supervised
counterparts. In this context, we introduce an SSL framework. This framework
aims for transferable representation learning and semantically meaningful
clustering by synergizing invariance loss and clustering loss in WSI analysis.
Notably, our approach outperforms common SSL methods in downstream
classification and clustering tasks, as evidenced by tests on the Camelyon16
and a pancreatic cancer dataset.Comment: Accepted by ICCV202
Knowledge from Large-Scale Protein Contact Prediction Models Can Be Transferred to the Data-Scarce RNA Contact Prediction Task
RNA, whose functionality is largely determined by its structure, plays an
important role in many biological activities. The prediction of pairwise
structural proximity between each nucleotide of an RNA sequence can
characterize the structural information of the RNA. Historically, this problem
has been tackled by machine learning models using expert-engineered features
and trained on scarce labeled datasets. Here, we find that the knowledge
learned by a protein-coevolution Transformer-based deep neural network can be
transferred to the RNA contact prediction task. As protein datasets are orders
of magnitude larger than those for RNA contact prediction, our findings and the
subsequent framework greatly reduce the data scarcity bottleneck. Experiments
confirm that RNA contact prediction through transfer learning using a publicly
available protein model is greatly improved. Our findings indicate that the
learned structural patterns of proteins can be transferred to RNAs, opening up
potential new avenues for research.Comment: Minor revision. The code is available at
https://github.com/yiren-jian/CoT-RNA-Transfe
Multilevel Hierarchical Network with Multiscale Sampling for Video Question Answering
Video question answering (VideoQA) is challenging given its multimodal
combination of visual understanding and natural language processing. While most
existing approaches ignore the visual appearance-motion information at
different temporal scales, it is unknown how to incorporate the multilevel
processing capacity of a deep learning model with such multiscale information.
Targeting these issues, this paper proposes a novel Multilevel Hierarchical
Network (MHN) with multiscale sampling for VideoQA. MHN comprises two modules,
namely Recurrent Multimodal Interaction (RMI) and Parallel Visual Reasoning
(PVR). With a multiscale sampling, RMI iterates the interaction of
appearance-motion information at each scale and the question embeddings to
build the multilevel question-guided visual representations. Thereon, with a
shared transformer encoder, PVR infers the visual cues at each level in
parallel to fit with answering different question types that may rely on the
visual information at relevant levels. Through extensive experiments on three
VideoQA datasets, we demonstrate improved performances than previous
state-of-the-arts and justify the effectiveness of each part of our method
Embedding Heterogeneous Networks into Hyperbolic Space Without Meta-path
Networks found in the real-world are numerous and varied. A common type of
network is the heterogeneous network, where the nodes (and edges) can be of
different types. Accordingly, there have been efforts at learning
representations of these heterogeneous networks in low-dimensional space.
However, most of the existing heterogeneous network embedding methods suffer
from the following two drawbacks: (1) The target space is usually Euclidean.
Conversely, many recent works have shown that complex networks may have
hyperbolic latent anatomy, which is non-Euclidean. (2) These methods usually
rely on meta-paths, which require domain-specific prior knowledge for meta-path
selection. Additionally, different down-streaming tasks on the same network
might require different meta-paths in order to generate task-specific
embeddings. In this paper, we propose a novel self-guided random walk method
that does not require meta-path for embedding heterogeneous networks into
hyperbolic space. We conduct thorough experiments for the tasks of network
reconstruction and link prediction on two public datasets, showing that our
model outperforms a variety of well-known baselines across all tasks.Comment: In proceedings of the 35th AAAI Conference on Artificial Intelligenc
A new micromechanical model of CNT-metal nanocomposites with random clustered distribution of CNTs
Uniform dispersion of carbon nanotubes (CNTs) is a key issue for utilization of their reinforcement potential in CNT-reinforced metal matrix nanocomposites (MMNCs). It was reported that CNT clusters often exist in MMNCs prepared by various techniques, which reduces the load transfer efficiency between the matrix and reinforcement. In this paper, a new micromechanical constitutive model of CNT-reinforced MMNCs is developed, which takes into account of the influences of CNT clusters and misorientations. The strength values of a CNT/Al nanocomposite predicted by the new model are compared first with experimental data for validation. Then, the developed model is applied to predict the size effect, temperature effect and strain rate effect of the nanocomposite in its overall elastoplastic response
- …