2 research outputs found
Model-Theoretic Logic for Mathematical Theory of Semantic Information and Communication
In this paper, we propose an advancement to Tarskian model-theoretic
semantics, leading to a unified quantitative theory of semantic information and
communication. We start with description of inductive logic and probabilities,
which serve as notable tools in development of the proposed theory. Then, we
identify two disparate kinds of uncertainty in semantic communication, that of
physical and content, present refined interpretations of semantic information
measures, and conclude with proposing a new measure for semantic
content-information and entropy. Our proposition standardizes semantic
information across different universes and systems, hence bringing
measurability and comparability into semantic communication. We then proceed
with introducing conditional and mutual semantic cont-information measures and
point out to their utility in formulating practical and optimizable lossless
and lossy semantic compression objectives. Finally, we experimentally
demonstrate the value of our theoretical propositions
CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Loss
This paper considers contrastive training for cross-modal 0-shot transfer
wherein a pre-trained model in one modality is used for representation learning
in another domain using pairwise data. The learnt models in the latter domain
can then be used for a diverse set of tasks in a zero-shot way, similar to
``Contrastive Language-Image Pre-training (CLIP)'' and ``Locked-image Tuning
(LiT)'' that have recently gained considerable attention. Most existing works
for cross-modal representation alignment (including CLIP and LiT) use the
standard contrastive training objective, which employs sets of positive and
negative examples to align similar and repel dissimilar training data samples.
However, similarity amongst training examples has a more continuous nature,
thus calling for a more `non-binary' treatment. To address this, we propose a
novel loss function called Continuously Weighted Contrastive Loss (CWCL) that
employs a continuous measure of similarity. With CWCL, we seek to align the
embedding space of one modality with another. Owing to the continuous nature of
similarity in the proposed loss function, these models outperform existing
methods for 0-shot transfer across multiple models, datasets and modalities.
Particularly, we consider the modality pairs of image-text and speech-text and
our models achieve 5-8% (absolute) improvement over previous state-of-the-art
methods in 0-shot image classification and 20-30% (absolute) improvement in
0-shot speech-to-intent classification and keyword classification.Comment: Accepted to Neural Information Processing Systems (NeurIPS) 2023
conferenc