11,041 research outputs found
Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision
The goal of this work is to train discriminative cross-modal embeddings
without access to manually annotated data. Recent advances in self-supervised
learning have shown that effective representations can be learnt from natural
cross-modal synchrony. We build on earlier work to train embeddings that are
more discriminative for uni-modal downstream tasks. To this end, we propose a
novel training strategy that not only optimises metrics across modalities, but
also enforces intra-class feature separation within each of the modalities. The
effectiveness of the method is demonstrated on two downstream tasks: lip
reading using the features trained on audio-visual synchronisation, and speaker
recognition using the features trained for cross-modal biometric matching. The
proposed method outperforms state-of-the-art self-supervised baselines by a
signficant margin.Comment: Under submission as a conference pape
MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images
Multi-modal fusion approaches aim to integrate information from different
data sources. Unlike natural datasets, such as in audio-visual applications,
where samples consist of "paired" modalities, data in healthcare is often
collected asynchronously. Hence, requiring the presence of all modalities for a
given sample is not realistic for clinical tasks and significantly limits the
size of the dataset during training. In this paper, we propose MedFuse, a
conceptually simple yet promising LSTM-based fusion module that can accommodate
uni-modal as well as multi-modal input. We evaluate the fusion method and
introduce new benchmark results for in-hospital mortality prediction and
phenotype classification, using clinical time-series data in the MIMIC-IV
dataset and corresponding chest X-ray images in MIMIC-CXR. Compared to more
complex multi-modal fusion strategies, MedFuse provides a performance
improvement by a large margin on the fully paired test set. It also remains
robust across the partially paired test set containing samples with missing
chest X-ray images. We release our code for reproducibility and to enable the
evaluation of competing models in the future
VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment
Vision-language pre-training (VLP) has recently proven highly effective for
various uni- and multi-modal downstream applications. However, most existing
end-to-end VLP methods use high-resolution image-text box data to perform well
on fine-grained region-level tasks, such as object detection, segmentation, and
referring expression comprehension. Unfortunately, such high-resolution images
with accurate bounding box annotations are expensive to collect and use for
supervision at scale. In this work, we propose VoLTA (Vision-Language
Transformer with weakly-supervised local-feature Alignment), a new VLP paradigm
that only utilizes image-caption data but achieves fine-grained region-level
image understanding, eliminating the use of expensive box annotations. VoLTA
adopts graph optimal transport-based weakly-supervised alignment on local image
patches and text tokens to germinate an explicit, self-normalized, and
interpretable low-level matching criterion. In addition, VoLTA pushes
multi-modal fusion deep into the uni-modal backbones during pre-training and
removes fusion-specific transformer layers, further reducing memory
requirements. Extensive experiments on a wide range of vision- and
vision-language downstream tasks demonstrate the effectiveness of VoLTA on
fine-grained applications without compromising the coarse-grained downstream
performance, often outperforming methods using significantly more caption and
box annotations
- …