83 research outputs found
From the Chair: The Top 10 Myths About Imaging Informatics Certification (January 2008–SIIM News)
Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains
Multi-modal foundation models are typically trained on millions of pairs of
natural images and text captions, frequently obtained through web-crawling
approaches. Although such models depict excellent generative capabilities, they
do not typically generalize well to specific domains such as medical images
that have fundamentally shifted distributions compared to natural images.
Building generative models for medical images that faithfully depict clinical
context may help alleviate the paucity of healthcare datasets. Thus, in this
study, we seek to research and expand the representational capabilities of
large pretrained foundation models to medical concepts, specifically for
leveraging the Stable Diffusion model to generate domain specific images found
in medical imaging. We explore the sub-components of the Stable Diffusion
pipeline (the variational autoencoder, the U-Net and the text-encoder) to
fine-tune the model to generate medical images. We benchmark the efficacy of
these efforts using quantitative image quality metrics and qualitative
radiologist-driven evaluations that accurately represent the clinical content
of conditional text prompts. Our best-performing model improves upon the stable
diffusion baseline and can be conditioned to insert a realistic-looking
abnormality on a synthetic radiology image, while maintaining a 95% accuracy on
a classifier trained to detect the abnormality.Comment: 17 pages, 8 figure
Toward expanding the scope of radiology report summarization to multiple anatomies and modalities
Radiology report summarization (RRS) is a growing area of research. Given the
Findings section of a radiology report, the goal is to generate a summary
(called an Impression section) that highlights the key observations and
conclusions of the radiology study. However, RRS currently faces essential
limitations.First, many prior studies conduct experiments on private datasets,
preventing reproduction of results and fair comparisons across different
systems and solutions. Second, most prior approaches are evaluated solely on
chest X-rays. To address these limitations, we propose a dataset (MIMIC-RRS)
involving three new modalities and seven new anatomies based on the MIMIC-III
and MIMIC-CXR datasets. We then conduct extensive experiments to evaluate the
performance of models both within and across modality-anatomy pairs in
MIMIC-RRS. In addition, we evaluate their clinical efficacy via RadGraph, a
factual correctness metric
ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data
Vision-language models (VLMs), such as CLIP and ALIGN, are generally trained
on datasets consisting of image-caption pairs obtained from the web. However,
real-world multimodal datasets, such as healthcare data, are significantly more
complex: each image (e.g. X-ray) is often paired with text (e.g. physician
report) that describes many distinct attributes occurring in fine-grained
regions of the image. We refer to these samples as exhibiting high pairwise
complexity, since each image-text pair can be decomposed into a large number of
region-attribute pairings. The extent to which VLMs can capture fine-grained
relationships between image regions and textual attributes when trained on such
data has not been previously evaluated. The first key contribution of this work
is to demonstrate through systematic evaluations that as the pairwise
complexity of the training dataset increases, standard VLMs struggle to learn
region-attribute relationships, exhibiting performance degradations of up to
37% on retrieval tasks. In order to address this issue, we introduce ViLLA as
our second key contribution. ViLLA, which is trained to capture fine-grained
region-attribute relationships from complex datasets, involves two components:
(a) a lightweight, self-supervised mapping model to decompose image-text
samples into region-attribute pairs, and (b) a contrastive VLM to learn
representations from generated region-attribute pairs. We demonstrate with
experiments across four domains (synthetic, product, medical, and natural
images) that ViLLA outperforms comparable VLMs on fine-grained reasoning tasks,
such as zero-shot object detection (up to 3.6 AP50 points on COCO and 0.6 mAP
points on LVIS) and retrieval (up to 14.2 R-Precision points).Comment: ICCV 202
Exploring Image Augmentations for Siamese Representation Learning with Chest X-Rays
Image augmentations are quintessential for effective visual representation
learning across self-supervised learning techniques. While augmentation
strategies for natural imaging have been studied extensively, medical images
are vastly different from their natural counterparts. Thus, it is unknown
whether common augmentation strategies employed in Siamese representation
learning generalize to medical images and to what extent. To address this
challenge, in this study, we systematically assess the effect of various
augmentations on the quality and robustness of the learned representations. We
train and evaluate Siamese Networks for abnormality detection on chest X-Rays
across three large datasets (MIMIC-CXR, CheXpert and VinDR-CXR). We investigate
the efficacy of the learned representations through experiments involving
linear probing, fine-tuning, zero-shot transfer, and data efficiency. Finally,
we identify a set of augmentations that yield robust representations that
generalize well to both out-of-distribution data and diseases, while
outperforming supervised baselines using just zero-shot transfer and linear
probes by up to 20%. Our code is available at
https://github.com/StanfordMIMI/siaug.Comment: Equal contributions. Oral paper at MIDL 2023. Additional experiments
in appendix in V2. Keywords: Data Augmentations, Self-Supervised Learning,
Medical Imaging, Chest X-rays, Siamese Representation Learnin
Contrastive Learning of Medical Visual Representations from Paired Images and Text
Learning visual representations of medical images is core to medical image
understanding but its progress has been held back by the small size of
hand-labeled datasets. Existing work commonly relies on transferring weights
from ImageNet pretraining, which is suboptimal due to drastically different
image characteristics, or rule-based label extraction from the textual report
data paired with medical images, which is inaccurate and hard to generalize. We
propose an alternative unsupervised strategy to learn medical visual
representations directly from the naturally occurring pairing of images and
textual data. Our method of pretraining medical image encoders with the paired
text data via a bidirectional contrastive objective between the two modalities
is domain-agnostic, and requires no additional expert input. We test our method
by transferring our pretrained weights to 4 medical image classification tasks
and 2 zero-shot retrieval tasks, and show that our method leads to image
representations that considerably outperform strong baselines in most settings.
Notably, in all 4 classification tasks, our method requires only 10% as much
labeled training data as an ImageNet initialized counterpart to achieve better
or comparable performance, demonstrating superior data efficiency
- …