Rapid progress has been made in instruction-learning for image editing with
natural-language instruction, as exemplified by InstructPix2Pix. In
biomedicine, such methods can be applied to counterfactual image generation,
which helps differentiate causal structure from spurious correlation and
facilitate robust image interpretation for disease progression modeling.
However, generic image-editing models are ill-suited for the biomedical domain,
and counterfactual biomedical image generation is largely underexplored. In
this paper, we present BiomedJourney, a novel method for counterfactual
biomedical image generation by instruction-learning from multimodal patient
journeys. Given a patient with two biomedical images taken at different time
points, we use GPT-4 to process the corresponding imaging reports and generate
a natural language description of disease progression. The resulting triples
(prior image, progression description, new image) are then used to train a
latent diffusion model for counterfactual biomedical image generation. Given
the relative scarcity of image time series data, we introduce a two-stage
curriculum that first pretrains the denoising network using the much more
abundant single image-report pairs (with dummy prior image), and then continues
training using the counterfactual triples. Experiments using the standard
MIMIC-CXR dataset demonstrate the promise of our method. In a comprehensive
battery of tests on counterfactual medical image generation, BiomedJourney
substantially outperforms prior state-of-the-art methods in instruction image
editing and medical image generation such as InstructPix2Pix and RoentGen. To
facilitate future study in counterfactual medical generation, we plan to
release our instruction-learning code and pretrained models.Comment: Project page & demo: https://aka.ms/biomedjourne