67 research outputs found
Bayesian Image Reconstruction using Deep Generative Models
Machine learning models are commonly trained end-to-end and in a supervised
setting, using paired (input, output) data. Examples include recent
super-resolution methods that train on pairs of (low-resolution,
high-resolution) images. However, these end-to-end approaches require
re-training every time there is a distribution shift in the inputs (e.g., night
images vs daylight) or relevant latent variables (e.g., camera blur or hand
motion). In this work, we leverage state-of-the-art (SOTA) generative models
(here StyleGAN2) for building powerful image priors, which enable application
of Bayes' theorem for many downstream reconstruction tasks. Our method,
Bayesian Reconstruction through Generative Models (BRGM), uses a single
pre-trained generator model to solve different image restoration tasks, i.e.,
super-resolution and in-painting, by combining it with different forward
corruption models. We keep the weights of the generator model fixed, and
reconstruct the image by estimating the Bayesian maximum a-posteriori (MAP)
estimate over the input latent vector that generated the reconstructed image.
We further use variational inference to approximate the posterior distribution
over the latent vectors, from which we sample multiple solutions. We
demonstrate BRGM on three large and diverse datasets: (i) 60,000 images from
the Flick Faces High Quality dataset (ii) 240,000 chest X-rays from MIMIC III
and (iii) a combined collection of 5 brain MRI datasets with 7,329 scans.
Across all three datasets and without any dataset-specific hyperparameter
tuning, our simple approach yields performance competitive with current
task-specific state-of-the-art methods on super-resolution and in-painting,
while being more generalisable and without requiring any training. Our source
code and pre-trained models are available online:
https://razvanmarinescu.github.io/brgm/.Comment: 27 pages, 17 figures, 5 table
Conditional Image Generation by Conditioning Variational Auto-Encoders
We present a conditional variational auto-encoder (VAE) which, to avoid the
substantial cost of training from scratch, uses an architecture and training
objective capable of leveraging a foundation model in the form of a pretrained
unconditional VAE. To train the conditional VAE, we only need to train an
artifact to perform amortized inference over the unconditional VAE's latent
variables given a conditioning input. We demonstrate our approach on tasks
including image inpainting, for which it outperforms state-of-the-art GAN-based
approaches at faithfully representing the inherent uncertainty. We conclude by
describing a possible application of our inpainting model, in which it is used
to perform Bayesian experimental design for the purpose of guiding a sensor.Comment: 37 pages, 20 figure
A multi-stage GAN for multi-organ chest X-ray image generation and segmentation
Multi-organ segmentation of X-ray images is of fundamental importance for
computer aided diagnosis systems. However, the most advanced semantic
segmentation methods rely on deep learning and require a huge amount of labeled
images, which are rarely available due to both the high cost of human resources
and the time required for labeling. In this paper, we present a novel
multi-stage generation algorithm based on Generative Adversarial Networks
(GANs) that can produce synthetic images along with their semantic labels and
can be used for data augmentation. The main feature of the method is that,
unlike other approaches, generation occurs in several stages, which simplifies
the procedure and allows it to be used on very small datasets. The method has
been evaluated on the segmentation of chest radiographic images, showing
promising results. The multistage approach achieves state-of-the-art and, when
very few images are used to train the GANs, outperforms the corresponding
single-stage approach
- …