Image restoration problems are typically ill-posed in the sense that each
degraded image can be restored in infinitely many valid ways. To accommodate
this, many works generate a diverse set of outputs by attempting to randomly
sample from the posterior distribution of natural images given the degraded
input. Here we argue that this strategy is commonly of limited practical value
because of the heavy tail of the posterior distribution. Consider for example
inpainting a missing region of the sky in an image. Since there is a high
probability that the missing region contains no object but clouds, any set of
samples from the posterior would be entirely dominated by (practically
identical) completions of sky. However, arguably, presenting users with only
one clear sky completion, along with several alternative solutions such as
airships, birds, and balloons, would better outline the set of possibilities.
In this paper, we initiate the study of meaningfully diverse image restoration.
We explore several post-processing approaches that can be combined with any
diverse image restoration method to yield semantically meaningful diversity.
Moreover, we propose a practical approach for allowing diffusion based image
restoration methods to generate meaningfully diverse outputs, while incurring
only negligent computational overhead. We conduct extensive user studies to
analyze the proposed techniques, and find the strategy of reducing similarity
between outputs to be significantly favorable over posterior sampling. Code and
examples are available in https://noa-cohen.github.io/MeaningfulDiversityInIRComment: Code and examples are available in
https://noa-cohen.github.io/MeaningfulDiversityInI