1 research outputs found
Toward Bridging the Simulated-to-Real Gap: Benchmarking Super-Resolution on Real Data
Capturing ground truth data to benchmark super-resolution (SR) is
challenging. Therefore, current quantitative studies are mainly evaluated on
simulated data artificially sampled from ground truth images. We argue that
such evaluations overestimate the actual performance of SR methods compared to
their behavior on real images. Toward bridging this simulated-to-real gap, we
introduce the Super-Resolution Erlangen (SupER) database, the first
comprehensive laboratory SR database of all-real acquisitions with pixel-wise
ground truth. It consists of more than 80k images of 14 scenes combining
different facets: CMOS sensor noise, real sampling at four resolution levels,
nine scene motion types, two photometric conditions, and lossy video coding at
five levels. As such, the database exceeds existing benchmarks by an order of
magnitude in quality and quantity. This paper also benchmarks 19 popular
single-image and multi-frame algorithms on our data. The benchmark comprises a
quantitative study by exploiting ground truth data and qualitative evaluations
in a large-scale observer study. We also rigorously investigate agreements
between both evaluations from a statistical perspective. One interesting result
is that top-performing methods on simulated data may be surpassed by others on
real data. Our insights can spur further algorithm development, and the publicy
available dataset can foster future evaluations.Comment: To appear in IEEE Transactions on Pattern Analysis and Machine
Intelligence; data and source code available at
https://superresolution.tf.fau.de