Recent advances in neural image compression (NIC) have produced models that
are starting to outperform classic codecs. While this has led to growing
excitement about using NIC in real-world applications, the successful adoption
of any machine learning system in the wild requires it to generalize (and be
robust) to unseen distribution shifts at deployment. Unfortunately, current
research lacks comprehensive datasets and informative tools to evaluate and
understand NIC performance in real-world settings. To bridge this crucial gap,
first, this paper presents a comprehensive benchmark suite to evaluate the
out-of-distribution (OOD) performance of image compression methods.
Specifically, we provide CLIC-C and Kodak-C by introducing 15 corruptions to
the popular CLIC and Kodak benchmarks. Next, we propose spectrally-inspired
inspection tools to gain deeper insight into errors introduced by image
compression methods as well as their OOD performance. We then carry out a
detailed performance comparison of several classic codecs and NIC variants,
revealing intriguing findings that challenge our current understanding of the
strengths and limitations of NIC. Finally, we corroborate our empirical
findings with theoretical analysis, providing an in-depth view of the OOD
performance of NIC and its dependence on the spectral properties of the data.
Our benchmarks, spectral inspection tools, and findings provide a crucial
bridge to the real-world adoption of NIC. We hope that our work will propel
future efforts in designing robust and generalizable NIC methods. Code and data
will be made available at https://github.com/klieberman/ood_nic.Comment: NeurIPS 202