Rethinking the Pipeline of Demosaicing, Denoising and Super-Resolution

Abstract

Incomplete color sampling, noise degradation, and limited resolution are the three key problems that are unavoidable in modern camera systems. Demosaicing (DM), denoising (DN), and super-resolution (SR) are core components in a digital image processing pipeline to overcome the three problems above, respectively. Although each of these problems has been studied actively, the mixture problem of DM, DN, and SR, which is a higher practical value, lacks enough attention. Such a mixture problem is usually solved by a sequential solution (applying each method independently in a fixed order: DM $\to$ DN $\to$ SR), or is simply tackled by an end-to-end network without enough analysis into interactions among tasks, resulting in an undesired performance drop in the final image quality. In this paper, we rethink the mixture problem from a holistic perspective and propose a new image processing pipeline: DN $\to$ SR $\to$ DM. Extensive experiments show that simply modifying the usual sequential solution by leveraging our proposed pipeline could enhance the image quality by a large margin. We further adopt the proposed pipeline into an end-to-end network, and present Trinity Enhancement Network (TENet). Quantitative and qualitative experiments demonstrate the superiority of our TENet to the state-of-the-art. Besides, we notice the literature lacks a full color sampled dataset. To this end, we contribute a new high-quality full color sampled real-world dataset, namely PixelShift200. Our experiments show the benefit of the proposed PixelShift200 dataset for raw image processing.Comment: Code is available at: https://github.com/guochengqian/TENe