Joint registration and super-resolution with omnidirectional images

This paper addresses the reconstruction of high resolution omnidirectional images from multiple low resolution images with inexact registration. When omnidirectional images from low resolution vision sensors can be uniquely mapped on the 2-sphere, such a reconstruction can be described as a transform domain super-resolution problem in the spherical imaging framework. We describe how several spherical images with arbitrary rotations in the SO(3) rotation group contribute to the reconstruction of a high resolution image with help of the Spherical Fourier Transform (SFT). As low resolution images might not be perfectly registered in practice, the impact of inaccurate alignment on the transform coefficients is further analyzed. We then cast the joint registration and super-resolution problem as a total least squares norm minimization problem in the SFT domain. A l1- regularized total least squares problem is also considered. The regularized problem is solved efficiently by interior point methods. Experiments with synthetic and natural images show that the proposed scheme leads to effective reconstruction of high resolution images even when large registration errors exist in the low resolution images. The quality of the reconstructed images also increases rapidly with the number of low resolution images, which demonstrates the benefits of the proposed solution in super-resolution schemes. Finally, we highlight the benefit of the additional regularization constraint that clearly leads to reduced noise and improved reconstruction quality

OmniZoomer: Learning to Move and Zoom in on Sphere at High-Resolution

Omnidirectional images (ODIs) have become increasingly popular, as their large field-of-view (FoV) can offer viewers the chance to freely choose the view directions in immersive environments such as virtual reality. The M\"obius transformation is typically employed to further provide the opportunity for movement and zoom on ODIs, but applying it to the image level often results in blurry effect and aliasing problem. In this paper, we propose a novel deep learning-based approach, called \textbf{OmniZoomer}, to incorporate the M\"obius transformation into the network for movement and zoom on ODIs. By learning various transformed feature maps under different conditions, the network is enhanced to handle the increasing edge curvatures, which alleviates the blurry effect. Moreover, to address the aliasing problem, we propose two key components. Firstly, to compensate for the lack of pixels for describing curves, we enhance the feature maps in the high-resolution (HR) space and calculate the transformed index map with a spatial index generation module. Secondly, considering that ODIs are inherently represented in the spherical space, we propose a spherical resampling module that combines the index map and HR feature maps to transform the feature maps for better spherical correlation. The transformed feature maps are decoded to output a zoomed ODI. Experiments show that our method can produce HR and high-quality ODIs with the flexibility to move and zoom in to the object of interest. Project page is available at http://vlislab22.github.io/OmniZoomer/.Comment: Accepted by ICCV 202

Super-resolution of 3-dimensional scenes

Super-resolution is an image enhancement method that increases the resolution of images and video. Previously this technique could only be applied to 2D scenes. The super-resolution algorithm developed in this thesis creates high-resolution views of 3-dimensional scenes, using low-resolution images captured from varying, unknown positions

Super-resolution from unregistered omnidirectional images

This paper addresses the problem of super- resolution from low resolution spherical images that are not perfectly registered. Such a problem is typically en- countered in omnidirectional vision scenarios with re- duced resolution sensors in imperfect settings. Several spherical images with arbitrary rotations in the SO(3) rotation group are used for the reconstruction of higher resolution images. We first describe the impact of the registration error on the Spherical Fourier Transform coefficients. Then, we formulate the joint registration and reconstruction problem as a least squares norm minimization problem in the transform domain. Exper- imental results show that the proposed scheme leads to effective approximations of the high resolution images, even with large registration errors. The quality of the reconstructed images also increases rapidly with the number of low resolution images, which demonstrates the benefits of the proposed solution in super-resolution schemes