Bennett, Jerome

Le Moigne, Jacqueline

Mareboyana, Manohar

NASA Technical Reports Server

High Resolution Image Reconstruction from 
Projection of Low Resolution Images Differing in 
Sub-pixel Shifts
Manohar Mareboyana1, Jacqueline Le Moigne2, Jerome Bennett3
1
Bowie State University, mmareboyana@bowiestate.edu
2,3 Goddard Space Flight Center, Greenbelt, MD
https://ntrs.nasa.gov/search.jsp?R=20160005762 2019-08-31T03:31:17+00:00Z
What is Super Resolution (SR)?
• A High Resolution (HR) image computed from several observed low-
resolution (LR) images that differ in sub-pixel shifts and rotation is 
called Super Resolution (SR). 
• SR Increases high frequency components and removing the 
degradations caused by the imaging process of the low resolution 
camera. 
Why Super Resolution?
• Higher spatial resolution gives more details
• Spatial Resolutions cannot be increased indefinitely by hardware
• The limitations of spatial resolution come from 
oDiffraction of optical system
o Signal to noise ratio of the sensor system
o Spacecraft orbits in space-borne imaging
Approaches to Computing SR 
• Frequency Domain Approach
• Spatial Domain Approaches
• Non-Iterative approach: Interpolation and Restoration
• Statistical Approaches: 
o Maximum A posterior (MAP) 
o Maximum Likelihood (MLE)
• Wavelet-Based construction
• Set Theoretic  Methods
Spatial Domain Approaches to SR
• Iterative Back Projection (IBP)
• Interpolations
• Nearest Neighbor (NN)
• Inverse Distance Weighted (IDW) Interpolation
• Maximum Likelihood Estimation (MLE)
• Interpolation using Radial Basis Functions (RBF)
Imaging Model
• High Resolution (HR) Image
• Let x be the HR image sought (in 1-D vector form scanned in lexicographical order)
• Low Resolution(LR) Images 
• Let y1, y2, .. yk be LR images that differ in translation and rotation w.r.t image y1 (reference 
image)
• Let Y be concatenation of y1, y2, .. yk
• Each of the LR images (yk) is given by
• yk = DkHkFk x + vk
Fk : motion matrix
Hk : Blurring matrix
Dk: down sampling
Vk : noise term
Imaging Model (2)
• The previous equation can be written as
y= Mx + V
oM = [D1H1F1 D2H2F2 … DkHkFk]
t
o y is concatenation of y1,….yk
oV concatenation of noise components V1.. Vk
oGiven y, to determine x; which is an ill-posed problem.  
oAlso in imaging system the matrices used in the above equations are not 
known.
The Generative Model of LR Images
𝑦𝑖 = 𝑇𝑖𝐵𝐷(𝑥)+ni, 
T is sub-pixel translation, B is blurring, and D down sample operators
ni is noise 
B D +n1
B D +n2
B D +nk
T2
Tk
X
y1
y2
yk
Spatial Domain Approach to SR
Sub-pixel Registration 
• Uses Discrete Wavelet Transforms (DWT)
• Decomposes the Images into LR image and High Frequency Components in 
Horizontal and Vertical Directions (LL,LH,HL,HH) in multi-resolution fashion 
into a number of levels thus building a hierarchy low resolution images
• The decomposition is done for both (reference and input) images that are 
to be registered.
• At each level of decomposition features are extracted from low-frequency 
and high-frequency subbands for reference and input images and 
transformation function is computed using correlations.
• The transformation function is improved iteratively using images from the 
hierarchy.
Interpolation onto HR Grid
• Interpolation algorithms used to project the LR images onto HR
• Nearest Neighbor (NN) Interpolation
• Inverse Distance Weighted (IDW) Interpolation
• Radial Basis Functions (RDF) Interpolation
Deblurring and Noise filter
• Simulation of LR images from Original Image includes a know Point 
Spread Function (PSF) and a noise component as shown in generative 
model LR images
• The reconstructed image deblurring is obtained by deconvolving using 
Weiner filter
• A low pass filter is used on the reconstructed SR image to filter out 
high frequency noise present in LR images
Statistical Methods in Computing SR image
• Maximum Likelihood Estimation (MLE) 
• In this method the SR image is estimated from a given set of LR images 
differing in sub-pixel shifts
• p(x|y1, y2. . yk) =
𝑝 𝑦1, 𝑦2, . . 𝑦𝑘 𝑥 𝑝(𝑥)
 𝑥 p y1, y2. . yk x 𝑝(𝑥)
• x is SR image, y1, y2, .. yk are LR images
• Log Likelihood function L(x)  = log(𝑝 𝑦1, 𝑦2, . . 𝑦𝑘 𝑥 𝑝(𝑥))
• MLE assumes flat a prior  p(x)
• 𝑥′ =
𝑎𝑟𝑔𝑚𝑎𝑥 𝐿(𝑥)
𝑥
• Every pixel on HR grid is the one that maximizes the L(x) 
Statistical Methods in Computing SR image (2) 
• Maximum A Posterior (MAP) method 
• p(x|y1, y2. . yk) =
𝑝 𝑦1, 𝑦2, . . 𝑦𝑘 𝑥 𝑝(𝑥)
 𝑥 p y1, y2. . yk x 𝑝(𝑥)
• 𝑥′ = 𝑎𝑟𝑔𝑚𝑎𝑥 p y1, y2, . . yk 𝑥 𝑝(𝑥)
𝑥
• This method allows inducing priors in estimating x’
• The results of MLE and MAP are similar
Iterative Back Projection (IBP)
• Initial HR (x’)image is reconstructed from the reference LR image (y1) 
by replicating pixels
• From x’ the LR images are generated using the regenerative model 
(y’1, y’2, .. Y’k) 
• 𝑥′ = 𝑥′ +  
𝑖=2
𝑘
𝑦𝑖 − 𝑦
′
𝑖
Projection of LR images on HR Grid
• Assume the HR grid is 2n*2n where LR images are n*n
• Number of LR images differing in sub-pixel shift are 22
• HR grid kn*kn, requires at k2 LR images for optimal reconstruction
• All LR images are assumed to differ in subpixel shifts that are different 
from each other.
• The subpixel shifts are global
Projection of LR images on HR Grid: NN Reconstruction
𝑥′ = 𝑈2 𝑦1
𝑈2 is upsample operator by 2
𝑥′ = 𝑈2 𝑦1 + 𝑘=2
4 𝑈2(𝑇𝑘(𝑦𝑘)
Tk Translation operator
Projection LR images on HR Grid using IDW
• Unlike in NN, all the LR images contribute to the estimation of every 
missing pixels of HR grid 
• Pixel value depends on distance of the LR image pixels from the 
missing HR grid point by inverse distance
• 𝑥′ = 𝑈2 𝑦1 +  𝑘=2
4 𝑈2(𝑤𝑘 𝑦𝑘)
• wk is inverse distance of yk from missing positions of x’
•  𝑘=2
4 𝑤𝑘 = 1
Interpolation using Radial Basis Functions (RBF)
• A radial basis function (RBF) is a real valued function whose value 
depends on the distance from a given point xi (x a variable not to be 
confused with image variable used for HR image). 
• ∅(𝑥, 𝑥𝑖) = ∅(| 𝑥 − 𝑥𝑖||
• When the distance function is Gaussian, the RBF is called Gaussian 
RBF
• ∅(𝑥, 𝑥𝑖) = 𝑒
− 𝑥−𝑥𝑖
2
Interpolation using Radial Basis Functions (RBF)
• The reference LR image(nxn) is used to fill up every other HR grid 
(2nx2n) pixels along the row and column. 
• The remaining LR images are used to estimate the 3 missing cells of 
2x2 HR grid as follows
• 𝑥 𝑖 =  𝑘=2
4 ∅ 𝑖 − 𝑇(𝑘) ∗ 𝑦𝑘
• 𝑥 𝑖 = 
𝑗=2
𝑘
𝑒− | 𝑖−𝑇 𝑗 | 𝑦𝑗
• In the above equation vector i is the position of the missing pixel from 
HR grid (0,1 for horizontal, 1,0 for vertical and 1,1 for diagonal)
• T(k) is the translation of LR image, yk with respect to  i. 
Experimental Results
• Simulation of LR images from Chesapeake Bay Landsat Image
Results 
IBP MLE  IDW NN RBF 
Results (SSE)
y2 :  tr2 = 0.9, tc2 = 0.24 
y3:  tr3 = 0.12, tc3 = 0.82
y4:  tr4 = 0.9,  tc4 = 0.55
Algorithm SSE
NN(Nearest Neighbor)  
interpolation
87799
IDW (inverse Distance Weighted) 128624
MLE (Maximum Likelihood) 167781
IBP (Iterative Back Projection) 220468
RBF (Radial Basis Function) 
Interpolation
77732
Results (SSE) 
y2:  tr2 = 082, tc2 = 0.08 
y3:  tr3 = 0.17, tc3 = 0.84
y4:  tr4 = 0..64,  tc4 = 0.56
Algorithm SSE
NN(Nearest Neighbor)  
interpolation
886717
IDW (inverse Distance 
Weighted)
120953
MLE (Maximum Likelihood) 167938
IBP (Iterative Back Projection) 226565
RBF (Radial Basis Function) 
Interpolation
71946
Concluding Remarks
• Various algorithms are implemented to compare their performance in 
reconstructing HR image from a set of LR images differing subpixel 
shifts
• Interpolation using RBFs performed the best in most cases.  
• HR reconstruction accuracy depends on the subpixel shifts and 
number of LR images
• We have experimented with four LR images to increase the spatial 
resolution by a factor of 2
• Further research is required to improve the spatial resolution by an 
arbitrary number with a given number of LR images


High Resolution Image Reconstruction from Projection of Low Resolution Images Differing in Sub-pixel Shifts

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server