In order to overcome spatial resolution limitations associated with physical sensor limitations when using smallsats and cubesats, we utilize an image processing technology referred to as Super-Resolution (SR). In general, software approaches are increasingly considered in connection with smaller satellites for which size, mass and power constraints limit the sensor capabilities. Being able to perform hardware vs. software trades might enable more capabilities for a lower cost. This paper describes recent experiments conducted to optimize the spatial enhancement of acquired observations using multiple sub-pixel shifted low resolution image

Dabney, Philip

Le Moigne, Jacqueline

Mareboyana, Manohar

NASA Technical Reports Server

IMPROVING IMAGING INSTRUMENT SPATIAL RESOLUTION USING SOFTWARE  
(SOFTWARE SOLUTIONS FOR IMRPOVING THE SPATIAL RESOLUTION OF IMAGING INSTRUMENTS FOR CUBESATS) 
 
Manohar Mareboyana1,2 , Jacqueline Le Moigne3, Philip Dabney4 
 
1. Bowie State University, Bowie MD 20715 
2. ASRC at NASA Goddard, Greenbelt, MD 20771 
3. Software Engineering Division, NASA Goddard, Greenbelt, MD 20771 
4. Biospheric Sciences Lab, NASA Goddard, Greenbelt, MD 20771 
 
ABSTRACT 
 
In order to overcome spatial resolution limitations 
associated with physical sensor limitations when using 
smallsats and cubesats, we utilize an image processing 
technology referred to as Super-Resolution (SR). In general, 
software approaches are increasingly considered in 
connection with smaller satellites for which size, mass and 
power constraints limit the sensor capabilities. Being able to 
perform hardware vs. software trades might enable more 
capabilities for a lower cost. This paper describes recent 
experiments conducted to optimize the spatial enhancement 
of acquired observations using multiple sub-pixel shifted 
low resolution image. 
Index Terms— super-resolution, radial basis functions 
 
1. INTRODUCTION 
 
With many future missions planning to use CubeSats and 
SmallSats, software approaches are increasingly considered 
to alleviate the size constraints of these platforms that limit 
the sensor capabilities.  
For example, the most common CubeSat sizes are 3U and 
6U, effectively limiting apertures and pupils to 
approximately 9 cm x 9 cm and possibly an ellipsoid of ~ 
9cm x 18 cm. This produces a hard cutoff of spatial 
frequencies above 1 line/ 2.5 meters with a steep roll-off 
leading up to that point. 
Furthermore, most low-power fine-pitch focal planes with 
high frame rates have low fill-factors when micro-lens arrays 
are eliminated to maximize the detector numerical aperture 
(NA) for fast optical systems and utilize the small 
instantaneous field of views (IFOVs) the small detector 
areas create. This low fill-factor produces an instantenously 
under-sampled and aliased image.  SR seeks to recover the 
higher resolution information that produces the alias and 
place the energy back in the appropriate location. It does this 
by intentionally moving the under-sampled alias image in 
sub-pixel pitch increments to capture all of the spatial 
energy delivered to the focal plane. The super resolution 
(SR) techniques also accommodate non-perfect sampling 
pattern inputs and the inverse transform filters effectively 
restore the spatial power up to the spatial cutoff created by 
the finite active pixel IFOV/footprint. 
Being able to perform hardware vs. software trades might 
enable more capabilities for a lower cost. In particular, a 
software approach is being proposed to overcome the 
physical sensor limitations related to spatial resolution by 
utilizing an image processing technology referred to as 
Super-Resolution (SR). SR computationally increases the 
spatial resolution of an image corresponding to a scene, 
either using a single image by retrieving information “lost” 
as part of the digitization and pixel integration process, or 
using a set of observed low resolution (LR) images of the 
same scene that differ by sub-pixel translations. 
This paper describes the experiments that were conducted 
for multi-image SR and the results that were obtained using 
both Landsat and Worldview images demonstrating an 
enhancement resolution factor of 2 or 3. 
 
2. VALIDATING SUPER-RESOLUTION 
ALGORITHMS 
 
In the multi-image SR method, the concept is to leverage the 
non-redundant information contained in sub-pixel shifts low-
resolution images to reconstruct the high-resolution image. 
As each LR image can be considered as a decimated and 
aliased representation of the observed scene, the non-
redundant information offered by multiple LR images 
provides additional high frequency components compared to 
those that exist in a single LR image, and are used to 
compute the high resolution (HR) image. The multiple sub-
pixel shifted images can be rapidly acquired while the 
observing system orbits the Earth; since the motion speed 
and the acquisition rate is known, the sub-pixel shifts can be 
easily calculated. Also, since the rate of acquisition is very 
high, we can reasonably assume that the calculated shift 
accurately represents the registration transformation between 
shifted low-resolution images. Figure 1 illustrates this 
concept. 
https://ntrs.nasa.gov/search.jsp?R=20180004897 2019-08-31T14:45:19+00:00Z
  
Figure 1 – Example of super-resolution algorithm using 9 input 
images, shifted from each other by a sub-pixel amount 
 
To test this type of methods, the absolute “ground truth” 
validation would be to have two datasets of the same area 
taken at exactly the same time by two sensors having the 
same spectral resolution but different spatial resolution. In 
the absence of such ground truth datasets, we will be using 
the simulated framework described below and shown in 
Figure 2 to validate our super-resolution algorithms. Ideally 
test images will be created from very high-resolution (HR) 
images such as Worldview-1 or Worldview-2, although any 
image at a reasonable resolution could be used in that 
framework. In a first step, the original HR image is being 
transformed by a number of sub-pixel shifts to create the HR 
shifted images {HRS1, HRS2, … , HRSn}. Then the Point 
Spread Function (PSF) of the instrument being targeted is 
applied to each of these HRSk images. The next step is then 
to down-sample each of the resulting images by the amount 
of resolution enhancement the super-resolution algorithm is 
being validated for, thus creating the low-resolution (LR) 
images that the SR algorithm will work from or {LRS1, LRS2, 
… , LRSn}. After the Multi-Image SR algorithm has been 
applied, the inverse PSF is applied to the output 
reconstructed HR image, RHR, thus creating the final 
“super-resolved” image, SRI. The SRI image is compared to 
the original HR image using a Mean Square Error (MSE) 
measurement which provides an assessment of the SR 
algorithm that was used for the reconstruction.  
 
Figure 2 – Super-Resolution Algorithms Validation Framework 
 
 
3. SUPER-RESOLUTION FROM MULTIPLE SUB-
PIXEL SHIFT IMAGES 
 
3.1 Comparison of Previously Developed Methods 
Multiple approaches have been previously proposed to 
reconstruct the HR image based on these LR sub-pixel 
images and our preliminary task has been to assess and 
compare these different previous approaches [1]. These are: 
 Frequency Domain Approach [2] with the following 
characteristics: 
o Computationally efficient (using Discrete or 
Continuous Fourier Transform and aliasing 
properties to combine LR images in the SR 
algorithm) 
o Regularization complicated as image degradation 
models become complex 
 Spatial Domain Approaches with the following specific 
methods: 
o Non-Iterative approaches including interpolation 
and restoration: 
 Radial Basis Function (RBF) 
 Inverse Distance Weighted (IDW) 
 Nearest Neighbor (NN) 
o Iterative Back Projection (IBP) [3] 
o Statistical Approaches such as: 
 Maximum A posterior (MAP) 
 Maximum Likelihood (MLE) 
Discrete Wavelet-Based construction (DWT) is another 
approach that was previously considered but because of its 
dyadic constraint, i.e., the resolution enhancements can only 
be power of 2, so it was not considered in the following 
experiments. 
Using the testing framework described in Section 2/Figure 2, 
these various methods were tested using a Landsat image 
that was shifted (with 4 different shifts) and then down-
sampled; the reconstructed HR images (by a resolution 
enhancement factor of 2) were then compared to the original 
image using the Mean Squared Error (MSE) and Peak Signal 
to Noise Ratio(PSNR). Table 1 shows the results of this 
testing for 2 sets of experiments (corresponding to 2 sets of 
4 shifts). 
Table 1 – Comparative MSE Results for Various Multi-Images SR 
Approaches for Experiments 1 and 2  
Method Experiment 1 Experiment 2 
MSE PSNR 
 dB 
MSE PSNR 
 dB 
NN (Nearest Neighbor) 
Interpolation 
3.16 38.81 5.43 40.78 
IDW (inverse Distance 
Weighted) 
3.18 38.78 5.47 40.75 
MLE (Maximum 
Likelihood) 
3.79 38.02 4.7 41.40 
IBP (Iterative Back 
Projection) 
4.14 37.63 6.12 40.26 
RBF (Radial Basis 
Function) Interpolation 
1.28 42.73 1.53 46.28 
EDRBF(Edge Directed 
RBF) 
1.22 42.94 1.53 46.28 
 
In these experiments, NN and RBF are the methods that 
consistently perform better than the other algorithms, with 
RBF performing best.  
3.2 Radial Basis Functions (RBF) and Edge-Directed 
Radial Basis Functions (EDRBF) 
Based on the previous results, our work then focused on the 
RBF technique and an extension of this method exploiting 
the directional information of edges to further improve the 
accuracy of RBF, the Edge-Directed Radial Basis Function 
(EDRBF) interpolation. The accuracy of SR depends on 
various factors besides the algorithm (i) number of sub-pixel 
shifted LR images (ii) accuracy with which the LR shifts are 
estimated by registration algorithms (iii) and the targeted 
spatial resolution of SR.  In our studies, the accuracy of RBF 
and EDRBF will be compared with other algorithms keeping 
these factors constant.  
Figure 3 shows the principle behind the RBF method. 
 
Figure 3 – Radial Basis Functions are used to compute 
interpolated pixels (in green) from the sub-pixel values given by 
the multiple LR images (in red) 
 
RBF are real-valued functions whose value depends on the 
distance from the origin, i.e.:  
                            (2)  
Interpolated pixels Z(x,y) values are determined from the 
shifted low-resolution values LRk(xi,yi) as follows: 
 
where the RBF is the Gaussian function, . (3) 
RBF and EDRBF performance was further analyzed using 
the 2 images shown in Figure 4 below. 
 
Figure 5 shows the 9 shifted LR images created from the 
Landsat image which are used to create the super-resolution 
image. 
 
Landsat 256x256 Image
Spatial Resolution 30m
WorldView-1 512x512 Image
Spatial Resolution 0.46m  
Figure 4 – Test Images Used for RBF-based SR Accuracy 
Assessment 
tx0 = 0.0 ; ty0 = 0.0; 
tx1 = 0.9; ty1 = 0.1; 
tx2 = 1.8; ty2 = 0.2
tx3 = 0.2; ty3 = 1.8; 
tx4 = 0.9; ty4 = 1.0;
tx5 = 1.8; ty5 = 0.8
tx6 = 0.3; ty6 = 2.4; 
tx7 = 1.1; ty7 = 1.8; 
tx8 = 1.9; ty8 = 1.8 
9 sub-pixel shifts were used to 
create 9 LR Images 
 
Figure 5 – Nine 60m resolution LR sub-pixel shifted images 
created from the Landsat 30m image 
 
Figure 6 shows the reconstructed SR image compared to the 
original Landsat image, and for this experiment, the 
computed Mean Square Error (MSE) is about 2.72. 
Similarly, Figure 7 shows the reconstructed HR image from 
a series of 9 sub-pixel shifted LR images created from the 
WorldView image. In this case, a Point Spread Function 
(PSF) based on the Wiener filter is also applied and the 
resulting MSE is of about 8.42. 
Of course, in both cases, if the shifted LR images correspond 
to whole pixel shifts instead of sub-pixel shifts, the 
reconstruction is perfect with an MSE equal to 0.0. 
More recent experiments using RBF, IDW, WT, and IBP 
have been conducted to obtain a resolution enhancement 
factor of 3. More results will be presented at the conference 
including experiments investigating the accuracy of the 
super-resolution as a function of the number of sub-pixel 
shifted images. 
 
Reconstructed SR image 
MSE: 2.7188603337162252
Input HR Landsat Image 256x256
 
Figure 6 – Comparing original and reconstructed images, for a 
SR resolution enhancement factor of 2, using 9 sub-pixel shifted 60 
m LR images created from the Landsat image shown in Figure 4 
 
• Using 9 Sub-Pixel Shifts:
tx1 = 0.9; ty1 = 0.1; tx2 = 1.8; ty2 = 0.2;
tx3 = 0.2; ty3 = 1.8; tx4 = 0.9; ty4 = 1.0; 
tx5 = 1.8; ty5 = 0.8; tx6 = 0.3; ty6 = 2.4; 
tx7 = 1.1; ty7 = 1.8; tx8 = 1.9; ty8 = 1.8 
• Using PSF Wiener Filter
MSE:8.4204123994395967
 
Figure 7 – Comparing original and reconstructed images, for a 
SR resolution enhancement factor of 2, using 9 sub-pixel shifted 
0.92 m LR images created from the WorldView 0.46m image 
shown in Figure 4 
4. CONCLUSION 
 
Experiments using various multi-image super-resolution 
algorithms have been conducted and RBF-based algorithms 
have proven to show the best results, respectively. Various 
assessments have been conducted for obtaining a resolution 
enhancement factor of 2 and have produced minimal 
reconstruction errors.  
Additionally, recent preliminary experiments have also 
shown that an RBF-based algorithm could also provide that 
a resolution enhancement factor of 3 is feasible without 
introducing any significant error. 
Results will be presented at the conference showing 
systematic assessment of both RBF and EDRBF methods 
with various types of test data, various numbers of sub-pixel 
shifted LR images, as well as various amounts of shifts. All 
algorithms will also be analyzed for speed and 
computational requirements. 
5. ACKNOWLEDGMENTS 
 
The authors wish to thank the Earth Science Technology 
Office (ESTO) Instrument Incubator Program (IIP) for 
funding this research. 
 
6. REFERENCES 
 
[1] M. Mareboyana, J. Le Moigne and J.D. Bennett, “High 
Resolution Image Reconstruction from Projection of 
Low Resolution Images Differing in Subpixel Shifts,” 
in Proceedings of 2016 SPIE Defense and Commercial 
Sensing (DCS) Conference, Computational Imaging, 
edited by A. Mahalanobis et al, Baltimore, MD, April 
17-21, 2016. 
[2] J. Yang and T. Huang, “ Image super-resolution: 
Historical overview and future challenges,” Google 
book library 2013. 
[3] M. Irani and S. Peleg, “Improving resolution by Image 
Registration,” Computer Vision Graphics and Image 
Processing, Graphical Models and Image Processing, 
Vol. 53, No.3, May 1991. 
 
 


Improving Imaging Instrument Spatial Resolution Using Software

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server