Image restoration can be implemented efficiently by calculating the convolution of the digital image and a small kernel during image acquisition. Processing the image in the focal-plane in this way requires less computation than traditional Fourier-transform-based techniques such as the Wiener filter and constrained least-squares filter. Here, the values of the convolution kernel that yield the restoration with minimum expected mean-square error are determined using a frequency analysis of the end-to-end imaging system. This development accounts for constraints on the size and shape of the spatial kernel and all the components of the imaging system. Simulation results indicate the technique is effective and efficient

Park, Stephen K.

Reichenbach, Stephen E.

English

NASA Technical Reports Server

Opt iiiial Focal-Plane Rest orat ioii 
Stephen E. Reichenbnch Stephen I<. Park 
C oinpu t er Science Department 
College of William and Ma.ry 
Willianisburg, Virginia 
Abstract 
Image restoration can be implemented efficiently by calculating the convolution of the digital 
image and a small kernel during image acquisition. Processing the image in the focal-plane in this 
way requires less computation than traditional Fourier-transform-based techniques such as the 
Wiener filter and constrained least-squares filter. In this paper, the values of the convolution 
kernel that yield the restoration with minimum expected mean-square error are determined using a 
frequency analysis of the end-teend imaging syst,em. This development accounts for constraints on 
the size and shape of the spatial kernel and all the coniponent,s of the imaging system. Simulation 
results indicate the technique is effective and efficient. 
1 Introduction 
The Wiener filter is probably the best known and most widely used restoration tool. Given a few 
assumptions and some knowledge of the system, the Wiener filter minimizes the expected 
mean-square-error (MSE) of the restoration. While h4SE is by no means a perfect yardstick for 
restoration quality, i t  is a useful ineasure and leads to an optimal filter. In many applications, such as 
those requiring television-rate processing (30 images per second), the most serious drawback of the 
Wiener filter is its high computational cost. Small spatial kernels can be applied with much less 
computation. This paper describes tlie design of small restoration kernels that, within the spatial 
constraints, niinimize restoration MSE. 
2 End-to-End Analysis and Wiener Restoration 
Traditionally, Wiener restoration has been based on a model of the imaging process with two 
components: the linear, shift-invariant point-spread function (PSF) of the image acquisition device 
and additive, signal-independent noise. This model ignores the significant impact of sampling and 
display reconstruction on image quality. A recent paper[l] presented a derivation of the Wiener filter 
that  is based on a more accurate model of the end-to-end imaging process. This model is illustrated 
in Figure 1. 
The end-to-end process is described equivalently by equations in either tlie spatial domain or 
frequency domain. The displayed (or resulting) image r is  
23 
https://ntrs.nasa.gov/search.jsp?R=19900006891 2020-03-20T00:08:36+00:00Z
Acquisition Sampling Digit,a1 
Lattice Sensor 
Noise PSF h m 
Display 
PSF 
Digital 
Res toration 
PSF 
f d 
Figure 1: End-to-End Ima.ging and Spatial Restoration Model 
Assuming the scene s is periodic, the equivalent frequency domain expression for the spectrum of the 
result i is 
?[VI = ( v,g03i[v’]ii[v’]& - v’] + i[v] f [ v ] J [ v ]  ) 
where the notation i [ v ]  indicates the spatial frequency v / N ,  v cycles per N spatial units, of the 
Fourier transform of the image T .  
The Wiener filter minimizes the expected mean-square difference between the scene s and the 
resulting image T :  
If the scene s and noise e are uncorrelated, st.ationa,ry processes with power spectra as and @ e  
respectively, the expected mean-square restoration error can be rewritten in a form that is suitable 
for minimization: 
Minimizing the mean-square error with respect to the filter transfer function values f [ v ]  leads to the 
24 
Reconstruction Blurs for DTF Factor = 0.3, 0.4, 0.5, 0.6 
.oooa .ooo7p ....... 
,0006 **... Image Blur 
t ,0003 
LL 
0 -- 
.30 .40 S O  .60 .70 
CTF FACTOR 
Figure 12: Reconstruction Blur vs CTF FACTOR (Cat) 
(for DTF FACTOR = 0.3 to 0.6) 
Sampling Degradations for DTF FACTOR = 0.3, 0.4, 0.5, 0.6 
,0008 
0.4 
I I I I 0.3 0 
.30 .40 .50 .60 .70 
CTF FACTOR 
Figure 13: Sampling Blur vs CTF FACTOR (Cat) 
(for DTF FACTOR = 0.3 to 0.6) 
Image Blur and SR Degradation for DTF FACTOR = 0.3, 0.4, 0.5, 0.6 
,0008 
.0006 
0 
5 ,0004 
a 
u 
LL 
.O0O2 
0 
.30 .40 .50 .60 .70 
CTF FACTOR 
Figure 14: Image and SR Blur vs CTF FACTOR (Cat) 
(for DTF FACTOR = 0.3 to 0.6) 
21 
Fig 15 : Original Dollar Image with no image 
and SR degradations. 
Fig 16 : Reconstructed Scene with CTF FACTOR=0.3 
and DTF FACTOR=0.3 
Fig 17 : Reconstructed Scene with CTF FACTORd.3 
and DTF FACTOR=0.7 
22 ORlGfNAL PAGE IS 
OF POOR QUALITY 
definition of the optimal filter: 
00 
@,[v’]Iz*[v’]~[v’]iI[v - v’] 
This is the optimal digital filter given in Equation 26 of [l]. 
The mathematics of the following section is siniplified by rewriting the expression for mean-square 
error in Equation 4 as 
s2 = 5 N (E[v] - i[v]f*[v] - i*[v]f[v] + &[VI If[v]l2) 
.=-m 
where 
i[v] = 2 ~,[v’]~~*[v’]j.[v’]ir[v - v’] 
Then, the optimal filter transfer function f given by Equation 5 is written: 
3 Imposing Spatial Constraints 
In the derivation of the previous section, the Wiener filter is determined by an equation in the 
frequency domain: 
&[v]p[v] = i[v] (11) 
The spatial equivalent of this frequency doma.in product is the spatial convolution: 
1 
AT (12) 
- a[n - n’]f[n’] = b [ n ]  
nr 
where 
Y 
b[n] = i)[Y]l;v;;n 
Y 
25 
This convolution is equivalently expressed as a linear system of N equations in N variables (the 
spatial filter values). The system of equations ca.n be expressed in matrix form: 
Af = b 
where the h7 x N coefficient matrix A is 
the N x 1 result matrix b is the array b defined in Equation 14, and f is the N x 1 matrix of digital 
restoration PSF values t o  be deterniincd. 
In the system of equations for the Wiener filter, there are a.s many equations as pixels in the image. 
However, if the size of the spatial restora.tion kcrnel is constrained, the system of independent 
equations caii only be as large as the number of nonzero elements in the spatial kernel. The 
spatially constrained kernel is designed by specifying the sys tem of linear equations whose solution 
will minimize mean-square-error within the constraints. 
The spatial constraint is expressed as a nonempty set of spatial locations, C ,  for which the 
restoration kernel caii be nonzero. The elements that are not in the constraint set must be zero: 
f [ n ]  = 0 if 71 $! C C ( 0 . .  .AT - l} (17) 
If all of the points in the restoration kernel are a.llowed to  be nonzero (i.e., C = ( 0 . .  . N - l}), then 
the optimal spatial kernel is the inverse transform of the Wiener filter (i.e., the solution of 
Equation 12 or 15). 
The expression for MSE is defined in Equation G in terms of the transfer function of the optimal 
filter. Before this expression can be minimized with respect t o  the restoration kernel values, it must 
be expressed in terms of those elements. The filter transfer function expressed in terms of the 
spatially constrained kernel values is 
Substituting this expression into Equation 6 ,  yields the MSE in terms of the constrained kernel values: 
Minimizing with respect to the restoration kernel elements yields 
This is a n  equation with a number of unknowns equal t o  tlie number of non-zero kernel values-ICI. 
There are IC( equations (differentiating with respect t o  each of the constrained kernel elements) in IC( 
unknowns (tlie IC1 kernel values). This system of equations ca.n be written as the matrix equation: 
Acfc = bc ( 2 1 )  
where Ac is the IC1 x IC1 coefficient matrix, fc is the IC1 x 1 matrix of kernel values, and bc is the 
JCI x 1 result matrix. 
Tlie output matrix bc  of Equation 21 for the constrained filter is a submatrix of the corresponding 
matrix b of Equation 15 for tlie Wiener (unconstrained) filter. The elements of the matrix bc are the 
elements of b that  are in the constraint set C. Similarly, Ac is a principal submatrix [2 ]  of tlie 
coefficient matrix A consisting only of the rows and columns of A named in the constraint set C. 
4 Simulation Results 
This section presents restoration results for artificial scenes degraded by simulated imaging devices 
(as described in [3]). Tlie problem design included two variables: the width of the acquisition transfer 
function and the noise level. Three cases for each variable were considered, producing a total of nine 
experimental restoration problems. Each of the nine problems was restored with kernels constrained 
t o  a number of sizes. Then, tlie accuracy of the constrained restorations was compared t o  tlie 
accuracy of the unrestored display and Wiener restoration. 
One-dimensional Fourier scenes were generated by specifying tlie spectral magnitude of a finite 
Fourier series and randomizing phase. Tlie scene spectral magnitude S, was set to 
K e s p  (- (1.1 /CY.,”) if 0 < 1.1 < 2 N  
0 otherwise 
ip[.] = 
with C Y ,  = N/ lG and ps = 0.75. Because the spectral nia.gnitude is zero at tlie origin ( i p [ O ]  = 0), tlie 
resulting ensemble of scenes is zero-mean. The constant K was defined as 0.0704946 so that the 
scenes had unit root-mean-squa.re RhlIS energy. 
The model of the acquisition device transfer functions was suggested by Johnson[4]: 
iLp[.] = esp (- (1.1 / a . h ) P h )  ( 2 3  1 
All three of the transfer functions in this section a.re bell curves ( p h  = 2 ) .  
With ah = 0.75, the transfer function roll off is mostly above the Nyquist limit. This function 
attenuates frequency components within the Nyquist limit only slightly and will therefore cause 
little blurring. However, the transfer function significantly passes components above the 
Nyquist limit and is therefore vulnerable to  aliasing. 
With ah = 0.50, the transfer function rolls off a t  a lower frequency and therefore causes 
somewhat more blurring, but is less vulnerable t o  aliasing. 
27 
0 With ah = 0.25, the transfer function is nearly zero beyond the Nyquist limit. This function 
virtually eliminates aliasing, but the resulting innges may be blurred substantially. 
Three levels of zero-mean white noise were considered. Signal-to-noise ratio (SNR) is tlie ratio of 
RMS energy of tlie scene to  RMS energy of the noise: 
For the low-noise images (high SNR), SNR=100. For the moderate-noise images, SNR=25. For the 
high-noise images (low SNR), SNR=5. 
Real display devices are a significant component of the end-to-end imaging process but are not 
usually a source of much variability. Therefore, the simulated display function was not varied in these 
experiments-a single display model was used for all of the simulations. Scliade[5] suggested a display 
model consisting of the sum of two Gaussian spots-the nucleus, a strongly-peaked central spot that  
contains most of tlie energy, and a broad flare spot around the nucleus. The composite display 
tra.nsfer function is 
The parameters for the functions are taken from Schade’s results: for the nucleus, D1 = 0.76 and 
cy1 = 0.4301454; for the flare, Dz = 0.24 and o2 = 0.0323514. For practical reasons, the display 
transfer function is cut off a t  twice the sampling rate f 2 N  (the same length as the Fourier series used 
to  generate tlie artificial scenes). The effect of the truncation is insignificant. 
Figure 2 illustrates tlie end-to-end imaging simulation for a representative scene. The top graph is 
the scene. Directly below it  is the image created by applying the acquisition function with medium 
blur ( o h  = 0.50) t o  the scene. The third grapli is tlie sampled image. Next is the sampled scene plus 
moderate noise (SNR = 25). The bottom graph of Figure 2 shows the unrestored display. Acquisition 
blurring, aliasing due t o  sampling, additive sensor noise, and display degradation are all present in the 
output of the system. The goal of restoration is t o  process the noisy digital image shown in the fourth 
graph so that  when it  is displayed, the output (the bottom line) is more like the input (the top line). 
The spatial kernels were constrained to  have zero value a t  all but a n  odd number of locations 
centered at the origin-the smallest kernel, with three elements, was allowed non-zero values only 
where In1 5 1; the nest smallest, with five elements, was allowed non-zero values only where In1 5 2; 
and so on. The largest constrained kernel has ( N  - 1) elements; only the element at n = N/2 was 
constrained t o  0. The next-largest optimal kernel (no elements constrained t o  0) is the spatial kernel 
of tlie Wiener filter. 
The optimal three-point and five-point kernels for the example of Figure 2 and the corresponding 
transfer functions are sliowii in Figure 3. The Wiener filter transfer function and part of the 
corresponding spatial kernel are also illustrated. Only the first few elements of the Wiener kernel are 
shown; the magnitude of the Wiener kernel eleiiieiits beyond G pixels from the origin is less than 
0.01N. Clearly, the optimal sinal1 kernels are quite different than the kernels produced by a 
truncating the Wiener F’SF. As can be seen by comparing the transfer functions, the optimal 
three-point kernel does a fair job of approximating the IViener filter at low frequencies but amplifies 
high-frequency components where SNR is lower much more than does the Wiener filter. The transfer 
28 
function of the optimal five-point kernel more closely approximates tlie Wiener filter, but is still quite 
different. 
Figure 4 shows the original scene, the unrestored output, the output with three-point restoration, the 
output with five-point restoration, and the output with Wiener restoration. Visual compa.rison is a 
subjective process, but it is c1ea.r that all of the restorations are more like the original scene than the 
unrestored output. It is more difficult to conclude from visual inspection which of the restorations is 
the best. Some of the features seem to be restored best by the three-point kernel; other features are 
best restored by the Wiener restoration. 
Figure 5 presents numeric measures of restoration accuracy as a function of kernel size. Restoration 
accuracy is described by the RhlS difference between the displayed image aiid the scene, relative to 
the RhlS energy of the scene: 
\I c lWI2 Relative RhlS Error = 
Each of the nine restoration problems was performed 32 times-that is, each execution used a 
different scene from the ensemble and different random noise. The plots show the relative RRIS error 
averaged over all 32 executions. The standard deviations of the relative RhIS error were so small that 
plotting them on these graphs proved impractical. The plots are shown only for kernels with 65 
elements or fewer (radius 32). In all cases, only negligible improvement occurred beyond 19 elements 
(radius 9). (The kernel of the IViener filter has 255 elements, a radius of 128.) 
In many cases, tlie three-point aiid five-point kernels yielded results that are nearly as accurate as the 
Wiener filter. This is particularly true when there is little noise (e.g., SNR=lOO-the leftmost 
column). Small kernels are relatively less successful in low SNR situations (e.g., SNR=5-the 
rightmost column). In low SNR problems, the restoration kernel should suppress noise by local 
averaging, but small kernels are restricted in doing so by their size. 
In the image with medium blur and medium noise (ah = 0.50 and SNR=25)-the middle graph of 
Figure 5-the average unrestored relative Rh4S error was 0.204613. The Wiener filter reduced this to 
0.051149, a decrease of 0.1536464 or 75%. The three-point kernel resulted in an error of 0.091685, a 
decrease of 0.112925 or 55%. The three-point kernel (radius 1) achieved 73% of the improvement of 
the Wiener filter. The five-point kernel (radius 2) reduced the relative RR4S error to  0.053614, a 
decrease of 0.120999 or 59%. This is 79% of tlie improvement of the IViener filter. These small 
kernels achieve a large portion of the iiiiprovement of the Wiener filter with less computation. 
5 Conclusion 
Restoration implemented by convolution with a. small kernel requires less processing than traditional 
Fourier-transform-based techniques such as the Wiener filter. Because convolution with a small kernel 
is a local operation, it is easily applied in parallel on all pixels in the focal-plane during image 
acquisition. The simulation results indicate that the optimal constrained restoration kernel effectively 
restores continuous, one-dimensional functions degraded by blurring, sampling, noise, and 
reconstruction-the types of degradations found in real ima.ging systems. Similar results were 
observed in simulations not presented in this paper using other one-dimensional scenes with different 
statistics. Two-dimensional simulations, and act.ua1 restorations are in preparation. 
29 
References 
[l] Carl L. Fales, Friedrich 0. Huck, Judith A. McCormick, and Stephen I<. Park. Wiener restoration 
of sampled image data: end-to-end analysis. Journal of the Optical Society of America A ,  
5(3):300-314, 1988. 
[2] Roger A. Born and Charles R. Johnson. Matrix Analpis. Cambridge University Press, New York, 
NY, 1985. 
[3] Stephen E. Reichenbach and Stephen I<. Pa.rk. Computer generated scenes and simulated 
imaging. In Technical Digest of the Optical Society of America Annual Meeting, page 170, 1988. 
[4] C. B. Johnson. A method for characterizing electro-optical device modulation transfer functions. 
Photographic Science and Engineering, 14( 6)  :4 13-4 15, 19’70. 
[5]  Otto H. Scliade, Sr. Image reproduction by a line raster process. In Lucien M. Biberman, editor, 
Perception of Displayed Information, chapter 6, pages 233-278, Plenum Press, New York, NY, 
1973. 
30 
0 2  
x g o  
rn .- 
-2 
I I I I I I 
0 50 100 150 200 250 
Figure 2: Simulated End- to-End Processing of a Representative Scene 
1 
a 
I 1.0 
0.0 . . . . . . .  
0 2 4 6 8  0.0 0.3 
3.0 r 5 r  
0- 
0.0 0.3 
- 1  .o 
0 2 4 6 8  
3.0 
.- E :E . . . . . . . .  
0.0 
-1.0- 
0 2 4 6 8  0.0 0.3 
Kernel f [  n]," Filter i' (v) 
Figure 3: Restoration Functions 
31 
a, 
c 
a, 
0 
(II 
D 
Y 
Y 
c 
m 
+ 
c 
0 
I 
.- 
a 
Y 
a, 
1 c 
e 
c 
0 
.- 
a 
I 
a, >
LL 
._  
L 
a, 
C 
a, .~ 
3 
2 
0 
2 
2 
0 
-2 
0 50 100 150 200 250 
Figure 4: Representative Restoration Results 
E .............................. 
... ................................ 
0.0 0.0 E 
0 20 40 60 0 20 40 60 0 20 40 60 
0.5 0 5  
m & 0.4 0.4 
............................... a, .> 
o w  ... ............................. 7J 5 0.1 
2 E 0.0 0.0 
0 20 40 60 G 20 40 60 0 20 40 60 
L 
3 - 
m 
? 
0 
A 
0.5 : 0,4[ 
.... 
._ 
c 
0 0.1 
0.0 
a, 
E 
0 20 40 60 
Kernel  Size 
0.5 
0.2 I::/ 
0.0 
0 20 40 60 
Kernel Size 
................................ 
0.1 
0.0 
0 20 40 60 
Kernel Size 
Low Noise Moderate Noise High Noise 
Figure 5 :  Relative Restoration Error 
32 


Optimal focal-plane restoration

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server