Scene-based nonuniformity correction with video sequences and registration

We describe a new, to our knowledge, scene-based nonuniformity correction algorithm for array detectors. The algorithm relies on the ability to register a sequence of observed frames in the presence of the fixed-pattern noise caused by pixel-to-pixel nonuniformity. In low-to-moderate levels of nonuniformity, sufficiently accurate registration may be possible with standard scene-based registration techniques. If the registration is accurate, and motion exists between the frames, then groups of independent detectors can be identified that observe the same irradiance (or true scene value). These detector outputs are averaged to generate estimates of the true scene values. With these scene estimates, and the corresponding observed values through a given detector, a curve-fitting procedure is used to estimate the individual detector response parameters. These can then be used to correct for detector nonuniformity. The strength of the algorithm lies in its simplicity and low computational complexity. Experimental results, to illustrate the performance of the algorithm, include the use of visible-range imagery with simulated nonuniformity and infrared imagery with real nonuniformity

Multi-Model Kalman Filtering for Adaptive Nonuniformity: Correction in Infrared Sensors

This paper presents an adaptive technique for the estimation of nonuniformity parameters of infrared focal-plane arrays that is robust with respect to changes and uncertainties in scene and sensor characteristics. The proposed algorithm is based on using a bank of Kalman filters in parallel. Each filter independently estimates state variables comprising the gain and the bias matrices of the sensor, according to its own dynamical-model parameters, which underly the statistics of the scene and the nonuniformity as well as the temporal drift in the nonuniformity. The supervising component of the algorithm then generates the final estimates of the state variables by forming a weighted superposition of all the estimates rendered by each Kalman filter. The weights are obtained according to the a posteriori -likelihood principle, applied to the family of models by considering the output residual errors associated with each filter. These weights are updated iteratively between blocks of data, providing the estimator the means to follow the dynamics of the scenes and the sensor. The performance of the proposed estimator and its ability to compensate for fixed-pattern noise are tested using both real and simulated data. The real data is obtained using two cameras operating in the mid- and long-wave infrared regime

Shift Estimation Algorithm for Dynamic Sensors With Frame-to-Frame Variation in Their Spectral Response

This study is motivated by the emergence of a new class of tunable infrared spectral-imaging sensors that offer the ability to dynamically vary the sensor\u27s intrinsic spectral response from frame to frame in an electronically controlled fashion. A manifestation of this is when a sequence of dissimilar spectral responses is periodically realized, whereby in every period of acquired imagery, each frame is associated with a distinct spectral band. Traditional scene-based global shift estimation algorithms are not applicable to such spectrally heterogeneous video sequences, as a pixel value may change from frame to frame as a result of both global motion and varying spectral response. In this paper, a novel algorithm is proposed and examined to fuse a series of coarse global shift estimates between periodically sampled pairs of nonadjacent frames to estimate motion between consecutive frames; each pair corresponds to two nonadjacent frames of the same spectral band. The proposed algorithm outperforms three alternative methods, with the average error being one half of that obtained by using an equal weights version of the proposed algorithm, one-fourth of that obtained by using a simple linear interpolation method, and one-twentieth of that obtained by using a nai¿ve correlation-based direct method

Projection-based image registration in the presence of fixed-pattern noise

A computationally efficient method for image registration is investigated that can achieve an improved performance over the traditional two-dimensional (2-D) cross-correlation-based techniques in the presence of both fixed-pattern and temporal noise. The method relies on transforming each image in the sequence of frames into two vector projections formed by accumulating pixel values along the rows and columns of the image. The vector projections corresponding to successive frames are in turn used to estimate the individual horizontal and vertical components of the shift by means of a one-dimensional (1-D) cross-correlation-based estimator. While gradient-based shift estimation techniques are computationally efficient, they often exhibit degraded performance under noisy conditions in comparison to cross-correlators due to the fact that the gradient operation amplifies noise. The projection-based estimator, on the other hand, significantly reduces the computational complexity associated with the 2-D operations involved in traditional correlation-based shift estimators while improving the performance in the presence of temporal and spatial noise. To show the noise rejection capability of the projection-based shift estimator relative to the 2-D cross correlator, a figure-of-merit is developed and computed reflecting the signal-to-noise ratio (SNR) associated with each estimator. The two methods are also compared by means of computer simulation and tests using real image sequences

Scene-based nonuniformity correction for focal plane arrays by the method of the inverse covariance form

What is to our knowledge a new scene-based algorithm for nonuniformity correction in infrared focal-plane array sensors has been developed. The technique is based on the inverse covariance form of the Kalman filter (KF), which has been reported previously and used in estimating the gain and bias of each detector in the array from scene data. The gain and the bias of each detector in the focal-plane array are assumed constant within a given sequence of frames, corresponding to a certain time and operational conditions, but they are allowed to randomly drift from one sequence to another following a discrete-time Gauss-Markov process. The inverse covariance form filter estimates the gain and the bias of each detector in the focal-plane array and optimally updates them as they drift in time. The estimation is performed with considerably higher computational efficiency than the equivalent KF. The ability of the algorithm in compensating for fixed-pattern noise in infrared imagery and in reducing the computational complexity is demonstrated by use of both simulated and real data

A MAP Estimator for Simultaneous Superresolution and Detector Nonunifomity Correct

During digital video acquisition, imagery may be degraded by a number of phenomena including undersampling, blur, and noise. Many systems, particularly those containing infrared focal plane array (FPA) sensors, are also subject to detector nonuniformity. Nonuniformity, or fixed pattern noise, results from nonuniform responsivity of the photodetectors that make up the FPA. Here we propose a maximuma posteriori (MAP) estimation framework for simultaneously addressing undersampling, linear blur, additive noise, and bias nonuniformity. In particular, we jointly estimate a superresolution (SR) image and detector bias nonuniformity parameters from a sequence of observed frames. This algorithm can be applied to video in a variety of ways including using amoving temporal window of frames to process successive groups of frames. By combining SR and nonuniformity correction (NUC) in this fashion, we demonstrate that superior results are possible compared with the more conventional approach of performing scene-based NUC followed by independent SR. The proposed MAP algorithm can be applied with or without SR, depending on the application and computational resources available. Even without SR, we believe that the proposed algorithm represents a novel and promising scene-based NUC technique. We present a number of experimental results to demonstrate the efficacy of the proposed algorithm. These include simulated imagery for quantitative analysis and real infrared video for qualitative analysis

Simultaneous temperature estimation and nonuniformity correction from multiple frames

Infrared (IR) cameras are widely used for temperature measurements in various applications, including agriculture, medicine, and security. Low-cost IR camera have an immense potential to replace expansive radiometric cameras in these applications, however low-cost microbolometer-based IR cameras are prone to spatially-variant nonuniformity and to drift in temperature measurements, which limits their usability in practical scenarios. To address these limitations, we propose a novel approach for simultaneous temperature estimation and nonuniformity correction from multiple frames captured by low-cost microbolometer-based IR cameras. We leverage the physical image acquisition model of the camera and incorporate it into a deep learning architecture called kernel estimation networks (KPN), which enables us to combine multiple frames despite imperfect registration between them. We also propose a novel offset block that incorporates the ambient temperature into the model and enables us to estimate the offset of the camera, which is a key factor in temperature estimation. Our findings demonstrate that the number of frames has a significant impact on the accuracy of temperature estimation and nonuniformity correction. Moreover, our approach achieves a significant improvement in performance compared to vanilla KPN, thanks to the offset block. The method was tested on real data collected by a low-cost IR camera mounted on a UAV, showing only a small average error of $0.27^\circ C-0.54^\circ C$ relative to costly scientific-grade radiometric cameras. Our method provides an accurate and efficient solution for simultaneous temperature estimation and nonuniformity correction, which has important implications for a wide range of practical applications

IEEE TRANSACTIONS ON IMAGE PROCESSING, 2014 (preprint) 1 Joint Removal of Random and Fixed-Pattern Noise through Spatiotemporal Video Filtering

Abstract—We propose a framework for the denoising of videos jointly corrupted by spatially correlated (i.e. non-white) random noise and spatially correlated fixed-pattern noise. Our approach is based on motion-compensated 3-D spatiotemporal volumes, i.e. a sequence of 2-D square patches extracted along the motion trajectories of the noisy video. First, the spatial and temporal correlations within each volume are leveraged to sparsify the data in 3-D spatiotemporal transform domain, and then the coefficients of the 3-D volume spectrum are shrunk using an adaptive 3-D threshold array. Such array depends on the particular motion trajectory of the volume, the individual power spectral densities of the random and fixed-pattern noise, and also the noise variances which are adaptively estimated in transform domain. Experimental results on both synthetically corrupted data and real infrared videos demonstrate a superior suppression of the random and fixed-pattern noise from both an objective and a subjective point of view. Index Terms—Video denoising, spatiotemporal filtering, fixedpattern noise, power spectral density, adaptive transforms, thermal imaging. I

Generalized Algebraic Algorithm for Scene-based Nonuniformity Correction

ABSTRACT This paper presents an overview of three recently developed scene-based nonuniformity correction techniques, namely, the algebraic scene-based algorithm (ASBA), the extended radiometrically accurate scene-based algorithm (RASBA) and the generalized algebraic scene-based algorithm (GASBA). The ASBA uses pairs of image frames that exhibit one-dimension sub-pixel motion to algebraically extract estimates of bias nonuniformity. The RASBA incorporates arbitrary sub-and super-pixel two-dimensional motion in conjunction with limited perimeter-only absolute calibration to obtain radiometrically accurate estimates of the bias nonuniformity. The RASBA provides the advantage of being able to maintain radiometry in the interior photodetectors without interrupting their operation. The GASBA is a generalized non-radiometric form of the algorithm that uses image pairs with arbitrary two-dimensional motion and encompasses both the ASBA and RASBA algorithms. This generalization is accomplished by initially guaranteeing bias uniformity in the perimeter detectors. This uniformity can be achieved by first applying the ASBA estimates. The generalized algorithm is then able to automatically maintain perimeter uniformity without the need for re-application of the ASBA. Thus, the GASBA is able to operate completely in a non-radiometric mode, alleviating the need for the perimeter calibration system if desired. The generalized algorithm is applied to real infrared imagery obtained from both cooled and uncooled infrared cameras. A hardware implementation of the proposed algorithm will also be discussed along with several ongoing commercial applications of the technology