This paper describes a new method for mitigating the effects of atmospheric distortion on observed images, particularly airborne turbulence which degrades a region of interest (ROI). In order to provide accurate detail from objects behind the dis-torting layer, a simple and efficient frame selection method is proposed to pick informative ROIs from only good-quality frames. We solve the space-variant distortion problem using region-based fusion based on the Dual Tree Complex Wavelet Transform (DT-CWT). We also propose an object alignment method for pre-processing the ROI since this can exhibit sig-nificant offsets and distortions between frames. Simple haze removal is used as the final step. The proposed method per-forms very well with atmospherically distorted videos and outperforms other existing methods. Index Terms — Image restoration, fusion, DT-CWT 1

Achim, Alin M

Anantrasirichai, N

Bull, David

Kingsbury, Nick

English

N. Anantrasirichai

Alin Achim

David Bull

Nick Kingsbury

CiteSeerX

Mitigating the effects of atmospheric distortion using dt-cwt fusion

This paper describes a new method for mitigating the effects of atmospheric distortion on observed images, particularly airborne turbulence which degrades a region of interest (ROI). In order to provide accurate detail from objects behind the distorting layer, a simple and efficient frame selection method is proposed to pick informative ROIs from only good-quality frames. We solve the space-variant distortion problem using region-based fusion based on the Dual Tree Complex Wavelet Transform (DT-CWT). We also propose an object alignment method for pre-processing the ROI since this can exhibit significant offsets and distortions between frames. Simple haze removal is used as the final step. The proposed method performs very well with atmospherically distorted videos and outperforms other existing methods

Explore Bristol Research

                          Anantrasirichai, N., Achim, A. M., Bull, D., & Kingsbury, N. (2012).Mitigating the effects of atmospheric distortion using DT-CWT fusion. InIEEE International Conference on Image Processing. (pp. 3033).10.1109/ICIP.2012.6467539Link to published version (if available):10.1109/ICIP.2012.6467539Link to publication record in Explore Bristol ResearchPDF-documentUniversity of Bristol - Explore Bristol ResearchGeneral rightsThis document is made available in accordance with publisher policies. Please cite only the publishedversion using the reference above. Full terms of use are available:http://www.bristol.ac.uk/pure/about/ebr-terms.htmlTake down policyExplore Bristol Research is a digital archive and the intention is that deposited content should not beremoved. However, if you believe that this version of the work breaches copyright law please contactopen-access@bristol.ac.uk and include the following information in your message:• Your contact details• Bibliographic details for the item, including a URL• An outline of the nature of the complaintOn receipt of your message the Open Access Team will immediately investigate your claim, make aninitial judgement of the validity of the claim and, where appropriate, withdraw the item in questionfrom public view.MITIGATING THE EFFECTS OF ATMOSPHERIC DISTORTION USING DT-CWT FUSIONN. Anantrasirichai, Alin Achim, David BullVisual Information LaboratoryUniversity of Bristol, UKNick KingsburySignal Processing LaboratoryUniversity of Cambridge, UKABSTRACTThis paper describes a new method for mitigating the effectsof atmospheric distortion on observed images, particularlyairborne turbulence which degrades a region of interest (ROI).In order to provide accurate detail from objects behind the dis-torting layer, a simple and efficient frame selection method isproposed to pick informative ROIs from only good-qualityframes. We solve the space-variant distortion problem usingregion-based fusion based on the Dual Tree Complex WaveletTransform (DT-CWT). We also propose an object alignmentmethod for pre-processing the ROI since this can exhibit sig-nificant offsets and distortions between frames. Simple hazeremoval is used as the final step. The proposed method per-forms very well with atmospherically distorted videos andoutperforms other existing methods.Index Terms— Image restoration, fusion, DT-CWT1. INTRODUCTIONVarious types of atmospheric distortion can influence the vi-sual quality of a captured video signal. Distortion types in-clude fog or haze which reduces contrast, and atmosphericturbulence due to temperature variations or airborne contam-inants. Such obscurations in the lower atmosphere reduce theability to see objects at a distance. A variation in temperaturecauses different interference patterns in the light refraction,leading to unclear, unsharp, waving images of the objects.Examples of this effect are found in areas such as hot roadsand deserts, as well in the proximity of aircraft jet exhausts.Examples are shown in columns 1 and 2 of Fig. 3.Since turbulence in the captured images makes it diffi-cult to interpret information behind the distorting layer, therehas been significant research activity trying to faithfully re-construct this useful information using various methods. Theperfect solution however seems impossible since this problemis irreversible, although it can be simply written as Eq. 1.Iobv = DIidl + ε (1)where Iobv and Iidl are the observed and ideal images respec-tively. D represents geometric distortion and blur, while εrepresents noise. Various approaches have attempted to solvethis problem by using blind deconvolution (BD) [1, 2]. Theresults, however, still exhibit artefacts since the point spreadfunction (PSF) is assumed to be space-invariant.It is obvious that using a single image is not sufficientto remove the visible ripples and waves, while utilising a setof images to construct one enhanced image makes more use-ful information available. There are two types of restorationprocess that use multiple images. The first employs imageregistration with deformation estimation to align objects tem-porally and to solve for small movements due to atmosphericrefraction [3, 4]. Then a deblurring process is applied to thecombined image (which is a challenging task as this blur isspace-variant). The other group employs image selection andfusion, known as ‘lucky region’ techniques [5]. The regionsof the input frames having the best quality in the temporal di-rection are selected and then are combined in an intelligentway. Recently this method has been improved by applyingimage alignment to those lucky regions [6].In this paper, we propose a new fusion method to reduceimage distortion caused by air turbulence. We employ aregion-based scheme to perform fusion at the feature level.This has advantages over pixel-based processing as more in-telligent semantic fusion rules can be considered based onactual features in the image. The fusion is performed in theDual Tree Complex Wavelet Transform (DT-CWT) domainas it provides near shift-invariance and directional selectiv-ity [7]. Additionally, the phase of a CWT coefficient is robustto noise and temporal intensity variations thereby providingan efficient tool for removing the distorting ripples.Before applying fusion, a set of selected images or ROIsmust be aligned. We introduce an object alignment approachfor distorted images. As randomly distorted images do notprovide identical features, we cannot use conventional meth-ods to find matching features. We, instead, employ a morpho-logical image processing technique. Subsequently we selectthe ROI (or whole image) from only the informative framesmeasured by a novel quality matric, based on sharpness, in-tensity similarity and ROI size. Then a non-rigid image regis-tration is applied. After the fusion, haze and fog are removedusing a locally adaptive histogram equalisation.The remaining part of this paper is organised as follows.The proposed scheme for mitigating the atmospheric distor-tion is described in details in Section 2. The performance ofthe method is evaluated on a set of images and is comparedwith other techniques in Section 3. Finally, Section 4 presentsthe conclusions of the study.2. PROPOSED MITIGATION SCHEMEThe proposed process is depicted in Fig. 1. Details of eachstep are described below.Object Alignment Shooting video in the far distance maycause the ROI in each frame to become misaligned. The inter-frame distance between the shaking objects may be too largeto apply image registration. Unfortunately matching algo-rithms which use feature detection are not suitable for the airturbulence problem as strong gradients within each frame aredistorted randomly. Here a simple approach using morpho-logical image processing is proposed. The ROI is markedin the first frame. The histogram, generated from the se-lected ROI and the surrounding area, is utilised to find an Otsuthreshold [8] which is used to convert the image to a binarymap. An erosion process is then applied and the areas con-nected to the edge of image are removed. This step is doneiteratvely until the area near the ROI is isolated. The sameOtsu threshold and the number of iterations are employed inother frames. The centre position of each mask is computed.If there is more than one isolated area, the area closest in sizeand location to the ROI in the first frame is used. Finally thecentre of the mask in each frame is utilised to shift the ROIto align across the set of frames. Note that the frames withthe incorrectly detected ROIs will be removed in the frameselection process. These frames are generally significantlydifferent from others.Frame Selection In our proposed method, not every framein the sequence is used to restore the undistorted image sincethe bad images (e.g. the very blurred ones) would possiblydeteriorate the fused result. A set of images are carfully se-lected using three factors; sharpness Gn, intensity similaritySn and detected ROI size An. Gn can be computed from asummation of intensity gradients or the magnitude of high-pass coefficients. For Sn, the average frame of the whole se-quence is used as a reference for calculating the mean squareerror (MSE). Then MSE−1 represents the similarity of eachframe. An is the total number of pixels contained in the ROI.This is used because larger ROIs contain more details. Thecost function Cn for frame n is computed using Eq. 2.Cn =wGGnλG + |Gn| +wSSnλS + |Sn| +wAAnλA + |An| (2)where wk and λk are the weight and slope control of the fac-tor k ∈ {G,S,A}, respectively. The sigmoid function is usedhere to prevent one factor dominating the others, e.g. a block-ing artefact may cause significantly high values of sharpness,yet this frame should not be included in the selected data set.Object AlignmentFrame SelectionImage RegistrationImage Fusion Haze RemovalDistorted VideoRestoredImageFig. 1. Block diagram of the proposed methodThe λk is set to equal the mean of factor k so that at the meanvalue, the cost value of such factor is 0.5. The cost Cn isranked from high to low. The Otsu method can be applied tofind how many frames should be included in the selected set.Image Registration Registration of non-rigid bodies usingthe phase-shift properties of the DT-CWT proposed in [9] isemployed in this paper. The algorithm is developed from theideas of phase-based multidimensional volume registration,which is robust to noise and temporal intensity variations.Motion estimation is performed iteratively, firstly by usingcoarser level complex coefficients to determine large motioncomponents and then by employing finer level coefficients torefine the motion field.Image Fusion We have adapted the region-based image fu-sion technique using complex wavelets proposed in [10] toaddress the air-turbulence problem. The method first trans-forms each image into the DT-CWT domain. Then it employsan adapted version of the combined morphological spectralunsupervised image segmentation and a multiscale watershedsegmentation [11] to divide the image into R regions. Thelowpass DT-CWT coefficients of the fused image are simplyconstructed from the average of the lowpass values of all im-ages, while the highpass coeffients are selected according toan activity measurement indicating the importance of that re-gion. In this paper, to produce sharper results, we operate oneach sub-band separately. The priority P of region rθn ∈ R inimage n is computed with the detail coefficients dθ,ln (x, y) oflevel l and sub-band θ as shown in Eq. 3. The fused image isconsequently constructed based on this priority map.P (rθn) =1|rθn|∑∀l,(x,y)∈rθn∣∣dθ,ln (x, y)∣∣ (3)The air-turbulence scenario is different from other image-fusion problems as the segmentation boundaries which sep-arate inhomogeneous regions vary significantly from frameto frame (due to turbulence distortion). The aim is to pro-vide the sharpest and most temporally consistent boundaries.Therefore, at the boundary of each region, we use the maxi-mum of DT-CWT coefficient magnitudes from all frames in-stead of using P (rθn). To reduce the distortion due to rippling,the phase of the complex wavelet coefficients play an impor-tant role since it corresponds to the type of dominant direc-tional features in its support regions. Hence, the DT-CWTFig. 2. Original image (left) and its contrast enhancement withglobal HE (middle) and CLAHE (right)coefficients, dθ,l(x, y), of the fused image are adjusted witha unit vector representing the average phase from all frames,N , used in the fusion process (Eq. 4). The average phasecan be used because, in turbulent motion, pixels deviate fromtheir actual positions with approximately zero mean and witha quasi-periodic motion [4].d˜θ,l(x, y) =∑Nn dθ,ln (x, y)∣∣∣∑Nn dθ,ln (x, y)∣∣∣∣∣dθ,l(x, y)∣∣ (4)To reduce noise, we apply a shrinkage function, As, de-rived as Maximum A Posteriori (MAP) estimators as in [12].Then, if sharpening is required, a gain Ag > 1 can be appliedto boost highpass-coefficient magnitudes. However this sim-ple technique might cause remaining noise to become clearlyvisible again. Therefore we recommend to enhance only thehigh-pass coefficients when they form regions of large mag-nitude. We create the binary mapMθ,l for each subband. TheMθ,l = 1 if∣∣dθ,l∣∣ > τ , where τ is a predefined threshold.The isolated pixels are subsequently removed fromMθ,l. Themodified highpass coeffients are finally rewritten as Eq. 5.dθ,l = Aθ,lg Mθ,lAθ,ls d˜θ,l (5)Haze Removal A problem that often accompanies turbu-lence is haze or fog. Simple and fast methods such as his-togram equalisation (HE) can be used to reduce this effect.However, in this paper, we consider the ROI and the mean-ing it carries. In this case, a contrast limited adaptive his-togram equalisation (CLAHE) is more suitable than globaladjustment. The method calculates local histograms derivedfrom a neighbourhood region. A threshold is predefined tolimit the cumulative values thereby preventing overamplify-ing small amounts of noise in largely homogeneous regions.An example of haze removal with HE and CLAHE is shownin Fig. 2. The number at the top of the car is much clearer inthe CLAHE result.3. RESULTS AND DISCUSSIONSDatasets Four sequences captured from long distance exhibit-ing significant turbulence distortions are used in our tests.Two datasets, Hill House and Hot Road, are greyscale only,while the other two, Cold Car and Number Plate, are coloursequences. The colour sequences are converted into greyscaleand colour channels, e.g. YCbCr or HSV. Only the greyscalechannel is used in our method. The results are subsequentlycombined with colour channels and converted back to theoriginal colour space. Only theNumber Plate sequence showsa significant shift of the ROI area between frames, so we haveartificially applied spatial shifts of between 1-20 pixels ran-domly to Hill House, Hot Road and Cold Car.Parameter Setting The DT-CWT is applied with 3 or 4 de-composition levels when the image resolution is smaller orlarger than 128 × 128 pixels, respectively. The weights forthe frame selection wG, wS and wA are set to 1. The valueτ for cleaning the map Mθ,l is individually set for each sub-band in each level to τθ,l = |d|θ,l − σθ,l, where |d|θ,l andσθ,l are a mean and a standard deviation of the magnitude ofhighpass subband. The gains Aθ,lg are equal for each subbandat the same level and are 2.5, 1.8, 1.2 and 1 for l = 1, 2, 3 and4, respectively. For CLAHE, a window of 8×8 pixels is usedto compute local histograms which are clipped at 1%.Experiment and Discussions The proposed method is com-pared with i) an average of registered images, ii) Shan’s BD[13] applied to (i) , iii) SVOLA by Hirsch et al. [3]. To makea fair comparison, these last two methods are applied to theregistered images constructed from the object alignment andframe selection. Two original frames which show the extentof distortion and the associated results are shown in Fig. 3 col-umn 1-2 and 3-6, respectively. Although ground truth imagesare not available, the subjective results clearly show that theproposed method removes atmosphoric distortion more effi-ciently than the other approaches. Shan’s blind deconvolutionis inefficient for air turbulence since the PSF is assumed to besimilar for the entire image, while our method runs a set of ho-mogeneous regions separately. Shan’s method also takes fourtimes longer to process than our proposed method, mainlyto estimate the PSF. SVOLA subdivides an image into over-lapped regions and estimates the PSF separately; as a result, itprovides better results compared to Shan’s method. Howeverthe computational time is even longer and the results are notas sharp as the proposed method. In addition, prior knowl-edge of PSF size is required for both previous methods.4. CONCLUSIONSThis paper has introduced a new method for mitigating at-mospheric distortion in long range surveillance imaging. Theimprovement of visibility of an ROI in the image sequence isachieved using region-based fusion in the DT-CWT domain.We also propose a simple object alignment method and a newcost funtion for frame selection to pre-process the distortedsequence. The process is completed with local contrast en-hancement to remove haze interference. Experiments withreal data show promising results and superior performanceOriginal frame 1 Original frame 2 Average Shan’s BD [13] SVOLA [3] proposedFig. 3. Original images, Average image, and results of Shan’s BD [13], SVOLA [3] and proposed methodcompared with the existing methods.5. ACKNOWLEDGEMENTSThe authors acknowledge the support of General Dynamics UnitedKingdom Limited.6. REFERENCES[1] Edmund Y. Lam and Joseph W. Goodman, “Iterative statistical ap-proach to blind image deconvolution,” J. Opt. Soc. Am. A, vol. 17, no.7, pp. 1177–1184, Jul 2000.[2] S. Harmeling, M. Hirsch, S. Sra, and B. Scholkopf, “Online blindimage deconvolution for astronomy,” in Proc of IEEE Conf Comp.Photogr., 2009.[3] M. Hirsch, S. Sra, B. Scholkopf, and S. Harmeling, “Efficient filterflow for space-variant multiframe blind deconvolution,” in ComputerVision and Pattern Recognition (CVPR), june 2010, pp. 607 –614.[4] Dalong Li, “Suppressing atmospheric turbulent motion in videothrough trajectory smoothing,” Signal Processing, vol. 89, no. 4, pp.649 – 655, 2009.[5] P. J. Kent, S. B. Foulkes, J. G. Burnett, S. C. Woods, and A. J. Turner,“Progress towards a real-time active lucky imaging system,” in EMRCDTC Technical Conference, 2010.[6] C. S. Huebner and C. Scheifling, “Software-based mitigation of imagedegradation due to atmospheric turbulence,” in SPIE conference onOptics in Atmospheric Propagation and Adaptive Systems, 2010.[7] I.W. Selesnick, R.G. Baraniuk, and N.G. Kingsbury, “The dual-treecomplex wavelet transform,” Signal Processing Magazine, IEEE, vol.22, no. 6, pp. 123 – 151, nov. 2005.[8] N. Otsu, “A threshold selection method from gray-level histograms,”Systems, Man and Cybernetics, IEEE Transactions on, vol. 9, no. 1, pp.62 –66, jan. 1979.[9] H. Chen and N. Kingsbury, “Efficient registration of nonrigid 3-d bod-ies,” Image Processing, IEEE Transactions on, vol. 21, no. 1, pp. 262–272, jan. 2012.[10] J. J. Lewis, R. J. O’Callaghan, S. G. Nikolov, D. R. Bull, C. N. Cana-garajah, and Essa Basaeed, “Region-based image fusion using complexwavelets,” in Proc. 7th Inter. Conf. Info. Fusion, 2004, pp. 555–562.[11] R.J. O’Callaghan and D.R. Bull, “Combined morphological-spectralunsupervised image segmentation,” Image Processing, IEEE Transac-tions on, vol. 14, no. 1, pp. 49 –62, jan. 2005.[12] Artur Loza, David Bull, Nishan Canagarajah, and Alin Achim, “Non-gaussian model-based fusion of noisy images in the wavelet domain,”Comput. Vis. Image Underst., vol. 114, pp. 54–65, January 2010.[13] Qi Shan, Jiaya Jia, and Aseem Agarwala, “High-quality motion de-blurring from a single image,” ACM Transactions on Graphics (SIG-GRAPH), 2008.

A threshold selection method from gray-level histograms,” Systems, Man and Cybernetics,

Combined morphological-spectral unsupervised image segmentation,” Image Processing,

Efﬁcient ﬁlter ﬂow for space-variant multiframe blind deconvolution,”

Efﬁcient registration of nonrigid 3-d bodies,” Image Processing,

High-quality motion deblurring from a single image,”

Iterative statistical approach to blind image deconvolution,”

Nishan Canagarajah, and Alin Achim, “Nongaussian model-based fusion of noisy images in the wavelet domain,”

Online blind image deconvolution for astronomy,”

Progress towards a real-time active lucky imaging system,”

Region-based image fusion using complex wavelets,” in

Software-based mitigation of image degradation due to atmospheric turbulence,”

Suppressing atmospheric turbulent motion in video through trajectory smoothing,”

The dual-tree complex wavelet transform,”

Mitigating the effects of atmospheric distortion using DT-CWT fusion

Crossref

https://research-information.bris.ac.uk/files/9033278/Anantrasirichai_icip12.pdf

Mitigating the effects of atmospheric distortion using DT-CWT fusion

Abstract

Similar works

Full text

Available Versions

CiteSeerX

Explore Bristol Research

Crossref