UAV-Multispectral Sensed Data Band Co-Registration Framework

Precision farming has greatly benefited from new technologies over the years. The use of multispectral and hyperspectral sensors coupled to Unmanned Aerial Vehicles (UAV) has enabled farms to monitor crops, improve the use of resources and reduce costs. Despite being widely used, multispectral images present a natural misalignment among the various spectra due to the use of different sensors. The variation of the analyzed spectrum also leads to a loss of characteristics among the bands which hinders the feature detection process among the bands, which makes the alignment process complex. In this work, we propose a new framework for the band co-registration process based on two premises: i) the natural misalignment is an attribute of the camera, so it does not change during the acquisition process; ii) the speed of displacement of the UAV when compared to the speed between the acquisition of the first to the last band, is not sufficient to create significant distortions. We compared our results with the ground-truth generated by a specialist and with other methods present in the literature. The proposed framework had an average back-projection (BP) error of 0.425 pixels, this result being 335% better than the evaluated frameworks.CAPES - Coordenação de Aperfeiçoamento de Pessoal de Nível SuperiorDissertação (Mestrado)A agricultura de precisão se beneficiou muito das novas tecnologias ao longo dos anos. O uso de sensores multiespectrais e hiperespectrais acoplados aos Veículos Aéreos Não Tripulados (VANT) permitiu que as fazendas monitorassem as lavouras, melhorassem o uso de recursos e reduzissem os custos. Apesar de amplamente utilizadas, as imagens multiespectrais apresentam um desalinhamento natural entre os vários espectros devido ao uso de diferentes sensores. A variação do espectro analisado também leva à perda de características entre as bandas, o que dificulta o processo de detecção de atributos entre as bandas, o que torna complexo o processo de alinhamento. Neste trabalho, propomos um novo framework para o processo de alinhamento entre as bandas com base em duas premissas: i) o desalinhamento natural é um atributo da câmera, e por esse motivo ele não é alterado durante o processo de aquisição; ii) a velocidade de deslocamento do VANT, quando comparada à velocidade entre a aquisição da primeira e a última banda, não é suficiente para criar distorções significativas. Os resultados obtidos foram comparados com o padrão ouro gerado por um especialista e com outros métodos presentes na literatura. O framework proposto teve um back-projection error (BP) de 0, 425 pixels, sendo este resultado 335% melhor aos frameworks avaliados

AN IMPROVED IMAGE REGISTRATION ALGORITHM FOR THERMAL INFRARED AND PANCHROMATIC IMAGE BASED ON GEOMETRIC STRUCTURAL PROPERTIES

In order to improve the image registration accuracy of heterogeneous remote sensing images with large radiation differences, an improved image registration algorithm for thermal infrared and panchromatic images is proposed. This method uses the phase consistency of intensity and direction to construct a geometric structure feature descriptor called the Histogram of Oriented Phase Congruency (HOPC) for image registration. It employs the Curvature Scale Space (CSS) corner detection to concentrate and extract feature points in contour areas, and defines a similarity metric (called HOPCn) based on the Euclidean distance between the descriptors for high-precision heterogeneous images registration. Five sets of panchromatic and thermal infrared images were selected for verification. The results show that the improved algorithm can match the homonymy points around the image contour, and the Correct Matching Rate (CMR) is improved by 2.7% and the Root Mean Square Error (RMSE) is improved by 3.7% compared to the original algorithm. This proves that the proposed method has good robustness under conditions of large radiation differences

Deraining and Desnowing Algorithm on Adaptive Tolerance and Dual-tree Complex Wavelet Fusion

Severe weather conditions such as rain and snow often reduce the visual perception quality of the video image system, the traditional methods of deraining and desnowing usually rarely consider adaptive parameters. In order to enhance the effect of video deraining and desnowing, this paper proposes a video deraining and desnowing algorithm based on adaptive tolerance and dual-tree complex wavelet. This algorithm can be widely used in security surveillance, military defense, biological monitoring, remote sensing and other fields. First, this paper introduces the main work of the adaptive tolerance method for the video of dynamic scenes. Second, the algorithm of dual-tree complex wavelet fusion is analyzed and introduced. Using principal component analysis fusion rules to process low-frequency sub-bands, the fusion rule of local energy matching is used to process the high-frequency sub-bands. Finally, this paper used various rain and snow videos to verify the validity and superiority of image reconstruction. Experimental results show that the algorithm has achieved good results in improving the image clarity and restoring the image details obscured by raindrops and snows

Spatial Stimuli Gradient Based Multifocus Image Fusion Using Multiple Sized Kernels

Multi-focus image fusion technique extracts the focused areas from all the source images and combines them into a new image which contains all focused objects. This paper proposes a spatial domain fusion scheme for multi-focus images by using multiple size kernels. Firstly, source images are pre-processed with a contrast enhancement step and then the soft and hard decision maps are generated by employing a sliding window technique using multiple sized kernels on the gradient images. Hard decision map selects the accurate focus information from the source images, whereas, the soft decision map selects the basic focus information and contains minimum falsely detected focused/unfocused regions. These decision maps are further processed to compute the final focus map. Gradient images are constructed through state-ofthe-art edge detection technique, spatial stimuli gradient sketch model, which computes the local stimuli from perceived brightness and hence enhances the essential structural and edge information. Detailed experiment results demonstrate that the proposed multi-focus image fusion algorithm performs better than the other well known state-of-the-art multifocus image fusion methods, in terms of subjective visual perception and objective quality evaluation metrics

Fusion of enhanced and synthetic vision system images for runway and horizon detection

Networked operation of unmanned air vehicles (UAVs) demands fusion of information from disparate sources for accurate flight control. In this investigation, a novel sensor fusion architecture for detecting aircraft runway and horizons as well as enhancing the awareness of surrounding terrain is introduced based on fusion of enhanced vision system (EVS) and synthetic vision system (SVS) images. EVS and SVS image fusion has yet to be implemented in real-world situations due to signal misalignment. We address this through a registration step to align EVS and SVS images. Four fusion rules combining discrete wavelet transform (DWT) sub-bands are formulated, implemented, and evaluated. The resulting procedure is tested on real EVS-SVS image pairs and pairs containing simulated turbulence. Evaluations reveal that runways and horizons can be detected accurately even in poor visibility. Furthermore, it is demonstrated that different aspects of EVS and SVS images can be emphasized by using different DWT fusion rules. The procedure is autonomous throughout landing, irrespective of weather. The fusion architecture developed in this study holds promise for incorporation into manned heads-up displays (HUDs) and UAV remote displays to assist pilots landing aircraft in poor lighting and varying weather. The algorithm also provides a basis for rule selection in other signal fusion applications

Non-Standard Imaging Techniques

The first objective of the thesis is to investigate the problem of reconstructing a small-scale object (a few millimeters or smaller) in 3D. In Chapter 3, we show how this problem can be solved effectively by a new multifocus multiview 3D reconstruction procedure which includes a new Fixed-Lens multifocus image capture and a calibrated image registration technique using analytic homography transformation. The experimental results using the real and synthetic images demonstrate the effectiveness of the proposed solutions by showing that both the fixed-lens image capture and multifocus stacking with calibrated image alignment significantly reduce the errors in the camera poses and produce more complete 3D reconstructed models as compared with those by the conventional moving lens image capture and multifocus stacking. The second objective of the thesis is modelling the dual-pixel (DP) camera. In Chapter 4, to understand the potential of the DP sensor for computer vision applications, we study the formation of the DP pair which links the blur and the depth information. A mathematical DP model is proposed which can benefit depth estimation by the blur. These explorations motivate us to propose an end-to-end DDDNet (DP-based Depth and Deblur Network) to jointly estimate the depth and restore the image . Moreover, we define a reblur loss, which reflects the relationship of the DP image formation process with depth information, to regularize our depth estimate in training. To meet the requirement of a large amount of data for learning, we propose the first DP image simulator which allows us to create datasets with DP pairs from any existing RGBD dataset. As a side contribution, we collect a real dataset for further research. Extensive experimental evaluation on both synthetic and real datasets shows that our approach achieves competitive performance compared to state-of-the-art approaches. Another (third) objective of this thesis is to tackle the multifocus image fusion problem, particularly for long multifocus image sequences. Multifocus image stacking/fusion produces an in-focus image of a scene from a number of partially focused images of that scene in order to extend the depth of field. One of the limitations of the current state of the art multifocus fusion methods is not considering image registration/alignment before fusion. Consequently, fusing unregistered multifocus images produces an in-focus image containing misalignment artefacts. In Chapter 5, we propose image registration by projective transformation before fusion to remove the misalignment artefacts. We also propose a method based on 3D deconvolution to retrieve the in-focus image by formulating the multifocus image fusion problem as a 3D deconvolution problem. The proposed method achieves superior performance compared to the state of the art methods. It is also shown that, the proposed projective transformation for image registration can improve the quality of the fused images. Moreover, we implement a multifocus simulator to generate synthetic multifocus data from any RGB-D dataset. The fourth objective of this thesis is to explore new ways to detect the polarization state of light. To achieve the objective, in Chapter 6, we investigate a new optical filter namely optical rotation filter for detecting the polarization state with a fewer number of images. The proposed method can estimate polarization state using two images, one with the filter and another without. The accuracy of estimating the polarization parameters using the proposed method is almost similar to that of the existing state of the art method. In addition, the feasibility of detecting the polarization state using only one RGB image captured with the optical rotation filter is also demonstrated by estimating the image without the filter from the image with the filter using a generative adversarial network