Subjectively optimised multi-exposure and multi-focus image fusion with compensation for camera shake

Multi-exposure image fusion algorithms are used for enhancing the perceptual quality of an image captured by sensors of limited dynamic range. This is achieved by rendering a single scene based on multiple images captured at different exposure times. Similarly, multi-focus image fusion is used when the limited depth of focus on a selected focus setting of a camera results in parts of an image being out of focus. The solution adopted is to fuse together a number of multi-focus images to create an image that is focused throughout. In this paper we propose a single algorithm that can perform both multi-focus and multi-exposure image fusion. This algorithm is a novel approach in which a set of unregistered multiexposure/focus images is first registered before being fused. The registration of images is done via identifying matching key points in constituent images using Scale Invariant Feature Transforms (SIFT). The RANdom SAmple Consensus (RANSAC) algorithm is used to identify inliers of SIFT key points removing outliers that can cause errors in the registration process. Finally we use the Coherent Point Drift algorithm to register the images, preparing them to be fused in the subsequent fusion stage. For the fusion of images, a novel approach based on an improved version of a Wavelet Based Contourlet Transform (WBCT) is used. The experimental results as follows prove that the proposed algorithm is capable of producing HDR, or multi-focus images by registering and fusing a set of multi-exposure or multi-focus images taken in the presence of camera shake

A Novel Multi-Focus Image Fusion Method Based on Stochastic Coordinate Coding and Local Density Peaks Clustering

abstract: The multi-focus image fusion method is used in image processing to generate all-focus images that have large depth of field (DOF) based on original multi-focus images. Different approaches have been used in the spatial and transform domain to fuse multi-focus images. As one of the most popular image processing methods, dictionary-learning-based spare representation achieves great performance in multi-focus image fusion. Most of the existing dictionary-learning-based multi-focus image fusion methods directly use the whole source images for dictionary learning. However, it incurs a high error rate and high computation cost in dictionary learning process by using the whole source images. This paper proposes a novel stochastic coordinate coding-based image fusion framework integrated with local density peaks. The proposed multi-focus image fusion method consists of three steps. First, source images are split into small image patches, then the split image patches are classified into a few groups by local density peaks clustering. Next, the grouped image patches are used for sub-dictionary learning by stochastic coordinate coding. The trained sub-dictionaries are combined into a dictionary for sparse representation. Finally, the simultaneous orthogonal matching pursuit (SOMP) algorithm is used to carry out sparse representation. After the three steps, the obtained sparse coefficients are fused following the max L1-norm rule. The fused coefficients are inversely transformed to an image by using the learned dictionary. The results and analyses of comparison experiments demonstrate that fused images of the proposed method have higher qualities than existing state-of-the-art methods

MDLatLRR: A novel decomposition method for infrared and visible image fusion

Image decomposition is crucial for many image processing tasks, as it allows to extract salient features from source images. A good image decomposition method could lead to a better performance, especially in image fusion tasks. We propose a multi-level image decomposition method based on latent low-rank representation(LatLRR), which is called MDLatLRR. This decomposition method is applicable to many image processing fields. In this paper, we focus on the image fusion task. We develop a novel image fusion framework based on MDLatLRR, which is used to decompose source images into detail parts(salient features) and base parts. A nuclear-norm based fusion strategy is used to fuse the detail parts, and the base parts are fused by an averaging strategy. Compared with other state-of-the-art fusion methods, the proposed algorithm exhibits better fusion performance in both subjective and objective evaluation.Comment: IEEE Trans. Image Processing 2020, 14 pages, 17 figures, 3 table

Multi-scale pixel-based image fusion using multivariate empirical mode decomposition.

A novel scheme to perform the fusion of multiple images using the multivariate empirical mode decomposition (MEMD) algorithm is proposed. Standard multi-scale fusion techniques make a priori assumptions regarding input data, whereas standard univariate empirical mode decomposition (EMD)-based fusion techniques suffer from inherent mode mixing and mode misalignment issues, characterized respectively by either a single intrinsic mode function (IMF) containing multiple scales or the same indexed IMFs corresponding to multiple input images carrying different frequency information. We show that MEMD overcomes these problems by being fully data adaptive and by aligning common frequency scales from multiple channels, thus enabling their comparison at a pixel level and subsequent fusion at multiple data scales. We then demonstrate the potential of the proposed scheme on a large dataset of real-world multi-exposure and multi-focus images and compare the results against those obtained from standard fusion algorithms, including the principal component analysis (PCA), discrete wavelet transform (DWT) and non-subsampled contourlet transform (NCT). A variety of image fusion quality measures are employed for the objective evaluation of the proposed method. We also report the results of a hypothesis testing approach on our large image dataset to identify statistically-significant performance differences

Multi Focus Image Fusion with variable size windows

[EN] In this paper we present the Linear Image Combination Algorithm with Variable Windows (CLI-VV) for the fusion of multifocus images. Unlike the CLI-S algorithm presented in a previous work, the CLI-VV algorithm allows to automatically determine the optimal size of the window in each pixel for the segmentation of the regions with the highest sharpness. We also present the generalized CLI-VV Algorithm for the fusion of sets of multi-focus images with more than two images. This new algorithm is called Variable Windows Multi-focus Fusion (FM-VV). The CLI-VV Algorithm was tested with 21 pairs of synthetic images and 29 pairs of real multi-focus images, and the FM-VV Algorithm on 5 trios of multi-focus images. In all the tests a competitive accuracy was obtained, with execution times lower than those reported in the literature.[ES] En este artículo presentamos el Algoritmo Combinación Lineal de Imágenes con Ventanas Variables (CLI-VV) para la fusión de imágenes multi-foco. A diferencia del Algoritmo CLI-S presentado en un trabajo anterior, el algoritmo CLI-VV permite determinar automáticamente el tamaño óptimo de la ventana en cada píxel para la segmentación de las regiones con la mayor nitidez. También presentamos la generalizado el Algoritmo CLI-VV para la fusión de conjuntos de imágenes multi-foco con más de dos imágenes. A este nuevo algoritmo lo denominamos Fusión Multi-foco con Ventanas Variables (FM-VV). El Algoritmo CLI-VV se probó con 21 pares de imágenes sintéticas y 29 pares de imágenes multi-foco reales, y el Algoritmo FM-VV sobre 5 tríos de imágenes multi-foco. En todos los ejemplos se obtuvo un porcentaje de acierto competitivos, producidos en tiempos de ejecución menores a los reportados en la literatura.Calderon, F.; Garnica-Carrillo, A.; Flores, JJ. (2018). Fusión de Imágenes Multi-Foco con Ventanas Variables. Revista Iberoamericana de Automática e Informática industrial. 15(3):262-276. https://doi.org/10.4995/riai.2017.8852OJS262276153Aslantas, V., Kurban, R., 2010. Fusion of multi-focus images using differential evolution algorithm. Expert Systems with Applications 37 (12), 8861 - 8870. https://doi.org/10.1016/j.eswa.2010.06.011Aslantas, V., Toprak, A. N., 2014. A pixel based multi-focus image fusion method. Optics Communications 332, 350 - 358. https://doi.org/10.1016/j.optcom.2014.07.044Aslantas, V., Toprak, A. N., 2017. Multi-focus image fusion based on optimal defocus estimation. Computers and Electrical Engineering. https://doi.org/10.1016/j.compeleceng.2017.02.003Assirati, L., Silva, N. R., Berton, L., Lopes, A. A., Bruno, O. M., 2014. Performing edge detection by difference of gaussians using q-gaussian kernels. Journal of Physics: Conference Series 490 (1), 012020. https://doi.org/10.1088/1742-6596/490/1/012020Bai, X., Zhang, Y., Zhou, F., Xue, B., 2015. Quadtree-based multi-focus image fusion using a weighted focus-measure. Information Fusion 22, 105 - 118. https://doi.org/10.1016/j.inffus.2014.05.003Calderon, F., Garnica, A., 2014. Multi focus image fusion based on linear combination of images. IEEE, pp. 1-7. https://doi.org/10.1109/ROPEC.2014.7036340Calderon, F., Garnica-Carrillo, A., Flores, J. J., 2016. Fusión de imágenes multi foco basado en la combinación lineal de imágenes utilizando imágenes incrementales. Revista Iberoamericana de Automática e Informática Industrial RIAI 13 (4), 450 - 461. https://doi.org/10.1016/j.riai.2016.07.002Cao, L., Jin, L., Tao, H., Li, G., Zhuang, Z., Zhang, Y., Feb 2015. Multi-focus image fusion based on spatial frequency in discrete cosine transform domain. Signal Processing Letters, IEEE 22 (2), 220-224. https://doi.org/10.1109/LSP.2014.2354534Chai, Y., Li, H., Li, Z., 2011. Multifocus image fusion scheme using focused region detection and multiresolution. Optics Communications 284 (19), 4376 - 4389. https://doi.org/10.1016/j.optcom.2011.05.046De, I., Chanda, B., 2013. Multi-focus image fusion using a morphology-based focus measure in a quad-tree structure. Information Fusion 14 (2), 136 - 146. https://doi.org/10.1016/j.inffus.2012.01.007Duan, J., Meng, G., Xiang, S., Pan, C., 2014. Multifocus image fusion via focus segmentation and region reconstruction. Neurocomputing 140, 193 - 209. https://doi.org/10.1016/j.neucom.2014.03.023Eskicioglu, A., Fisher, P., Dec 1995. Image quality measures and their performance. Communications, IEEE Transactions on 43 (12), 2959-2965. https://doi.org/10.1109/26.477498Kong, W., Lei, Y., 2017. Multi-focus image fusion using biochemical ion exchange model. Applied Soft Computing 51, 314 - 327. https://doi.org/10.1016/j.asoc.2016.11.033Kuthirummal, S., Nagahara, H., Zhou, C., Nayar, S., Jan 2011. Flexible depth of field photography. Pattern Analysis and Machine Intelligence, IEEE Transactions on 33 (1), 58-71. https://doi.org/10.1109/TPAMI.2010.66Lewis, J. J., O'Callaghan, R. J., Nikolov, S. G., Bull, D. R., Canagarajah, N., 2007. Pixel- and region-based image fusion with complex wavelets. Information Fusion 8 (2), 119 - 130, special Issue on Image Fusion: Advances in the State of the Art. https://doi.org/10.1016/j.inffus.2005.09.006Li, H., Chai, Y., Li, Z., 2013a. Multi-focus image fusion based on nonsubsampled contourlet transform and focused regions detection. Optik - International Journal for Light and Electron Optics 124 (1), 40 - 51. https://doi.org/10.1016/j.ijleo.2011.11.088Li, H., Chai, Y., Li, Z., 2013b. A new fusion scheme for multifocus images based on focused pixels detection. Machine vision and applications 24 (6), 1167-1181. https://doi.org/10.1007/s00138-013-0502-4Li, H., Manjunath, B., Mitra, S., 1995. Multisensor image fusion using the wavelet transform. Graphical Models and Image Processing 57 (3), 235 - 245. https://doi.org/10.1006/gmip.1995.1022Li, S., Kang, X., Fang, L., Hu, J., Yin, H., 2017. Pixel-level image fusion: A survey of the state of the art. Information Fusion 33, 100 - 112. https://doi.org/10.1016/j.inffus.2016.05.004Li, S., Kwok, J. T., Wang, Y., 2001. Combination of images with diverse focuses using the spatial frequency. Information Fusion 2 (3), 169 - 176. https://doi.org/10.1016/S1566-2535(01)00038-0Li, S., Kwok, J. T., Wang, Y., 2002. Multifocus image fusion using artificial neural networks. Pattern Recognition Letters 23 (8), 985 - 997. https://doi.org/10.1016/S0167-8655(02)00029-6Li, S., Yang, B., 2008a. Multifocus image fusion by combining curvelet and wavelet transform. Pattern Recognition Letters 29 (9), 1295-1301. https://doi.org/10.1016/j.patrec.2008.02.002Li, S., Yang, B., 2008b. Multifocus image fusion using region segmentation and spatial frequency. Image and Vision Computing 26 (7), 971 - 979. https://doi.org/10.1016/j.imavis.2007.10.012Li, X., He, M., Roux, M., August 2010. Multifocus image fusion based on redundant wavelet transform. Image Processing, IET 4 (4), 283-293. https://doi.org/10.1049/iet-ipr.2008.0259Liu, Y., Chen, X., Peng, H., Wang, Z., 2017a. Multi-focus image fusion with a deep convolutional neural network. Information Fusion 36, 191 - 207. https://doi.org/10.1016/j.inffus.2016.12.001Liu, Z., Chai, Y., Yin, H., Zhou, J., Zhu, Z., 2017b. A novel multi-focus image fusion approach based on image decomposition. Information Fusion 35, 102 - 116. https://doi.org/10.1016/j.inffus.2016.09.007Long, J., Shelhamer, E., Darrell, T., 2014. Fully convolutional networks for semantic segmentation. CoRR abs/1411.4038.Luo, X., Zhang, J., Dai, Q., 2012. A regional image fusion based on similarity characteristics. Signal Processing 92 (5), 1268 - 1280. https://doi.org/10.1016/j.sigpro.2011.11.021Ma, Y., Zhan, K.,Wang, Z., service), S. O., 2011. Applications of pulse-coupled neural networks.Malviya, A., Bhirud, S., Dec 2009. Wavelet based multi-focus image fusion. In: Methods and Models in Computer Science, 2009. ICM2CS 2009. Proceeding of International Conference on. pp. 1-6. https://doi.org/10.1109/ICM2CS.2009.5397990Nejati, M., Samavi, S., Shirani, S., 2015. Multi-focus image fusion using dictionary-based sparse representation. Information Fusion 25, 72 - 84. https://doi.org/10.1016/j.inffus.2014.10.004Orozco, R. I., 2013. Fusión de imágenes multifoco por medio de filtrado de regiones de alta y baja frecuencia. Master's thesis, División de Estudios de Postgrado. Facultad de Ingeniería Eléctrica. UMSNH, Morelia Michoacan Mexico.Pagidimarry, M., Babu, K. A., 2011. An all approach for multi-focus image fusion using neural network. Artificial Intelligent Systems and Machine Learning 3 (12), 732-739.Pajares, G., de la Cruz, J. M., 2004. A wavelet-based image fusion tutorial. Pattern Recognition 37 (9), 1855 - 1872. https://doi.org/10.1016/j.patcog.2004.03.010Piella, G., 2003. A general framework for multiresolution image fusion: from pixels to regions. Information Fusion 4 (4), 259 - 280. https://doi.org/10.1016/S1566-2535(03)00046-0Pramanik, S., Prusty, S., Bhattacharjee, D., Bhunre, P. K., 2013. A region-topixel based multi-sensor image fusion. Procedia Technology 10, 654 - 662. https://doi.org/10.1016/j.protcy.2013.12.407Qu, X., Hou, Y., Lam, F., Guo, D., Zhong, J., Chen, Z., 2014. Magnetic resonance image reconstruction from undersampled measurements using a patchbased nonlocal operator. Medical Image Analysis 18 (6), 843 - 856, sparse Methods for Signal Reconstruction and Medical Image Analysis. https://doi.org/10.1016/j.media.2013.09.007Riaz, M., Park, S., Ahmad, M., Rasheed, W., Park, J., 2008. Generalized laplacian as focus measure. In: Bubak, M., van Albada, G., Dongarra, J., Sloot, P. (Eds.), Computational Science ICCS 2008. Vol. 5101 of Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp. 1013-1021. https://doi.org/10.1007/978-3-540-69384-0_106Rivera, M., Ocegueda, O., Marroquin, J., Dec 2007. Entropy-controlled quadratic markov measure field models for efficient image segmentation. Image Processing, IEEE Transactions on 16 (12), 3047-3057. https://doi.org/10.1109/TIP.2007.909384Sezan, M., Pavlovic, G., Tekalp, A., Erdem, A., Apr 1991. On modeling the focus blur in image restoration. In: Acoustics, Speech, and Signal Processing, 1991. ICASSP-91., 1991 International Conference on. pp. 2485-2488 vol.4. https://doi.org/10.1109/ICASSP.1991.150905Shah, P., Merchant, S. N., Desai, U. B., 2013. Multifocus and multispectral image fusion based on pixel significance using multiresolution decomposition. Signal, Image and Video Processing 7 (1), 95-109. https://doi.org/10.1007/s11760-011-0219-7Shi, W., Zhu, C., Tian, Y., Nichol, J., 2005. Wavelet-based image fusion and quality assessment. International Journal of Applied Earth Observation and Geoinformation 6 (3-4), 241 - 251. https://doi.org/10.1016/j.jag.2004.10.010Tian, J., Chen, L., Sept 2010. Multi-focus image fusion using wavelet-domain statistics. In: Image Processing (ICIP), 2010 17th IEEE International Conference on. pp. 1205-1208. https://doi.org/10.1109/ICIP.2010.5651791Viola, P., Jones, M., 2001. Rapid object detection using a boosted cascade of simple features. In: Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on. Vol. 1. pp. I-511-I-518 vol.1. https://doi.org/10.1109/CVPR.2001.990517Yang, Y., 2011. A novel fDWTg based multi-focus image fusion method. Procedia Engineering 24 (0), 177 - 181, international Conference on Advances in Engineering 2011.Yang, Y., Huang, S., Gao, J., Qian, Z., 2014. Multi-focus image fusion using an effective discrete wavelet transform based algorithm. Measurement Science Review 14 (2), 102 - 108. https://doi.org/10.2478/msr-2014-0014Yang, Y., Tong, S., Huang, S., Lin, P., 2015. Multifocus image fusion based on nsct and focused area detection. IEEE Sensors Journal 15 (5), 2824-2838. Zhang, B., Lu, X., Pei, H., Liu, H., Zhao, Y., Zhou, W., 2016a. Multi-focus image fusion algorithm based on focused region extraction. Neurocomputing 174, 733 - 748. https://doi.org/10.1016/j.neucom.2015.09.092Zhang, Q., long Guo, B., 2009. Multifocus image fusion using the nonsubsampled contourlet transform. Signal Processing 89 (7), 1334 - 1346. https://doi.org/10.1016/j.sigpro.2009.01.012Zhang, Y., Chen, L., Zhao, Z., Jia, J., 2016b. Multi-focus image fusion based on cartoon-texture image decomposition. Optik - International Journal for Light and Electron Optics 127 (3), 1291 - 1296. https://doi.org/10.1016/j.ijleo.2015.10.098Zhang, Z., Blum, R., Aug 1999. A categorization of multiscale-decompositionbased image fusion schemes with a performance study for a digital camera application. Proceedings of the IEEE 87 (8), 1315-1326. https://doi.org/10.1109/5.775414Zhao, H., Li, Q., Feng, H., 2008. Multi-focus color image fusion in the HSI space using the sum-modified-laplacian and a coarse edge map. Image and Vision Computing 26 (9), 1285 - 1295. https://doi.org/10.1016/j.imavis.2008.03.007Zhou, L., Ji, G., Shi, C., Feng, C., Nian, R., 2006. A Multi-focus Image Fusion Method Based on Image Information Features and the Artificial Neural Networks. Vol. 344. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 747-752. https://doi.org/10.1007/978-3-540-37256-1_91Zhou, Z., Li, S., Wang, B., 2014. Multi-scale weighted gradient-based fusion for multi-focus images. Information Fusion 20 (0), 60 - 72. https://doi.org/10.1016/j.inffus.2013.11.00

HoEnTOA: Holoentropy and Taylor Assisted Optimization based Novel Image Quality Enhancement Algorithm for Multi-Focus Image Fusion

In machine vision as well as image processing applications, multi-focus image fusion strategy carries a prominent exposure. Normally, image fusion is a method of merging of information extracted out of two or more than two source images fused to produce a solitary image, which is much more instructive as well as much suitable for computer processing and visual perception. In this research paper authors have devised a novel image quality enhancement algorithm by fusing multi-focus images, in short, termed as HoEnTOA. Initially, contourlet transform is incorporated to both of the input images for generation of four respective sub-bands of each of input image. After converting into sub-bands further holoentropy along with proposed HoEnTOA is introduced to fuse multi-focus images. Here, the developed HoEnTOA is integration of Taylor series with ASSCA. After fusion, the inverse contourlet transform is incorporated for obtaining last fused image. Thus, the proposed HoEnTOA effectively performs the image fusion and has demonstrated better performance utilizing the five metrics i.e. Root Mean Square Error with a minimum value of 3.687, highest universal quality index value of 0.984, maximum Peak Signal to Noise Ratio of 42.08dB, maximal structural similarity index measurement of 0.943, as well as maximum mutual information of 1.651

HoEnTOA: Holoentropy and Taylor Assisted Optimization based Novel Image Quality Enhancement Algorithm for Multi-Focus Image Fusion

875-886In machine vision as well as image processing applications, multi-focus image fusion strategy carries a prominent exposure. Normally, image fusion is a method of merging of information extracted out of two or more than two source images fused to produce a solitary image, which is much more instructive as well as much suitable for computer processing and visual perception. In this research paper authors have devised a novel image quality enhancement algorithm by fusing multi-focus images, in short, termed as HoEnTOA. Initially, contourlet transform is incorporated to both of the input images for generation of four respective sub-bands of each of input image. After converting into sub-bands further holoentropy along with proposed HoEnTOA is introduced to fuse multi-focus images. Here, the developed HoEnTOA is integration of Taylor series with ASSCA. After fusion, the inverse contourlet transform is incorporated for obtaining last fused image. Thus, the proposed HoEnTOA effectively performs the image fusion and has demonstrated better performance utilizing the five metrics i.e. Root Mean Square Error with a minimum value of 3.687, highest universal quality index value of 0.984, maximum Peak Signal to Noise Ratio of 42.08dB, maximal structural similarity index measurement of 0.943, as well as maximum mutual information of 1.651