Multi Focus Image Fusion with variable size windows

[EN] In this paper we present the Linear Image Combination Algorithm with Variable Windows (CLI-VV) for the fusion of multifocus images. Unlike the CLI-S algorithm presented in a previous work, the CLI-VV algorithm allows to automatically determine the optimal size of the window in each pixel for the segmentation of the regions with the highest sharpness. We also present the generalized CLI-VV Algorithm for the fusion of sets of multi-focus images with more than two images. This new algorithm is called Variable Windows Multi-focus Fusion (FM-VV). The CLI-VV Algorithm was tested with 21 pairs of synthetic images and 29 pairs of real multi-focus images, and the FM-VV Algorithm on 5 trios of multi-focus images. In all the tests a competitive accuracy was obtained, with execution times lower than those reported in the literature.[ES] En este artículo presentamos el Algoritmo Combinación Lineal de Imágenes con Ventanas Variables (CLI-VV) para la fusión de imágenes multi-foco. A diferencia del Algoritmo CLI-S presentado en un trabajo anterior, el algoritmo CLI-VV permite determinar automáticamente el tamaño óptimo de la ventana en cada píxel para la segmentación de las regiones con la mayor nitidez. También presentamos la generalizado el Algoritmo CLI-VV para la fusión de conjuntos de imágenes multi-foco con más de dos imágenes. A este nuevo algoritmo lo denominamos Fusión Multi-foco con Ventanas Variables (FM-VV). El Algoritmo CLI-VV se probó con 21 pares de imágenes sintéticas y 29 pares de imágenes multi-foco reales, y el Algoritmo FM-VV sobre 5 tríos de imágenes multi-foco. En todos los ejemplos se obtuvo un porcentaje de acierto competitivos, producidos en tiempos de ejecución menores a los reportados en la literatura.Calderon, F.; Garnica-Carrillo, A.; Flores, JJ. (2018). Fusión de Imágenes Multi-Foco con Ventanas Variables. Revista Iberoamericana de Automática e Informática industrial. 15(3):262-276. https://doi.org/10.4995/riai.2017.8852OJS262276153Aslantas, V., Kurban, R., 2010. Fusion of multi-focus images using differential evolution algorithm. Expert Systems with Applications 37 (12), 8861 - 8870. https://doi.org/10.1016/j.eswa.2010.06.011Aslantas, V., Toprak, A. N., 2014. A pixel based multi-focus image fusion method. Optics Communications 332, 350 - 358. https://doi.org/10.1016/j.optcom.2014.07.044Aslantas, V., Toprak, A. N., 2017. Multi-focus image fusion based on optimal defocus estimation. Computers and Electrical Engineering. https://doi.org/10.1016/j.compeleceng.2017.02.003Assirati, L., Silva, N. R., Berton, L., Lopes, A. A., Bruno, O. M., 2014. Performing edge detection by difference of gaussians using q-gaussian kernels. Journal of Physics: Conference Series 490 (1), 012020. https://doi.org/10.1088/1742-6596/490/1/012020Bai, X., Zhang, Y., Zhou, F., Xue, B., 2015. Quadtree-based multi-focus image fusion using a weighted focus-measure. Information Fusion 22, 105 - 118. https://doi.org/10.1016/j.inffus.2014.05.003Calderon, F., Garnica, A., 2014. Multi focus image fusion based on linear combination of images. IEEE, pp. 1-7. https://doi.org/10.1109/ROPEC.2014.7036340Calderon, F., Garnica-Carrillo, A., Flores, J. J., 2016. Fusión de imágenes multi foco basado en la combinación lineal de imágenes utilizando imágenes incrementales. Revista Iberoamericana de Automática e Informática Industrial RIAI 13 (4), 450 - 461. https://doi.org/10.1016/j.riai.2016.07.002Cao, L., Jin, L., Tao, H., Li, G., Zhuang, Z., Zhang, Y., Feb 2015. Multi-focus image fusion based on spatial frequency in discrete cosine transform domain. Signal Processing Letters, IEEE 22 (2), 220-224. https://doi.org/10.1109/LSP.2014.2354534Chai, Y., Li, H., Li, Z., 2011. Multifocus image fusion scheme using focused region detection and multiresolution. Optics Communications 284 (19), 4376 - 4389. https://doi.org/10.1016/j.optcom.2011.05.046De, I., Chanda, B., 2013. Multi-focus image fusion using a morphology-based focus measure in a quad-tree structure. Information Fusion 14 (2), 136 - 146. https://doi.org/10.1016/j.inffus.2012.01.007Duan, J., Meng, G., Xiang, S., Pan, C., 2014. Multifocus image fusion via focus segmentation and region reconstruction. Neurocomputing 140, 193 - 209. https://doi.org/10.1016/j.neucom.2014.03.023Eskicioglu, A., Fisher, P., Dec 1995. Image quality measures and their performance. Communications, IEEE Transactions on 43 (12), 2959-2965. https://doi.org/10.1109/26.477498Kong, W., Lei, Y., 2017. Multi-focus image fusion using biochemical ion exchange model. Applied Soft Computing 51, 314 - 327. https://doi.org/10.1016/j.asoc.2016.11.033Kuthirummal, S., Nagahara, H., Zhou, C., Nayar, S., Jan 2011. Flexible depth of field photography. Pattern Analysis and Machine Intelligence, IEEE Transactions on 33 (1), 58-71. https://doi.org/10.1109/TPAMI.2010.66Lewis, J. J., O'Callaghan, R. J., Nikolov, S. G., Bull, D. R., Canagarajah, N., 2007. Pixel- and region-based image fusion with complex wavelets. Information Fusion 8 (2), 119 - 130, special Issue on Image Fusion: Advances in the State of the Art. https://doi.org/10.1016/j.inffus.2005.09.006Li, H., Chai, Y., Li, Z., 2013a. Multi-focus image fusion based on nonsubsampled contourlet transform and focused regions detection. Optik - International Journal for Light and Electron Optics 124 (1), 40 - 51. https://doi.org/10.1016/j.ijleo.2011.11.088Li, H., Chai, Y., Li, Z., 2013b. A new fusion scheme for multifocus images based on focused pixels detection. Machine vision and applications 24 (6), 1167-1181. https://doi.org/10.1007/s00138-013-0502-4Li, H., Manjunath, B., Mitra, S., 1995. Multisensor image fusion using the wavelet transform. Graphical Models and Image Processing 57 (3), 235 - 245. https://doi.org/10.1006/gmip.1995.1022Li, S., Kang, X., Fang, L., Hu, J., Yin, H., 2017. Pixel-level image fusion: A survey of the state of the art. Information Fusion 33, 100 - 112. https://doi.org/10.1016/j.inffus.2016.05.004Li, S., Kwok, J. T., Wang, Y., 2001. Combination of images with diverse focuses using the spatial frequency. Information Fusion 2 (3), 169 - 176. https://doi.org/10.1016/S1566-2535(01)00038-0Li, S., Kwok, J. T., Wang, Y., 2002. Multifocus image fusion using artificial neural networks. Pattern Recognition Letters 23 (8), 985 - 997. https://doi.org/10.1016/S0167-8655(02)00029-6Li, S., Yang, B., 2008a. Multifocus image fusion by combining curvelet and wavelet transform. Pattern Recognition Letters 29 (9), 1295-1301. https://doi.org/10.1016/j.patrec.2008.02.002Li, S., Yang, B., 2008b. Multifocus image fusion using region segmentation and spatial frequency. Image and Vision Computing 26 (7), 971 - 979. https://doi.org/10.1016/j.imavis.2007.10.012Li, X., He, M., Roux, M., August 2010. Multifocus image fusion based on redundant wavelet transform. Image Processing, IET 4 (4), 283-293. https://doi.org/10.1049/iet-ipr.2008.0259Liu, Y., Chen, X., Peng, H., Wang, Z., 2017a. Multi-focus image fusion with a deep convolutional neural network. Information Fusion 36, 191 - 207. https://doi.org/10.1016/j.inffus.2016.12.001Liu, Z., Chai, Y., Yin, H., Zhou, J., Zhu, Z., 2017b. A novel multi-focus image fusion approach based on image decomposition. Information Fusion 35, 102 - 116. https://doi.org/10.1016/j.inffus.2016.09.007Long, J., Shelhamer, E., Darrell, T., 2014. Fully convolutional networks for semantic segmentation. CoRR abs/1411.4038.Luo, X., Zhang, J., Dai, Q., 2012. A regional image fusion based on similarity characteristics. Signal Processing 92 (5), 1268 - 1280. https://doi.org/10.1016/j.sigpro.2011.11.021Ma, Y., Zhan, K.,Wang, Z., service), S. O., 2011. Applications of pulse-coupled neural networks.Malviya, A., Bhirud, S., Dec 2009. Wavelet based multi-focus image fusion. In: Methods and Models in Computer Science, 2009. ICM2CS 2009. Proceeding of International Conference on. pp. 1-6. https://doi.org/10.1109/ICM2CS.2009.5397990Nejati, M., Samavi, S., Shirani, S., 2015. Multi-focus image fusion using dictionary-based sparse representation. Information Fusion 25, 72 - 84. https://doi.org/10.1016/j.inffus.2014.10.004Orozco, R. I., 2013. Fusión de imágenes multifoco por medio de filtrado de regiones de alta y baja frecuencia. Master's thesis, División de Estudios de Postgrado. Facultad de Ingeniería Eléctrica. UMSNH, Morelia Michoacan Mexico.Pagidimarry, M., Babu, K. A., 2011. An all approach for multi-focus image fusion using neural network. Artificial Intelligent Systems and Machine Learning 3 (12), 732-739.Pajares, G., de la Cruz, J. M., 2004. A wavelet-based image fusion tutorial. Pattern Recognition 37 (9), 1855 - 1872. https://doi.org/10.1016/j.patcog.2004.03.010Piella, G., 2003. A general framework for multiresolution image fusion: from pixels to regions. Information Fusion 4 (4), 259 - 280. https://doi.org/10.1016/S1566-2535(03)00046-0Pramanik, S., Prusty, S., Bhattacharjee, D., Bhunre, P. K., 2013. A region-topixel based multi-sensor image fusion. Procedia Technology 10, 654 - 662. https://doi.org/10.1016/j.protcy.2013.12.407Qu, X., Hou, Y., Lam, F., Guo, D., Zhong, J., Chen, Z., 2014. Magnetic resonance image reconstruction from undersampled measurements using a patchbased nonlocal operator. Medical Image Analysis 18 (6), 843 - 856, sparse Methods for Signal Reconstruction and Medical Image Analysis. https://doi.org/10.1016/j.media.2013.09.007Riaz, M., Park, S., Ahmad, M., Rasheed, W., Park, J., 2008. Generalized laplacian as focus measure. In: Bubak, M., van Albada, G., Dongarra, J., Sloot, P. (Eds.), Computational Science ICCS 2008. Vol. 5101 of Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp. 1013-1021. https://doi.org/10.1007/978-3-540-69384-0_106Rivera, M., Ocegueda, O., Marroquin, J., Dec 2007. Entropy-controlled quadratic markov measure field models for efficient image segmentation. Image Processing, IEEE Transactions on 16 (12), 3047-3057. https://doi.org/10.1109/TIP.2007.909384Sezan, M., Pavlovic, G., Tekalp, A., Erdem, A., Apr 1991. On modeling the focus blur in image restoration. In: Acoustics, Speech, and Signal Processing, 1991. ICASSP-91., 1991 International Conference on. pp. 2485-2488 vol.4. https://doi.org/10.1109/ICASSP.1991.150905Shah, P., Merchant, S. N., Desai, U. B., 2013. Multifocus and multispectral image fusion based on pixel significance using multiresolution decomposition. Signal, Image and Video Processing 7 (1), 95-109. https://doi.org/10.1007/s11760-011-0219-7Shi, W., Zhu, C., Tian, Y., Nichol, J., 2005. Wavelet-based image fusion and quality assessment. International Journal of Applied Earth Observation and Geoinformation 6 (3-4), 241 - 251. https://doi.org/10.1016/j.jag.2004.10.010Tian, J., Chen, L., Sept 2010. Multi-focus image fusion using wavelet-domain statistics. In: Image Processing (ICIP), 2010 17th IEEE International Conference on. pp. 1205-1208. https://doi.org/10.1109/ICIP.2010.5651791Viola, P., Jones, M., 2001. Rapid object detection using a boosted cascade of simple features. In: Computer Vision and Pattern Recognition, 2001. CVPR 2001. Proceedings of the 2001 IEEE Computer Society Conference on. Vol. 1. pp. I-511-I-518 vol.1. https://doi.org/10.1109/CVPR.2001.990517Yang, Y., 2011. A novel fDWTg based multi-focus image fusion method. Procedia Engineering 24 (0), 177 - 181, international Conference on Advances in Engineering 2011.Yang, Y., Huang, S., Gao, J., Qian, Z., 2014. Multi-focus image fusion using an effective discrete wavelet transform based algorithm. Measurement Science Review 14 (2), 102 - 108. https://doi.org/10.2478/msr-2014-0014Yang, Y., Tong, S., Huang, S., Lin, P., 2015. Multifocus image fusion based on nsct and focused area detection. IEEE Sensors Journal 15 (5), 2824-2838. Zhang, B., Lu, X., Pei, H., Liu, H., Zhao, Y., Zhou, W., 2016a. Multi-focus image fusion algorithm based on focused region extraction. Neurocomputing 174, 733 - 748. https://doi.org/10.1016/j.neucom.2015.09.092Zhang, Q., long Guo, B., 2009. Multifocus image fusion using the nonsubsampled contourlet transform. Signal Processing 89 (7), 1334 - 1346. https://doi.org/10.1016/j.sigpro.2009.01.012Zhang, Y., Chen, L., Zhao, Z., Jia, J., 2016b. Multi-focus image fusion based on cartoon-texture image decomposition. Optik - International Journal for Light and Electron Optics 127 (3), 1291 - 1296. https://doi.org/10.1016/j.ijleo.2015.10.098Zhang, Z., Blum, R., Aug 1999. A categorization of multiscale-decompositionbased image fusion schemes with a performance study for a digital camera application. Proceedings of the IEEE 87 (8), 1315-1326. https://doi.org/10.1109/5.775414Zhao, H., Li, Q., Feng, H., 2008. Multi-focus color image fusion in the HSI space using the sum-modified-laplacian and a coarse edge map. Image and Vision Computing 26 (9), 1285 - 1295. https://doi.org/10.1016/j.imavis.2008.03.007Zhou, L., Ji, G., Shi, C., Feng, C., Nian, R., 2006. A Multi-focus Image Fusion Method Based on Image Information Features and the Artificial Neural Networks. Vol. 344. Springer Berlin Heidelberg, Berlin, Heidelberg, pp. 747-752. https://doi.org/10.1007/978-3-540-37256-1_91Zhou, Z., Li, S., Wang, B., 2014. Multi-scale weighted gradient-based fusion for multi-focus images. Information Fusion 20 (0), 60 - 72. https://doi.org/10.1016/j.inffus.2013.11.00

Spatial Stimuli Gradient Based Multifocus Image Fusion Using Multiple Sized Kernels

Multi-focus image fusion technique extracts the focused areas from all the source images and combines them into a new image which contains all focused objects. This paper proposes a spatial domain fusion scheme for multi-focus images by using multiple size kernels. Firstly, source images are pre-processed with a contrast enhancement step and then the soft and hard decision maps are generated by employing a sliding window technique using multiple sized kernels on the gradient images. Hard decision map selects the accurate focus information from the source images, whereas, the soft decision map selects the basic focus information and contains minimum falsely detected focused/unfocused regions. These decision maps are further processed to compute the final focus map. Gradient images are constructed through state-ofthe-art edge detection technique, spatial stimuli gradient sketch model, which computes the local stimuli from perceived brightness and hence enhances the essential structural and edge information. Detailed experiment results demonstrate that the proposed multi-focus image fusion algorithm performs better than the other well known state-of-the-art multifocus image fusion methods, in terms of subjective visual perception and objective quality evaluation metrics

A quadtree driven image fusion quality assessment

In this paper a new method to compute saliency of source images is presented. This work is an extension to universal quality index founded by Wang and Bovik and improved by Piella. It defines the saliency according to the change of topology of quadratic tree decomposition between source images and the fused image. The saliency function provides higher weight for the tree nodes that differs more in the fused image in terms topology. Quadratic tree decomposition provides an easy and systematic way to add a saliency factor based on the segmented regions in the images. <br /

Geo-Adaptive Deep Spatio-Temporal predictive modeling for human mobility

Deep learning approaches for spatio-temporal prediction problems such as crowd-flow prediction assumes data to be of fixed and regular shaped tensor and face challenges of handling irregular, sparse data tensor. This poses limitations in use-case scenarios such as predicting visit counts of individuals' for a given spatial area at a particular temporal resolution using raster/image format representation of the geographical region, since the movement patterns of an individual can be largely restricted and localized to a certain part of the raster. Additionally, current deep-learning approaches for solving such problem doesn't account for the geographical awareness of a region while modelling the spatio-temporal movement patterns of an individual. To address these limitations, there is a need to develop a novel strategy and modeling approach that can handle both sparse, irregular data while incorporating geo-awareness in the model. In this paper, we make use of quadtree as the data structure for representing the image and introduce a novel geo-aware enabled deep learning layer, GA-ConvLSTM that performs the convolution operation based on a novel geo-aware module based on quadtree data structure for incorporating spatial dependencies while maintaining the recurrent mechanism for accounting for temporal dependencies. We present this approach in the context of the problem of predicting spatial behaviors of an individual (e.g., frequent visits to specific locations) through deep-learning based predictive model, GADST-Predict. Experimental results on two GPS based trace data shows that the proposed method is effective in handling frequency visits over different use-cases with considerable high accuracy

Fast and Robust Hand Tracking Using Detection-Guided Optimization

Markerless tracking of hands and fingers is a promising enabler for human-computer interaction. However, adoption has been limited because of tracking inaccuracies, incomplete coverage of motions, low framerate, complex camera setups, and high computational requirements. In this paper, we present a fast method for accurately tracking rapid and complex articulations of the hand using a single depth camera. Our algorithm uses a novel detection-guided optimization strategy that increases the robustness and speed of pose estimation. In the detection step, a randomized decision forest classifies pixels into parts of the hand. In the optimization step, a novel objective function combines the detected part labels and a Gaussian mixture representation of the depth to estimate a pose that best fits the depth. Our approach needs comparably less computational resources which makes it extremely fast (50 fps without GPU support). The approach also supports varying static, or moving, camera-to-scene arrangements. We show the benefits of our method by evaluating on public datasets and comparing against previous work

A Novel Multi-Focus Image Fusion Method Based on Stochastic Coordinate Coding and Local Density Peaks Clustering

abstract: The multi-focus image fusion method is used in image processing to generate all-focus images that have large depth of field (DOF) based on original multi-focus images. Different approaches have been used in the spatial and transform domain to fuse multi-focus images. As one of the most popular image processing methods, dictionary-learning-based spare representation achieves great performance in multi-focus image fusion. Most of the existing dictionary-learning-based multi-focus image fusion methods directly use the whole source images for dictionary learning. However, it incurs a high error rate and high computation cost in dictionary learning process by using the whole source images. This paper proposes a novel stochastic coordinate coding-based image fusion framework integrated with local density peaks. The proposed multi-focus image fusion method consists of three steps. First, source images are split into small image patches, then the split image patches are classified into a few groups by local density peaks clustering. Next, the grouped image patches are used for sub-dictionary learning by stochastic coordinate coding. The trained sub-dictionaries are combined into a dictionary for sparse representation. Finally, the simultaneous orthogonal matching pursuit (SOMP) algorithm is used to carry out sparse representation. After the three steps, the obtained sparse coefficients are fused following the max L1-norm rule. The fused coefficients are inversely transformed to an image by using the learned dictionary. The results and analyses of comparison experiments demonstrate that fused images of the proposed method have higher qualities than existing state-of-the-art methods

Reverse spatial visual top-k query

With the wide application of mobile Internet techniques an location-based services (LBS), massive multimedia data with geo-tags has been generated and collected. In this paper, we investigate a novel type of spatial query problem, named reverse spatial visual top- $k$ query (RSVQ k ) that aims to retrieve a set of geo-images that have the query as one of the most relevant geo-images in both geographical proximity and visual similarity. Existing approaches for reverse top- $k$ queries are not suitable to address this problem because they cannot effectively process unstructured data, such as image. To this end, firstly we propose the definition of RSVQ k problem and introduce the similarity measurement. A novel hybrid index, named VR 2 -Tree is designed, which is a combination of visual representation of geo-image and R-Tree. Besides, an extension of VR 2 -Tree, called CVR 2 -Tree is introduced and then we discuss the calculation of lower/upper bound, and then propose the optimization technique via CVR 2 -Tree for further pruning. In addition, a search algorithm named RSVQ k algorithm is developed to support the efficient RSVQ k query. Comprehensive experiments are conducted on four geo-image datasets, and the results illustrate that our approach can address the RSVQ k problem effectively and efficiently