This paper proposes a novel approach to recovering depth from defocus, which is a deterministic approach in spatial domain. Two defocused gray images from the same scene are obtained by changing two parameters (image distance and focal length of camera) other than only parameter (image distance). The idea of this approach is to convert the gray images into the gradient images by Canny operator other than Sobel operator, then calculate the ratio of the area of region with large gradient value to that of the whole image region in each block for each defocused image by moment-preserving method, and recover depth from scene according to the ratio of the ratio of one gradient image to that of the other gradient image. The experimental results show that the proposed approach is more accurate and efficient than the traditional approach

Fu Lifang

He Zhipan

Liu Zhenzhong

Wu Qiufeng

Directory of Open Access Journals

Sensors & Transducers, Vol. 160, Issue 12, December 2013, pp. 36-41  36   Sensors & Transducers© 2013 by IFSA http://www.sensorsportal.com      A Novel Approach to Recovering Depth from Defocus  1 He Zhipan, 2 Liu Zhenzhong, 2 Wu Qiufeng and 2 Fu Lifang 1 College of Engineering, Northeast Agricultural University, 150030, Harbin, China 2 College of Science, Northeast Agricultural University, 150030, Harbin, China 1 Tel.: 086+18845572179,  E-mail: hezhipan19880103@163.com   Received: 18 September 2013   /Accepted: 22 November 2013   /Published: 30 December 2013   Abstract: This paper proposes a novel approach to recovering depth from defocus, which is a deterministic approach in spatial domain. Two defocused gray images from the same scene are obtained by changing two parameters (image distance and focal length of camera) other than only parameter (image distance). The idea of this approach is to convert the gray images into the gradient images by Canny operator other than Sobel operator, then calculate the ratio of the area of region with large gradient value to that of the whole image region in each block for each defocused image by moment-preserving method, and recover depth from scene according to the ratio of the ratio of one gradient image to that of the other gradient image. The experimental results show that the proposed approach is more accurate and efficient than the traditional approach. Copyright © 2013 IFSA.  Keywords: Depth from defocus, Canny operator, Gradient images, The ratio, Moment-preserving method.    1. Introduction  Computer vision is to use computer to realize people’s visual function that people can find 3D structure from 2D known images to identify 3D world. In computer vision, 3D reconstruction of scene is an important content, while the key of 3D reconstruction is to calculate the distance between each point in scene and camera, which is also called depth recovery. There are many approaches to estimating depth of the scene, including Structure from Motion (SFM) [1], Depth from Stereo (DFS) [2], and Depth from Focus (DFF) [3], etc. But in DFS, correspondence problem need be solved well; in SFM and DFF, many images are needed, what’s more, requirement for measurement accuracy is higher, the measurement process is more complex. Therefore, this paper focuses on depth from defocus (DFD) [4-6], which can avoid correspondence problem [7], high requirement and complexity. DFD is to recover depth of the scene from two defocused gray images of the same scene at a single viewpoint. In 1987, DFD was firstly proposed by Pentland [4]. After that, many approaches were proposed to solve DFD. Nowadays, these approaches are divided into two classes: statistical approaches  [8-9] and deterministic approaches [6]. Deterministic approaches can be divided into frequency domain approaches [10-12] and spatial domain approaches [6] again. But, statistical approaches have high computational cost. Furthermore, deterministic approaches in frequency domain can be inaccurate owing to window effect, edge and noise etc. By contrast, deterministic approaches in spatial domain are both simple and real-time. Therefore, this paper focuses on deterministic approach in spatial domain. Many deterministic approaches in spatial domain are to deduce deterministic relationships between depth and some parameters (e.g., the spread parameter [13], deformation parameter [14], etc.). As long as these parameters are estimated, the depth can Article number P_1574Sensors & Transducers, Vol. 160, Issue 12, December 2013, pp. 36-41  37be obtained. This paper calculates the ratio of the area of region with large gradient value to that of the whole image region in each block for each defocused image by moment-preserving method to recover depth from scene [15]. In [15-16], converting the gray images into the gradient images leads to wide edges by Sobel operator. However, this paper uses Canny operator to avoid the above shortcomings. This can obtain more accurate gradient images which can improve the depth recovery effectively. Furthermore, in some papers, the relationships between depth and the ratios are obtained by changing only parameter (image distance [15]); however, this paper generalizes the above formulas, and not only can change one parameter, but also can change more parameters. This paper is discussed by changing two parameters (image distance and focal length of camera).  This paper proposes a novel deterministic approach to recovering depth from defocus according to the above improvements. In Section 2 the deterministic depth from defocus approach is proposed. Section 2.1 elaborates the relationships between depth and the ratio parameter. Section 2.2 presents Canny operator and moment-preserving method which are used to calculate the ratio parameter. The steps of depth estimation are shown in Section 2.3. Experimental results with synthetic defocused images are carried out in Section 3.    2. Method  2.1. Deterministic Relationship Between Depth and Ratio Parameter β  In Fig. 1, according to the basic imaging principle, when the camera is focused, object distance D , focal length lF , and image distance v , there are following geometric relationships:     Fig. 1. Schematic diagram of lens imaging    1 1 1lD v F  , (1)  While, when the camera is defocused, the point on the object becomes a fuzzy disk, radius of the disk (defocused radius) is br . Defocused radius br , radius of the lens 0r , and actual image distance 0v  have the following geometric relationships:   0 001 1 1blr v rF D v     , (2)  So the formula of  depth from scene is:   0 00 0 0ll b lv r FDv r F r r F , (3)  Two defocused gray images from the same scene are obtained by changing two parameters (image distance and focal length of camera), and image distance and object distance of the first defocused image respectively are 01v  and 1lF , and that of the second one respectively are 02v  and 2lF . According to (3), depth is represented as the following:   01 0 1 02 0 201 0 1 0 1 1 02 0 2 0 2 2l ll l l lv r F v r FDv r F r r F v r F r r F     (4)  where 1r is the defocused radius of the first defocused image, 2r  is that of the second one. According to (4), the expression of 0r  is:   01 2 02 1 1 2001 02 1 2 02 01 1 2( )( ) ( )l ll l l lv r v r F Frv v F F v v F F  , (5)  Plug the expression of 0r in (4), this paper gets deterministic relationship between depth and  :   01 02 1 201 1 2 02 2 1( )( ) ( )l ll l l lv v F FDv F F v F F  , (6)  where 12rr  .  2.2. Canny Operator and Moment-Preserving Method  By equation (6), D can be known after calculating  , and 12rr  , but solving 1r and 2r  directly is very difficult, this paper uses moment-preserving method to solve  . This paper gets two gray image    1 2, , ,f x y f x y with different defocused radiuses by changing 0v  and lF . Converting the gray Sensors & Transducers, Vol. 160, Issue 12, December 2013, pp. 36-41  38 images into the gradient images leads to wide edges by Sobel operator. So this paper uses Canny operator to obtain more accurate gradient images which can improve the depth recovery effectively. The gradient images of images    1 2, , ,f x y f x y  respectively are:     2 21 1 1, x yg x y g g  , (7)     2 22 2 2, x yg x y g g  , (8)  where   ( , ) ( , ),( , ) ( , ),1,2; 1,2.mx m xj iny n yj ig f x i y j w i jg f x i y j w i jm n        , (9)  Level Canny operator   1 11( , ) , 1 , 1,1 12xw i j i j      , (10)  Vertical Canny operator   1 11( , ) , 1 , 1.1 12yw i j i j       , (11)  The gradient image is divided into several blocks, this paper assumes that multiple pixels have the same depth values in the same area, and in each block, the bright area represents the edges, the dark area represents the non-edges. Each block of the scene area, the gray level of the edges of image with larger gradient values changes drastically, the gray level of the non-edges of image with smaller gradient values changes not drastically. Assuming that the gray values of bright area and dark area in the gradient image ( , )g x y respectively are ,e bz z ; the ratio of the size of bright area (dark area) to that of the whole image area is ( )e bq q . For focused image, the value of eq is smaller, for defocused image, the value of eq  is larger. As the image from focused to defocused, the value of eq changes from small to big, so we can think that the defocused radius of the image is proportional to eq , that is   er k q  , (12)  In (12), k  is the constant. Define the first three moments of the image  1 ,g x y  are:   1( , ) ( , )1[ ( , )] , 1, 2,3jjx y N x ym g x y jn   , (13) where ( , )N x y  as the image area, and n  as the pixel numbers of image region, using moment-preserving method, it can get the following four equations:    1 112 223 331e be e b be e b be e b bq qq z q z mq z q z mq z q z m     , (14)  solution for (14)     221421421e b b eb b b ee bee bz c c cz c c cz mqz z         , (15)  where   22 1 322 13 1 222 1ebm m mcm mm m mcm m      ,    In image 2 ( , )g x y , The ratio of the size of bright area (dark area) to that of the whole image area is ( )e bw w . As a result,    12e ee ek q qrr k w w  , (16)   2.3. Depth Estimation  In this paper, the steps of depth from scene can be simply summarized as follows: Step1: Two defocused gray images from the same scene are obtained by changing two parameters (image distance and focal length of camera). Shorthand for respectively    1 2, , ,f x y f x y . Step2: According to (7), (8), (9), (10), (11), the gray images can be converted to gradient images, shorthand for respectively    1 2, , ,yg x g x y . Step3: The gradient images    1 2, , ,yg x g x y  are divided into many blocks with same size,  among them, every block of  1 ,g x y and  2 ,g x y  is one-to-one, according to (12), (13), (14), (15), (16), it uses moment-preserving method to calculate the values 1 2( , ,..., )n     of every block. Step4: Plug the values 1 2( , ,..., )n    of   every block in  (6), it can calculate the depth values 1 2( , ,..., )nD D D D  of every block of the scene. Sensors & Transducers, Vol. 160, Issue 12, December 2013, pp. 36-41  393. Experimental Results  3.1. Depth recovery Results  Two images with two steps are adopted in the experiment, the size are both 400 400 , the distance from the front of the scene to lens is 400 mm, the distance from the back of the scene to lens is  1000 mm. When the zoom lens focus to 1.3 m, focal distance of camera is 34 mm, and the zoom lens focus to 1.6 m, focal distance of camera is 35 mm respectively, two original defocused gray images are gotten as shown in Fig. 2. Then, two gray images are converted to two gradient images. Last, the two gradient images are divided into several sub-images with the same size, the size of sub-image is10 10 . According to one-to-one sub-images of the two images, we can calculate depth from scene. Among them, Fig. 3 are the gradient images of the two gray images after using Canny operator, and Fig. 4 are the recovered depth map and depth surface by the approach of this paper (change the camera two parameters, Canny operator); When the zoom lens focus to 1.3 m and 1.6 m respectively, we can get the other two original defocused gray images as shown in Fig. 5. Fig. 6 are the gradient images of the two gray images after using Sobel operator, and Fig. 7 are the recovered depth map and depth surface by the traditional approach (change the camera only one parameter, Sobel operator).     (a)    (b)  Fig. 2. Two original defocused gray images: (a) the original defocused gray image when focal distance of camera is 34 mm and camera focuses on 1.3 m, (b) the original defocused gray image when focal distance of camera is 35 mm and camera focuses on 1.6 m.   (a)    (b)  Fig. 3. The two gradient images: (a) gradient image of handling Fig. 2a by Canny operator, (b) gradient image  of handling Fig. 2b by Canny operator.     (a)    (b)  Fig. 4. The recovered depth map and depth surface by the approach of this paper (change the camera two parameters, Canny operator): (a) recovered depth map by the approach of this paper, (b) recovered depth surface by the approach of this paper. Sensors & Transducers, Vol. 160, Issue 12, December 2013, pp. 36-41  40    (a)  (b)  Fig. 5. The other two original defocused gray images: (a) the original defocused gray image when camera focuses on 1.3m, (b) the original defocused gray image when camera focuses on 1.6 m.     (a)  (b)  Fig. 6. The other two gradient images: (a) gradient image of handling Fig. 5a by Sobel operator, (b) gradient image  of handling Fig. 5b by Sobel operator.   (a)  (b)  Fig. 7. The recovered depth map and depth surface by the traditional approach (change the camera only one parameter, Sobel operator): (a) recovered depth map by the traditional approach, (b) recovered depth surface by the traditional approach.   3.2. Error Analysis  Error can be measured in three indicators: the average (Means), standard deviation (Std), and mean square error (RMS). Their formulas are as follows:   121211 ˆ ,1 ˆ( ) ,1 ˆ( )nkkkknknkDD DMeansnStdnRMS DnD   (17) where ˆ kD  is the measured value, D is the measured average value, D is the real value, n is the pixel numbers in an area.  Each error value of measuring depth of using the approach of this paper (change the camera two parameters at the same time, solve gradient images by Canny operator) and using the traditional approach (change the camera only one parameter, solve gradient images using Sobel operator) as shown in Table 1 and Table 2. From Table 1 and Table 2, it can be seen that the depth measurement results by the approach of this paper are better than the depth measurement results by the traditional approach, the average value is close to the true value, and its errors are smaller. Sensors & Transducers, Vol. 160, Issue 12, December 2013, pp. 36-41  41Table 1. The errors of the approach of this paper (change the camera two parameters, Canny operator).  Depth (mm) The values of three indicators  Means Std RMS 400 398.89 0.19 1.13 1000 1016.40 0.66 16.43   Table 2. The errors of the traditional approach (change the camera only one parameter, Sobel operator).  Depth (mm) The values of three indicators Means Std RMS 400 384.22 0.64 15.79 1000 741.11 0.00 258.89   4. Conclusions  This paper proposes a novel approach to recovering depth from defocus, the approach is spatial domain deterministic approach, two defocused gray images from the same scene are obtained by changing two parameters (image distance and focal length of camera), and converts the gray images into the gradient images by Canny operator, and uses moment-preserving method to recover the depth from the scene. Under the premise of guarantee of accuracy, it is simple.  The experimental results show that the approach is accurate and efficient, and it also can be seen that the approach is also feasible for the complex scene. The approach recovers the depth from the scene based on the value eq of gradient image, it is hardly affected by illumination change, but it requests that target scene must have evident clear edges or textures.    Acknowledgments  This work was supported in part by the Nature Science Foundation of Northeast Agricultural University under contract No.2011RCA01, and Science and Technology Foundation of Education Department in Heilongjiang province under the contract No. 11551037.   References  [1]. J. Civera, A. J. Davison, J. M. Martinez Montiel, Structure from Motion using the Extended Kalman Filter, Springer Tracts in Advanced Robotics,  Vol. 75, 2012. [2]. M. Bleyer, Ch. Rhemann, and C. Rother, Extracting 3D Scene-Consistent Object Proposals and Depth from Stereo Images, Lecture Notes in Computer Science, Vol. 7576, 2012, pp. 467-481. [3]. K. B. Gyung, and G. T. Tian, A novel depth-from focus-based measurement system for the reconstruction of surface morphology with depth discontinuity, International Journal of Advanced Manufacturing Technology, February 2009, Vol. 40, Issue 11-12, pp. 1158-1165. [4]. A. P. Pentland, A new sense for depth of field, IEEE Trans on Pattern Analysis and Machine Intelligence, 9, 4, 1987, pp. 523-531. [5]. M. Subbarao, Parallel depth recovery by changing camera parameters, in Proceedings of the IEEE International Conference on Computer Vision, Florida, 1988, pp. 143-155. [6]. M. Sunnarao, G. Surya, Depth from defocus: A spatial domain approach, International Journal of Computer Vision, 13, 3, 1994, pp. 271-294. [7].  Y. Y. Schechner, N. Kiryati, Depth from defocus vs. stereo: how different really are they, Int. J. Comput. Vis, Vol. 39, 2000, pp. 141-162. [8]. A. N. Rajagopalan, S. Chaudhuri, An MRF model-based approach to simultaneous recovery of depth and restoration from defocused images, IEEE Trans. Pattern Anal. Mach. Intell., Vol. 21, 1999,  pp. 577-589. [9].  A. N. Rajagopalan, S. Chaudhuri, Optimal recovery   of depth from defocused images using an MRF model, in Proceedings of IEEE Conference on Computer Vision Institute of Electrical and Electronics Engineers, New York, 1998,  pp. 1047-1052. [10].  M. Gokstorp, Computing depth from out-of-focus blur using a local frequency representation, in Proceedings of IEEE Conference on Pattern Recognition, Institute of Electrical and Electronics Engineers, New York, 1994, pp. 153-158. [11]. M. Watanabe, S. K. Nayar, Rational filters for passive depth from defocus, Int .J. Comput. Vis, Vol.27, 1998, pp. 203-225. [12]. A. N. Joseph Raj, R. C. Staunton, Rational filter design for depth from defocus, Pattern Recognit., 45, 2012, pp. 198-207. [13]. G. Surya, M. Subbarao, Depth from defocus by changing camera aperture: a spatial domain approach, in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR '93), 1993, pp. 61-67. [14]. Ch. Hong, Z. Quanbing, G. Yanyan, A New Depth Recovery Algorithm Based on Defocus Image, Computer Applications and Software, Vol. 27 No. 2, Feb. 2010, pp. 271-273. [15]. T. Tian, J. Pan, Depth Estimation from Defocus Based on Moment-Preserving, Journal of Shanghai Jiaotong University, Vol. 34 No. 7, Jul. 2000,  pp. 917-920. [16]. T. Du-Ming, L. Chin-Tun, A moment-preserving approach for depth from defocus, Pattern Recognition, 31, 5, 1998, pp. 551-560.  ___________________   2013 Copyright ©, International Frequency Sensor Association (IFSA). All rights reserved. (http://www.sensorsportal.com)  

A Novel Approach to Recovering Depth from Defocus

http://www.sensorsportal.com/HTML/DIGEST/december_2013/PDF_vol_160/P_1574.pdf

A Novel Approach to Recovering Depth from Defocus

Abstract

Similar works

Full text

Available Versions

Directory of Open Access Journals