Range Camera Self-Calibration Based on Integrated Bundle Adjustment via Joint Setup with a 2D Digital Camera

Time-of-flight cameras, based on Photonic Mixer Device (PMD) technology, are capable of measuring distances to objects at high frame rates, however, the measured ranges and the intensity data contain systematic errors that need to be corrected. In this paper, a new integrated range camera self-calibration method via joint setup with a digital (RGB) camera is presented. This method can simultaneously estimate the systematic range error parameters as well as the interior and external orientation parameters of the camera. The calibration approach is based on photogrammetric bundle adjustment of observation equations originating from collinearity condition and a range errors model. Addition of a digital camera to the calibration process overcomes the limitations of small field of view and low pixel resolution of the range camera. The tests are performed on a dataset captured by a PMD[vision]-O3 camera from a multi-resolution test field of high contrast targets. An average improvement of 83% in RMS of range error and 72% in RMS of coordinate residual, over that achieved with basic calibration, was realized in an independent accuracy assessment. Our proposed calibration method also achieved 25% and 36% improvement on RMS of range error and coordinate residual, respectively, over that obtained by integrated calibration of the single PMD camera

Accuracy Investigation for Structured-light Based Consumer 3D Sensors

This work focuses on the performance investigation of consumer 3D sensors with respect to their repeatability and accuracy. It explores currently available sensors based on the 3D sensing technology developed by PrimeSense and introduced to the market in the form of the Microsoft Kinect. Accuracy and repeatability can be crucial criteria for the use of these sensors outside their intended use for home entertainment. The test strategies for the study are motivated by the VDI/VDE 2634 guideline. At the core of the work is the investigation of several units of the Asus Xtion Pro and the PrimeSense Developer Kit and a comparison of their performance. Altogether eighteen sensor units were tested. The results of the proposed test scenario for the sensor units show excellent repeatability at a few millimetres. However, absolute accuracy is worse and can be up to a few centimetres. Sensor performance varies greatly both for sensors of the same manufacturer and in-between manufacturers

System Calibration between Non-metric Optical Camera and Range Camera

학위논문 (석사)-- 서울대학교 대학원 : 건설환경공학부, 2013. 8. 김용일.최근 실내 3차원 모델링은 다양한 분야에서 주목받고 있으며, 이에 따라 관련 연구에 대한 필요성이 증가하고 있다. 특히 이종 센서 영상간의 융합을 통한 실내 3차원 모델링 연구의 필요성이 대두되고 있다. 이때 이종 센서 영상간의 융합에 있어, 정밀한 시스템 캘리브레이션은 매우 필수적인 요소이다. 따라서 본 연구는 비측정용 광학 카메라와 레인지 카메라로 구성된 카메라 시스템에 대한 캘리브레이션을 수행하였다. 기존의 광학 카메라와 레인지 카메라간의 시스템 캘리브레이션에 대한 연구는 주로 특정 객체만을 3차원으로 구성하는 것에 연구가 집중되었다. 또한 시스템 캘리브레이션 방법 간의 비교 평가가 부족하였으며, 검정 대상지 설계가 미흡하였다는 한계가 존재한다. 이에 본 연구에서는 비측정용 광학 및 레인지 카메라의 특성을 고려하여 기존과 다른 검정 대상지를 새롭게 설계하였다. 또한 단사진 표정 및 블록 조정의 두 가지 방법을 통하여 시스템 캘리브레이션을 수행하여 상대표정요소를 도출하고, 그 결과에 대하여 비교 평가하였다. 결과적으로 상대표정요소간의 상관관계 및 표준편차를 낮추는 것이 정확도에 큰 영향을 주는 것을 확인하였으며, 블록 조정을 통하여 보다 신뢰도 높은 결과를 얻을 수 있음을 확인하였다. 본 연구에서는 이를 바탕으로 보다 효율적인 시스템 캘리브레이션 수행 방법 및 검정 대상지 설계와 영상 촬영 방법을 제안하였다.Recently, indoor 3D modeling has attracted attention in various fields, and the needs of its related research needs is increasing. Especially, indoor 3D modeling studies by using image fusion technique with different types of sensors are becoming a necessity. For a image fusion between two kinds of sensors, precise system calibration is essential. Therefore, system calibration was performed on the camera system consisting of non-metric optical camera and range camera in this study. Previous studies about system calibration between non-metric optical camera and range camera were mainly focused on constructing certain object in 3d model. And previous test-bed design was not sufficient for system calibration. In this study, test-bed for calibration was designed by considering the characteristics of non-metric optical camera and range camera. Also, relative orientation parameters were derived by performing a system calibration using single photo resection and block adjustment. As a result, it was confirmed that it is important to reduce correlation between relative orientation parameters and standard deviation to obtain result with high accuracy. Also, it was confirmed that through block-adjustment method to get more reliable results. Finally, a efficient way to perform system calibration, test-bed design and image shooting methods were proposed.1. 서론 1 1.1. 연구배경 및 동기 1 1.2. 연구동향 3 1.3. 연구의 목적 및 범위 5 2. 시스템 캘리브레이션 수학적 모델 7 2.1. 광학 카메라 영상의 수학적 모델식 7 2.2. 레인지카메라 영상의 수학적 모델식 8 2.2.1. 거리 관측값 8 2.2.2. 정오차(systematic error) 8 2.3. 단사진표정의 수학적 모델식 10 2.4. 블록조정에서의 수학적 모델식 13 3. 카메라의 특징 및 검정 대상지 구성 14 3.1. 광학 및 레인지 카메라의 제원 및 특징 14 3.1.1. 광학카메라 14 3.1.2. 레인지 카메라 16 3.2. 카메라 및 검정 대상지 좌표계 설정 19 3.3. 검정 대상지 구성 20 3.3.1. 광학카메라용 검정 대상지 구성 20 3.3.2. 레인지카메라용 검정 대상지 구성 21 4. 시스템 캘리브레이션 시뮬레이션 25 4.1. 시뮬레이션 진행 설계 및 순서 25 4.2. 시뮬레이션 검정 대상지 및 표정요소 결정 27 4.2.1. 검정 대상지 및 지상 기준점 설계 27 4.2.2. 내부표정요소 29 4.2.3. 외부표정요소 및 상대표정요소의 결정 31 4.3. 시뮬레이션 영상 제작 36 4.3.1. 광학 카메라 시뮬레이션 영상 제작 36 4.3.2. 레인지 카메라 시뮬레이션 영상 제작 39 4.4. 시뮬레이션 시스템 캘리브레이션 수행 43 4.4.1. 단사진 표정 시뮬레이션 43 4.4.2. 블록조정 시뮬레이션 45 5. 시스템 캘리브레이션 46 5.1. 시스템 캘리브레이션 진행 설계 및 순서 46 5.2. 검정 대상지 구현 47 5.2.1. 검정 대상지 프레임 47 5.2.2. 광학 카메라용 지상 기준점 48 5.2.3. 레인지 카메라용 지상 기준점 50 5.2.4. 지상 기준점 좌표 측정 55 5.3. 실영상 촬영 56 5.3.1. 카메라간의 위치 관계 56 5.3.2. 실영상 촬영 57 5.4. 시스템 캘리브레이션 수행 60 5.4.1. 내부표정요소 도출 60 5.4.2. 단사진 표정 61 5.4.3. 블록조정 61 6. 캘리브레이션 결과 및 평가 62 6.1. 시뮬레이션 캘리브레이션 결과 62 6.1.1. 단사진 표정을 통한 시스템 캘리브레이션 시뮬레이션 62 6.1.2. 블록 조정을 통한 시스템 캘리브레이션 75 6.1.3. 시뮬레이션 단사진 표정과 블록 조정 결과 비교 81 6.2. 실제 영상을 이용한 시스템 캘리브레이션 결과 83 6.2.1. 단사진 표정을 통한 시스템 캘리브레이션 83 6.2.2. 블록 조정을 통한 시스템 캘리브레이션 93 6.2.3. 단사진 표정 및 블록 조정 결과 비교 99 6.3. 시뮬레이션과 실제 시스템 캘리브레이션 비교 평가 101 7. 결론 103 참고문헌 107 부 록 118 A.1. 최소제곱법 118 A.2. 축차근사법 120 A.3. 공선조건식(collinearity condition) 121 A.4. 광학카메라 단사진 표정 모델식 및 행렬구성 125 A.5. 레인지카메라 단사진 표정 모델식 131 A.6. 블록조정 수학적 모델 135Maste

A Comparison of Three Geometric Self-Calibration Methods for Range Cameras

Significant instrumental systematic errors are known to exist in data captured with range cameras using lock-in pixel technology. Because they are independent of the imaged object scene structure, these errors can be rigorously estimated in a self-calibrating bundle adjustment procedure. This paper presents a review and a quantitative comparison of three methods for range camera self-calibration in order to determine which, if any, is superior. Two different SwissRanger range cameras have been calibrated using each method. Though differences of up to 2 mm (in object space) in both the observation precision and accuracy measures exist between the methods, they are of little practical consequence when compared to the magnitude of these measures (12 mm to 18 mm). One of the methods was found to underestimate the principal distance but overestimate the rangefinder offset in comparison to the other two methods whose estimates agreed more closely. Strong correlations among the rangefinder offset, periodic error terms and the camera position co-ordinates are indentified and their cause explained in terms of network geometry and observation range

A Comparison of Three Geometric Self-Calibration Methods for Range Cameras

Significant instrumental systematic errors are known to exist in data captured with range cameras using lock-in pixel technology. Because they are independent of the imaged object scene structure, these errors can be rigorously estimated in a self-calibrating bundle adjustment procedure. This paper presents a review and a quantitative comparison of three methods for range camera self-calibration in order to determine which, if any, is superior. Two different SwissRanger range cameras have been calibrated using each method. Though differences of up to 2 mm (in object space) in both the observation precision and accuracy measures exist between the methods, they are of little practical consequence when compared to the magnitude of these measures (12 mm to 18 mm). One of the methods was found to underestimate the principal distance but overestimate the rangefinder offset in comparison to the other two methods whose estimates agreed more closely. Strong correlations among the rangefinder offset, periodic error terms and the camera position co-ordinates are indentified and their cause explained in terms of network geometry and observation range

ToF Camera calibration: an automatic setting of its integration time and an experimental analysis of its modulation frequency

[ES] La percepción de profundidad se hace imprescindible en muchas tareas de manipulación, control visual y navegación de robots. Las cámaras de tiempo de vuelo (ToF: Time of Flight) generan imágenes de rango que proporcionan medidas de profundidad en tiempo real. No obstante, el parámetro distancia que calculan estas cámaras es fuertemente dependiente del tiempo de integración que se configura en el sensor y de la frecuencia de modulación empleada por el sistema de iluminación que integran. En este artículo, se presenta una metodología para el ajuste adaptativo del tiempo de integración y un análisis experimental del comportamiento de una cámara ToF cuando se modifica la frecuencia de modulación. Este método ha sido probado con éxito en algoritmos de control visual con arquitectura ‘eye-in-hand’ donde el sistema sensorial está compuesto por una cámara ToF. Además, la misma metodología puede ser aplicada en otros escenarios de trabajo.[EN] The depth perception is essential in many manipulation tasks, visual inspection and robot navigation. The cameras of Time of Flight (TOF) generate range images which provide depth measurements in real time. However, the distance parameter computed from these cameras is strongly dependent on the integration time set for the sensor and the frequency of modulation used by the integrated lighting system. In this paper, a methodology for automatic setting of integration time and an experimental analysis of ToF camera behavior adjusting its modulation frequency is presented. This method has been successfully tested on visual servoing algorithms with architecture ‘eye-in-hand’ in which the sensory system consists of a ToF camera, in addition this methodology can be applied to other workspaces and scenarios.Este trabajo ha sido co-financiado por el Gobierno regional de la Generalitat Valenciana, Universidad de Alicante y CICYT través de los proyectos GV2012/102, GRE10-16 y DPI2012-32390.Gil, P.; Kisler, T.; García, G.; Jara, C.; Corrales, J. (2013). Calibración de cámaras de tiempo de vuelo: Ajuste adaptativo del tiempo de integración y análisis de la frecuencia de modulación. Revista Iberoamericana de Automática e Informática industrial. 10(4):453-464. https://doi.org/10.1016/j.riai.2013.08.002OJS453464104Bouguet, J.Y., 2000. Pyramidal implementation of affine Lucas Kanade feature tracker. Intel Corporation- Microprocessor Research Labs, OpenCV Library.Chaumette, F., Hutchinson, S., 2006. Visual servo control. I. Basic approaches. IEEE Robotics and Automation Magazine 13, IEEE Press, pp. 82-90.Chiabrando, F., Chiabrando, R., Piatti, D., Rianudo, F., 2009. Sensors for 3d imaging: metric evualuation an calibration of CCD/CMOS time-of-flight camera. Sensors 9(9), pp. 10080-10096. DOI: 10.3390/s91210080.Distante, C., Diraco, G., Leone, A., 2010. Active range imaging dataset for indoor surveillance. In Proc. of British Machine Vision Conference (BMVC), BMVA Press, vol. 2, pp.1-16.Foix, S., Alenya, G., & Torras, C. (2011). Lock-in Time-of-Flight (ToF) Cameras: A Survey. IEEE Sensors Journal, 11(9), 1917-1926. doi:10.1109/jsen.2010.2101060Plaue, M. (2009). Theoretical and experimental error analysis of continuous-wave time-of-flight range cameras. Optical Engineering, 48(1), 013602. doi:10.1117/1.3070634Fuchs, S. Hirzinger, G., 2008. Extrinsic and depth calibration of ToF-cameras. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Press Society, pp. 1-6. DOI: 10.1109/CVPR.2008.4587828.Garcia, F., Aouada D., Mirbach, B., Solignac T., Ottersten B., 2011. Real-time hybrod ToF multi-camera rig fusion system for depth map enhancement. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Press Society, pp. 1-8. DOI: 10.1109/CVPRW.2011.5981740.Gil, P., Pomares, J., Torres, F., 2010. Analysis and adaptation of integration time in PMD camera for visual servoing. In Proc. of 20th International Conference on Pattern Recognition (ICPR), IEEE Press Society, pp. 311-315. DOI: 10.1109/ICPR.2010.85.Herrera, D.C., Kannala, J., Heikkila, J., 2011. Accurate and practical calibration of a depth and color camera pair. In Proc. of 14th International Conference on Computer Analysis of Images and Patterns (CAIP), vol 2, Ed. Springer-Verlag Berlín, Heidelberg, pp. 437-445.Hussman, S., Liepert, T., 2009. Three-dimensional tof robot vision system. IEEE Transactions on Instrumentation and Measurement 58(1), pp. 141-146. DOI: 10.1109/TIM.2008.928409.Hussman, S., Edeler, T., 2010. Robot vision using 3d tof systems. En: Ales Ude, (Ed.), Robot Vision. Intech Press, pp. 293-306.Kakiuchi, Y., Ueda, R., Kobayashi, K., Okada, K., Inaba, M., 2010. Working with movable obstacles using on-line environmet perception reconstruction using active sensing and color range sensor. In Proc. of International Conference on Intelligent Robots and Systems (IROS), IEEE Press, pp. 1696-1701. DOI: 10.1109/IROS.2010.5650206.Kim, Y.M., Chan, D., Theobalt, C., Thrun, S., 2008. Design and calibration of a multi-view ToF sensor fusion system. In Proc. of 22nd Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Press Society, pp. 1524-1530. DOI: 10.1109/CVPRW.2008.4563160.Kisler, T., Gil, P., 2011. Detección y seguimiento de objetos sólidos con cámaras ToF. Actas de XXXII Jornadas de Automática (JA), CEA-IFAC Actas. Sevilla (Spain).Khoshelham, K. 2011. Accuracy analysis of Kinect depth data. En: D.D., Lichti, A.F., Habbib, (Ed.). In Proc of ISPRS Journal of Photogrammetry and Remote Sensing-Workshop on Laser Scanning, vol. 38(5), pp. 29-31.Kolb, A., Barth, E., Koch, E., Larse, R., 2010. Time-of-flight Cameras in Computer Graphics. Computer Graphics Forum, vol. 29(1), pp. 141-159. DOI: 10.1111/j.1467-8659.2009.01583.x.Kuehnle, J.U., Xue, Z., Sotz, M., Zoellner, J.M., Verl, A., Dillmann, R., 2008. Grasping in depth maps of time-of-flight cameras. In Proc. of IEEE International Workshop on Robotic and Sensors Environments (ROSE). pp. 132-137. DOI: 10.1109/ROSE.2008.466914.Lai, K., Liefeng Bo, Xiaofrng Ren, Fox, D., 2011. Spares distance learning for object recognition combining RGB and depth information. In Proc. of International Conference on Robotics and Automation (ICRA), IEEE Press Society, pp. 4007-4013. DOI: 10.1109/ICRA.2011.5980377.Lichti, D., 2008. Self-calibration of a 3D range camera. In Proc of International Society for Photogrammetry and Remote Sensing 37(3), pp.1-6.Lichti, D. Rouzaud, D., 2009. Surface-dependent 3d range camera self- calibration. En: A., Beraldin, G.S., Cheok, M., McCarthy, (Ed.), In Proc. of SPIE vol. 72390, pp. DOI: 10.1117/12.805509.Lichti, D., Kim, C., 2011. A comparison of three geometric self-calibration methods for range cameras. Remote Sensing 11(3), pp. 1014-1028. DOI: 10.3390/rs3051014.Lindner, M., Kolb, A., Ringbeck, T., 2008. New insights into the calibration of ToF-sensors. In Proc. of 22nd Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Press Society, pp. 1603-1607. DOI: 10.1109/CVPRW.2008.4563172.Lindner, M., Schiller, I., Kolb, A., & Koch, R. (2010). Time-of-Flight sensor calibration for accurate range sensing. Computer Vision and Image Understanding, 114(12), 1318-1328. doi:10.1016/j.cviu.2009.11.002May, S., Werner, B., Surmann, H., Pervölz, K., 2006. “3d time-of-flight cameras for mobile robotics. In Proc. of International Conference on Intelligent Robots and Systems (IROS), IEEE Press, 790-795, DOI: 10.1109/IROS.2006.281670.May, S., Fuchs, S., Droeschel, D. Holz, D., Nüchter, A., 2009. Robust 3d-mapping with time-of-flight cameras. In Proc. of International Conference on Intelligent Robots and Systems (IROS), IEEE Press Society, pp 1673-1678.May, S., Droeschel, D., Holz, D., Fuchs, S., Malis, E., Nüchter, A., Hertzberg, J., 2009. Three-dimensional mapping with time-of-light cameras. Journal of Field Robotics. Special Issue on Three-dimensional Mapping Part 2, 26(11-12), pp. 934-965. DOI: 10.1002/ROB.20321.Mufti, F., & Mahony, R. (2011). Statistical analysis of signal measurement in time-of-flight cameras. ISPRS Journal of Photogrammetry and Remote Sensing, 66(5), 720-731. doi:10.1016/j.isprsjprs.2011.06.004Pattison, T., 2010. Quantification and description of distance measurement errors of a time-of-flight camera. M. Sc. Thesis. University of Stuttgart, Stuttgart (Germany).Pomares, J., Gil, P., Torres, F., 2010. Visual control of robots using range images. Sensors 10(8), pp. 7303-7322. DOI: 10.3290/s100807303.Rapp, H., Frank, M., Hamprecht, F.A., Jähne, B., 2008. A theoretical and experimental investigation of the systematic errors and statistical uncertainties of time-of-flight-cameras. International Journal of Intelligent Systems Technologies and Applications vol. 5(3-4), pp. 402-413. DOI: 10.1504/IJISTA.2008.021303.Shahbazi, M., Homayouni, S., Saadatseresht, M., & Sattari, M. (2011). Range Camera Self-Calibration Based on Integrated Bundle Adjustment via Joint Setup with a 2D Digital Camera. Sensors, 11(9), 8721-8740. doi:10.3390/s110908721Schaller, C., 2011, Time-of-Flight-A new Modality for Radiotherapy, M. Sc. Thesis. University Erlangen-Nuremberg, Erlagen (Germany).Schiller, I., Beder, C., Koch, R., 2008. Calibration of a PMD-camera using a planar calibration pattern together with a multi-camera setup. In Proc. of ISPRS Journal of Photogrammetry and Remote Sensing vol. 37, pp. 297-302.Schwarz, L., Mateus, D., Castaneda, V., Navab, N., 2010. Manifold learning for ToF-based human body tracking and activity recognition. In Proc. of British Machine Vision Conference (BMVC), BMVA Press, pp.1-11. DOI: 10.5244/C.24.80.Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., Blake, A., 2011. Real-time human pose recognition in parts from single depth images. In Proc of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE Press Society, pp. 1297-1304.Smisek, J., 2011. 3D Camera Calibration. MSc. Thesis. Czech Technnical Univesity, Prague (Czech).Weyer, C.A., Bae, K.H., Lim, K., Lichti, D., 2008. Extensive metric performance evaluation of a 3D range camera. In Proc. of ISPRS Journal of Photogrammetry and Remote Sensing vol.37(5), pp.939-944.Wiedemann M., Sauer M., Driewer F. Schilling K., 2008. Analysis and characterization of the PMD camera for aplication in mobile robots. M. J. Chung and P. Misra (Ed.). In Proc. of 17th World Congress of the International Federation of Automotic Control, IFAC Press, pp.13689-13694.Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11), 1330-1334. doi:10.1109/34.888718Zhu, J., Wang, L., Yang, R., Davis, J., 2008. Fusion of time-of-flight depth and stereo for high accuracy depth maps. In Proc. of Computer Vision and Pattern Recognition (CVPR), IEEE Press Society, pp. 1-8. DOI: 10.1109/CVPR.2008.4587761.Zhu, J., Yang, R., & Xiang, X. (2011). Eye contact in video conference via fusion of time-of-flight depth sensor and stereo. 3D Research, 2(3). doi:10.1007/3dres.03(2011)5Zinber, T., Schmidt, J., Niemann, H., 2003. A refined ICP algorithm for robust 3d correspondence estimation. In Proc. of Conference on Image Processing (ICIP), IEEE Press, pp. 695-698