To reconstruct three-dimensional (3D) structure of objects and measure their geometric parameters using stereoscopic imaging systems, it is necessary to implement a number of image processing algorithms. For higher system efficiency, the choice of these algorithms and mathematical models should be taken into account at the stage of optical system design. We demonstrate the capabilities of optical design software to perform computer simulation of geometrical calibration for stereoscopic systems. The simulation allows comparison of mathematical models used for 3D reconstruction and estimation of 3D measurements errors caused by tolerances of optical elements, temperature variations and other factors. Using this technique, we analyze the design of prism-based stereoscopic system and show that the proposed ray tracing camera model considering pupil aberrations provides higher measurement precision. The results of computer simulation are confirmed by experiments with the self-developed stereoscopic system

Batshev, V.I.

Gorevoy, A.V.

Koluchkin, V.Ya.

Machikhin, A.S.

Samara University

Компьютерная оптика и нанофотоника V Международная конференция и молодёжная школа «Информационные технологии и нанотехнологии» (ИТНТ-2019) Calibration simulation for stereoscopic optical systems using optical design software A.V. Gorevoy1,2, A.S. Machikhin1,2, V.I. Batshev1,3, V.Ya. Koluchkin3 1Scientific and Technological Center of Unique Instrumentation RAS, Butlerova street 15, Moscow, Russia, 117342 2National Research University «Moscow Power Engineering Institute», Krasnokazarmennaya street 14, Moscow, Russia, 111250 3Bauman Moscow State Technical University, 2nd Baumanskaya street 5, Moscow, Russia, 105005  Abstract. To reconstruct three-dimensional (3D) structure of objects and measure their geometric parameters using stereoscopic imaging systems, it is necessary to implement a number of image processing algorithms. For higher system efficiency, the choice of these algorithms and mathematical models should be taken into account at the stage of optical system design. We demonstrate the capabilities of optical design software to perform computer simulation of geometrical calibration for stereoscopic systems. The simulation allows comparison of mathematical models used for 3D reconstruction and estimation of 3D measurements errors caused by tolerances of optical elements, temperature variations and other factors. Using this technique, we analyze the design of prism-based stereoscopic system and show that the proposed ray tracing camera model considering pupil aberrations provides higher measurement precision. The results of computer simulation are confirmed by experiments with the self-developed stereoscopic system. 1. Introduction Modern optoelectronic systems extensively use digital image processing for image enhancement, data extraction and precise measurements. Furthermore, computational imaging allows retrieving information, which is difficult, costly or even impossible to obtain using conventional imaging [1, 2]. At the same time, the optical system and the image processing pipeline are designed and optimized sequentially in the conventional methodology of optoelectronic system design [3, 4]. This hampers the design process and makes it difficult to find the optimal design solution. As shown in [3], the joint optimization of the design parameters of the optical system and the parameters of the image processing algorithm leads to better results compared to the traditional approach.  Stereoscopic imaging systems implement a number of algorithms to reconstruct three-dimensional (3D) structure of objects and measure their geometric parameters. These algorithms include image enhancement and rectification, stereo matching, calibration and 3D reconstruction algorithms, which contain a mathematical model of image formation (also referred as ‘camera model’) [5-10]. Thus, the joint design approach for stereoscopic systems implies that the choice of these algorithms and camera model should be taken into account at the stage of optical system design and the evaluation of the entire system performance should be based on 3D measurement errors. Компьютерная оптика и нанофотоника     A.V. Gorevoy, A.S. Machikhin, V.I. Batshev, V.Ya. Koluchkin V Международная конференция и молодёжная школа «Информационные технологии и нанотехнологии» (ИТНТ-2019)   154 The optimal choice of camera model is often a key condition for precise measurements. It is a compromise between the sufficient description of the ray tracing through the optical system (which requires the model to have more parameters) and the reliability (the fewer number of parameters is better). On the one hand, the optical system may be designed with significant residual distortion and reduced cost if the camera model properly accounts for this distortion and allows its digital correction.  On the other hand, if the residual distortion is large, its mathematical description may require many parameters, which leads to complicated calibration procedure, expensive calibration equipment and lower reliability. For complex optical systems that use diffraction gratings [5], mirrors [6, 7] or prisms [8-10] to obtain stereoscopic images, the choice of the optimal camera model at the initial design stage is not obvious. As shown in [11,12], the optical design software (e.g. Zemax) together with Matlab may be used for the computer simulation of the camera calibration procedure. In particular, this simulation was applied to assess the impact of temperature variations and tolerancing on the pinhole camera model parameters.  We demonstrate that the computer simulation using the optical design software allows to estimate systematic and random errors of 3D measurements and evaluate appropriateness of camera models and calibration equipment for the designed optical system. This simulation is necessary to implement the joint design approach for optical system, calibration equipment and software as shown in figure 1.  Figure 1. The framework of traditional and joint design strategies for optical system, calibration equipment and software.  In the traditional design approach, insufficient accuracy or poor reliability of the chosen camera model or the calibration procedure may be revealed after the prototype of the entire system has been assembled and tested. Moreover, the results of a single prototype tests can not ensure their appropriateness in mass production. The joint design approach using computer simulation allows estimating the impact of tolerances for imaging system and calibration equipment, temperature variations, image noise and other factors on measurements errors. If the results of the computer simulation are unsatisfactory, it is possible to return to any design stage and change the design parameters of the optical system as well as the camera model.  In this work, we apply this technique to the design of the prism-based stereoscopic system and evaluate several camera models considering the impact of tolerancing on calibration accuracy. 2. Mathematical models and algorithms for 3D reconstruction We consider the design of prism-based stereoscopic system capable to obtain two images of the object from different viewpoints on a single sensor (as shown in figure 2). Such an optical system may be Компьютерная оптика и нанофотоника     A.V. Gorevoy, A.S. Machikhin, V.I. Batshev, V.Ya. Koluchkin V Международная конференция и молодёжная школа «Информационные технологии и нанотехнологии» (ИТНТ-2019)   155 used in video endoscopes, so it should be miniature and has as few components as possible to fit in a small diameter of the probe. The usage of the prism leads to strong image distortion and pupil aberrations which can not be completely corrected by lenses [13] and should be properly accounted for by a camera model. The peculiarities of this system make it a good example to illustrate the joint design approach.    Figure 2. Basic layout of prism-based stereoscopic system: 1 – object, 2 – prism, 3 – auxiliary lens, 4 – aperture diaphragm, 5 – main lens, 6a and 6b – two parts of the stereoscopic image, 7 – image sensor. As shown in [9, 10], the camera model using the backward ray tracing through the prism provides better measurement accuracy for prism-based stereoscopic systems compared to simple pinhole models. In addition to the distortion induced by the prism, the model used in [10] separately describes the distortion of the auxiliary and main lenses by the radial polynomial up to 7th order. At the same time, it does not separately consider pupil aberrations of the lenses. To improve the results, we have transferred the radial distortion coefficients to denominator and added the same radial polynomial to take into account the ray shift caused by pupil aberrations. Therefore, the improved model defines coordinates of each ray in the object space by coordinates in two planes similar to the models in [14, 15] and deviations of coordinates in these planes are described by radial polynomials. The details about the calculation of 3D point coordinates and the calibration algorithm may be found in [10]. We used flat calibration targets, so we adopted the cost function based on the distance in the target plane and utilized the Levenberg-Marquardt method to solve the minimization problem. 3. Computer simulation We have designed the miniature prism-based stereoscopic system and performed the computer simulation of calibration using Zemax optical design software. The designed system has the following characteristics: f/11, field of view 40°×45° in each channel, range of working distances 5-40 mm and 1/6” image sensor with 1920×1080 pixels.  We simulated the calibration procedure utilizing 3 flat calibration targets with 25×25 grid of marker points and 0.5, 1 and 2 mm distance between points. Each target was placed at 6 positions including ones perpendicular to the optical axis of the main lens (z-axis) and rotated by 30° around transverse axes (x, y) to cover the total range of distances from 8 to 32 mm. The coordinates of image points for all positions of calibration targets were calculated by forward tracing of chief ray from each point of target using Zemax. The collected data was used to calculate the parameters of considered camera models. To simulate the acquisition of test sequences for measurements [10], the small and medium targets were positioned perpendicular to z-axis and shifted along it with 1 mm step. According to the usual applications of 3D measurement endoscopic systems for geometric measurements, we have chosen the uncertainty of segment length measurement as the criterion to evaluate the error for these systems. Therefore, we divided the obtained 3D data into zones according to the distance along the z-axis and calculated mean and standard deviation (STD) of the segment length along x, y and z axes for every zone. The results correspond to systematic error caused by insufficient description of the optical system provided by considered camera models. The STD values actually show the differences in the measurement of segment length at the specified distance across the field of view. As shown in figure 3, the improved model provides smaller systematic errors. Hence, the separate description of pupil aberrations for the lens is indeed necessary for the designed system.  Компьютерная оптика и нанофотоника     A.V. Gorevoy, A.S. Machikhin, V.I. Batshev, V.Ya. Koluchkin V Международная конференция и молодёжная школа «Информационные технологии и нанотехнологии» (ИТНТ-2019)   156  Figure 3. The results of computer simulation: dependence of mean value (solid) and standard deviation (dotted) of difference between reference and measured lengths of the 1 mm segment along x (left), y (center) and z (right) axes on the distance to the target.  Afterwards, we have performed tolerance analysis of the designed system using Zemax. The obtained tolerances were used to generate 50 optical systems with random variations of the design parameters. Next, the computer simulation of calibration and measurements was repeated for each of these systems. The average STD values for the improved model were about 1.5 times higher and the maximum STD values were 3 times higher than for nominal system, while the average mean values were approximately the same. The increase of systematic error is mainly caused by decentering of lenses leading to non-radial distortion which is not present in the considered camera models. 4. Experiments with the manufactured stereoscopic system The prototype of the designed prism-based stereoscopic system has been manufactured and used for experiments to verify the computer simulation results. All details of calibration procedure are the same as for the computer simulation. The calibration targets contained chessboard pattern produced by chrome etching on glass with inaccuracy about 1 μm. Since the implemented calibration method does not require precise targets positioning, no specific equipment was necessary to capture calibration image sequences. To capture test sequences, the linear translation stage was used to shift target along the z-axis. The results obtained by calibration and measurements using two considered camera models are presented in figure 4.  Figure 4. The results of experiments with the manufactured system.  The experimental results demonstrate the same trends as computer simulation. The advantage of the improved model is vivid for shorter distances where the impact of random errors is lower and the measurement accuracy is mainly determined by systematic errors. 5. Conclusion We have utilized the computer simulation using the optical design software to estimate systematic errors of 3D measurements and evaluated two camera models for the designed prism-based stereoscopic system. The results of the computer simulation have been confirmed by the experiments with the manufactured system. We have demonstrated the use of this simulation as a basic and Компьютерная оптика и нанофотоника     A.V. Gorevoy, A.S. Machikhin, V.I. Batshev, V.Ya. Koluchkin V Международная конференция и молодёжная школа «Информационные технологии и нанотехнологии» (ИТНТ-2019)   157 powerful tool for implementation of the joint design approach. It may be effective as for stereoscopic optical systems so for other 3D optical measurement systems.  6. References [1] Zhou, C. Computational Cameras: Convergence of Optics and Processing / C. Zhou, S.K. Nayar // IEEE Trans. Image Process. – 2011. – Vol. 20(12). – P. 3322-3340. [2] Mait, J.N. Computational imaging / J.N. Mait, G.W. Euliss, R.A. Athale // Adv. Opt. Photon. – 2018. – Vol. 10(2). – P. 409-483. [3] Stork, D.G. Theoretical foundations for joint digital-optical analysis of electro-optical imaging systems / D.G. Stork, M.D. Robinson // Appl. Opt. – 2008. – Vol. 47(10). – P. B64-B75. [4] Stork, D.G. Toward a signal-processing foundation for computational sensing and imaging: Electro-optical basis and merit functions / D.G. Stork // APSIPA Transactions on Signal and Information Processing. – 2018. – Vol. 6. – E8.  [5]  Pan, B. Single-camera microscopic stereo digital image correlation using a diffraction grating / B. Pan, Q. Wang // Opt. Exp. – 2013. – Vol. 21. – P. 25056-25068. [6]  Zhu, J.G. Design and calibration of a single-camera-based stereo vision sensor / J.G. Zhu, Y.J. Li, S.H. Ye // Opt. Eng. – 2006. – Vol. 45(8). – 083001. [7]  Zhou, F.Q. A novel way of understanding for calibrating stereo vision sensor constructed by a single camera and mirrors / F.Q. Zhou, Y.X. Wang, B. Peng, Y. Cui // Measurement. – 2013. – Vol. 46. – P. 1147-1160. [8] Cui, X. Accurate geometrical optic model for single-lens stereovision system using a prism / X. Cui, K.B. Lim, Q. Guo, D. Wang // J. Opt. Soc. Am. A. – 2012. – Vol. 29(9). – P. 1828-1837.  [9] Wu, L. Single-lens 3D digital image correlation system based on a bilateral telecentric lens and a biprism: validation and application / L. Wu, J. Zhu, H. Xie // Appl. Opt. – 2015. – Vol. 54(26). – P. 7842-7850. [10] Gorevoy, A. Optimal calibration of a prism-based videoendoscopic system for precise 3D measurements / A. Gorevoy, A. Machikhin // Computer Optics. – 2017. – Vol. 41(4). – P. 535-544. DOI: 10.18287/2412-6179-2017-41-4-535-544. [11] Poulin-Girard, A.-S. Virtual camera calibration using optical design software / A.-S. Poulin-Girard, X. Dallaire, S. Thibault, D. Laurendeau // Appl. Opt. – 2014. – Vol. 53(13). – P. 2822-2827.  [12] Poulin-Girard, A.-S. Influence of camera calibration conditions on the accuracy of 3D reconstruction / A.-S. Poulin-Girard, S. Thibault, D. Laurendeau // Opt. Exp. – 2016. – Vol. 24(3). – P. 2678-2686. [13] Batshev, V. Stereoscopic tip for a video endoscope: problems in design / V. Batshev, A. Machikhin, Y. Kachuri // Proc. SPIE. – 2017. – Vol. 10466. – 104664D. [14] Sturm, P. Camera models and fundamental concepts used in geometric computer vision / P. Sturm, S. Ramalingam, J.-P. Tardif, S. Gasparini, J. Barreto // Foundations and Trends in Computer Graphics and Vision. – 2011. – Vol. 6. – P. 1-183. [15] Matsuzawa, T. Camera calibration based on the principal rays model of imaging optical systems / T. Matsuzawa // J. Opt. Soc. Am. A – 2017. – Vol. 34(4). – P. 624-639. Acknowledgments The Russian Science Foundation (project #17-19-01355) financially supported the work.  

Calibration simulation for stereoscopic optical systems using optical design software

http://repo.ssau.ru/bitstream/Informacionnye-tehnologii-i-nanotehnologii/simulation-for-stereoscopic-optical-systems-using-optical-design-software-75265/1/paper29.pdf

Calibration simulation for stereoscopic optical systems using optical design software

Abstract

Similar works

Full text

Available Versions

Samara University