Calibration of structured light system using unidirectional fringe patterns

3D shape measurement has a variety of applications in many areas, such as manufacturing, design, medicine and entertainment. There are many technologies that were successfully implemented in the past decades to measure three dimensional information of an object. The measurement techniques can be broadly classified into contact and non-contact measurement methods. One of the most widely used contact method is Coordinate Measuring Machine (CMM) which dates back to late 1950s. The method by far is one of the most accurate method as it can have sub-micrometer accuracy. But it becomes difficult to use this technique for soft objects as the probe might deform the surface of the object being measured. Also the scanning could be a time-consuming process. In order to address the problems in contact methods, non-contact methods such as time of flight (TOF), triangulation based laser scanner techniques, depth from defocus and stereo vision were invented. The main limitation with the time of flight laser scanner is that it does not give a high depth resolution. On the other hand, triangulation based laser scanning method scans the object line by line which might be time consuming. The depth from defocus method obtains 3D information of the object by relating depth to defocus blur analysis. However, it is difficult to capture the 3D geometry of objects that does not have a rich texture. The stereo vision system imitates human vision. It uses two cameras for capturing pictures of the object from different angles. The 3D coordinate information is obtained using triangulation. The main limitation with this technology is: when the object has a uniform texture, it becomes difficult to find corresponding pairs between the two cameras. Therefore, the structured light system (SLS) was introduced to address the above mentioned limitations. SLS is an extension of stereo vision system with one of the cameras being replaced by a projector. The pre-designed structured patterns are projected on to the object using a video projector. The main advantage with this system is that it does not use the object\u27s texture for identifying the corresponding pairs. But the patterns have to be coded in a certain way so that the camera-projector correspondence can be established. There are many codifications techniques such as pseudo-random codification, binary and N-ary codification. Pseudo-random codification uses laser speckles or structure-coded speckle patterns that vary in both the directions. However, the resolution is limited because each coded structure occupies multiple pixels in order to be unique. On the other hand, binary codifications projects a sequence of binary patterns. The main advantage with such a codification is that it is robust to noise as only two intensity levels are used (0s and 255). However, the resolution is limited because the width of the narrowest coding stripe should be more than the pixel size. Moreover, it takes many images to encode a scene that occupies a large number of pixels. To address this, N-ary codification makes use of multiple intensity levels between 0 and 255. Therefore the total number of coded patterns can be reduced. The main limitation is that the intensity-ratio analysis may be subject to noise. Digital Fringe Projection (DFP) system was developed to address the limitations of binary and N-ary codifications. In DFP computer generated sinusoidal patterns are projected on to the object and then the camera captures the distorted patterns from another angle. The main advantage of this method is that it is robust to the noise, ambient light and reflectivity as phase information is used instead of intensity. Albeit the merit of using phase, to achieve highly accurate 3D geometric reconstruction, it is also of crucial importance to calibrate the camera-projector system. Unlike the camera calibration, the projector calibration is difficult. This is mainly because the projector cannot capture images like a camera. Early attempts were made to calibrate the camera-projector system using a reference plane. The object geometry was reconstructed by comparing the phase difference between the object and the reference plane. However, the chosen reference plane needs to simultaneously possess a high planarity and a good optical property, which is typically difficult to achieve. Also, such calibration may be inaccurate if non-telecentric lenses are used. Calibration of the projector can also be done by treating it as the inverse of a camera. This method addressed the limitations of reference plane based method, as the exact intrinsic and extrinsic parameters of the imaging lenses are obtained. So a perfect reference plane is no longer required. The calibration method typically requires projecting orthogonal patterns on to the object. However, this method of calibration can be used only for structured light system with video projector. Grating slits and interferometers cannot be calibrated by this method as we cannot produce orthogonal patterns with such systems. In this research we have introduced a novel calibration method which uses patterns only in a single direction. We have theoretically proved that there exists one degree-of-freedom of redundancy in the conventional calibration methods, thus making it possible to use unidirectional patterns instead of orthogonal fringe patterns. Experiments show that under a measurement range of 200mm x 150mm x 120mm, our measurement results are comparable to the results obtained using conventional calibration method. Evaluated by repeatedly measuring a sphere with 147.726 mm diameter, our measurement accuracy on average can be as high as 0.20 mm with a standard deviation of 0.12 mm

High-quality 3D shape measurement with binarized dual phase-shifting method

ABSTRACT 3-D technology is commonplace in today\u27s world. They are used in many dierent aspects of life. Researchers have been keen on 3-D shape measurement and 3-D reconstruction techniques in past decades as a result of inspirations from dierent applications ranging from manufacturing, medicine to entertainment. The techniques can be broadly divided into contact and non-contact techniques. The contact techniques like coordinate measuring machine (CMM) dates way back to 1950s. It has been used extensively in the industries since then. It becomes predominant in industrial inspections owing to its high accuracy in the order of m. As we know that quality control is an important part of modern industries hence the technology is enjoying great popularity. However, the main disadvantage of this method is its slow speeds due to its requirement of point-by-point touch. Also, since this is a contact process, it might deform a soft object while performing measurements. Such limitations led the researchers to explore non-contact measurement technologies (optical metrology techniques). There are a variety of optical techniques developed till now. Some of the well-known technologies include laser scanners, stereo vision, and structured light systems. The main limitation of laser scanners is its limited speed due to its point-by-point or line-by-line scanning process. The stereo vision uses two cameras which take pictures of the object at two dierent angles. Then epipolar geometry is used to determine the 3-D coordinates of points in real-world. Such technology imitates human vision, but it had a few limitations too like the diculty of correspondence detection for uniform or periodic textures. Hence structured light systems were introduced which addresses the aforementioned limitations. There are various techniques developed including 2-D pseudo-random codication, binary codication, N-ary codication and digital fringe projection (DFP). The limitation of 2-D pseudo-random codication technique is its inability to achieve high spatial resolution since any uniquely generated and projected feature requires a span of several projector pixels. The binary codication techniques reduce the requirement of 2-D features to 1-D ones. However, since there are only two intensities, it is dicult to differentiate between the individual pixels within each black or white stripe. The other disadvantage is that n patterns are required to encode 2n pixels, meaning that the measurement speeds will be severely affected if a scene is to be coded with high-resolution. Dierently, DFP uses continuous sinusoidal patterns. The usage of continuous patterns addresses the main disadvantage of binary codication (i.e. the inability of this technique to differentiate between pixels was resolved by using sinusoid patterns). Thus, the spatial resolution is increased up to camera-pixel-level. On the other hand, since the DFP technique used 8-bit sinusoid patterns, the speed of measurement is limited to the maximum refreshing rate of 8-bit images for many video projectors (e.g. 120 Hz). This made it inapplicable for measurements of highly dynamic scenes. In order to overcome this speed limitation, the binary defocussing technique was proposed which uses 1-bit patterns to produce sinusoidal prole by projector defocusing. Although this technique has signicantly boosted the measurement speed up to kHz-level, if the patterns are not properly defocused (nearly focused or overly defocused), increased phase noise or harmonic errors will deteriorate the reconstructed surface quality. In this thesis research, two techniques are proposed to overcome the limitations of both DFP and binary defocusing technique: binarized dual phase shifting (BDPS) technique and Hilbert binarized dual phase shifting technique (HBDPS). Both techniques were able to achieve high-quality 3-D shape measurements even when the projector is not sufficiently defocused. The harmonic error was reduced by 47% by the BDPS method, and 74% by the HBDPS method. Moreover, both methods use binary patterns which preserve the speed advantage of the binary technology, hence it is potentially applicable to simultaneous high-speed and high-accuracy 3D shape measurements

3D Shape Measurement of Objects in Motion and Objects with Complex Surfaces

This thesis aims to address the issues caused by high reflective surface and object with motion in the three dimensional (3D) shape measurement based on phase shifting profilometry (PSP). Firstly, the influence of the reflectivity of the object surface on the fringe patterns is analysed. One of the essential factors related to phase precision is modulation index, which has a direct relationship with the surface reflectivity. A comparative study focusing on the modulation index of different materials is presented. The distribution of modulation index for different material samples is statistically analysed, which leads to the conclusion that the modulation index is determined by the diffuse reflectivity. Then the method based on optimized combination of multiple reflected image patterns is proposed to address the saturation issue and improve the accuracy for the reconstruction of object with high reflectivity.A set of phase shifted sinusoidal fringe patterns with different exposure time are projected to the object and then captured by camera. Then a set of masks are generated to select the data for the compositing. Maximalsignal-to-noise ratio combining model is employed to form the composite images pattern. The composite images are then used to phase mapping.Comparing to the method only using the highest intensity of pixels for compositing image, the signal noise ratio (SNR) of composite image is increased due to more efficient use of information carried by the images

Superfast three-dimensional (3D) shape measurement with binary defocusing techniques and its applications

High-speed and high-accuracy three-dimensional (3D) shape measurement has enormous potential to benefit numerous areas including advanced manufacturing, medical imaging, and diverse scientific research fields. For example, capturing the rapidly pulsing wings of a flying insect could enhance our understanding of flight and lead to better and safer aircraft designs. Even though there are numerous 3D shape measurement techniques in the literature, it remains extremely difficult to accurately capture rapidly changing events. Due to the potential for achieving high speed and high measurement accuracy, the digital fringe projection (DFP) techniques have been exhaustively studied and extensively applied to numerous disciplines. Real-time (30 Hz or better) 3D shape measurement techniques have been developed with DFP methods, yet the upper speed limit is typically 120 Hz, the refresh rate of a typical digital video projector. 120 Hz speed can accurately measure the slowly changing objects, such as human facial expressions, but it is far from sufficient to capture high-speed motions (e.g., live, beating hearts or flying insects). To overcome this speed limitation, the binary defocusing technique was recently proposed. Instead of using 8-bit sinusoidal patterns, the binary defocusing technique generates sinusoidal patterns by properly defocusing squared 1-bit binary patterns. Using this technique, kilo-Hertz (kHz) 3D shape measurement rate has been achieved. However, the binary defocusing technique suffers three major limitations: 1) low phase quality due to the influence of high-frequency harmonics; 2) smaller depth measurement range; and 3) low measurement accuracy due to the difficulty of applying existing calibration methods to the system with an out-of-focus projector. The goal of this dissertation research is to achieve superfast 3D shape measurement by overcoming the major limitations of the binary defocusing technique. Once a superfast 3D shape measurement platform is developed, numerous applications could be benefited. To this end, this dissertation research look into verifying its value by applying to the biomedical engineering field. Specifically, this dissertation research has made major contributions by conquering some major challenges associated with the binary defocusing technique. The first challenge this dissertation addresses is associated with the limited depth range and low phase quality of the binary defocusing method. The binary defocusing technique essentially generates quasi-sinusoidal fringe patterns by suppressing high-frequency harmonics through lens defocusing. However, the optical engines of the majority of digital video projectors are designed and optimized for applications with large depth of focus; for this reason, good quality sinusoids can only be generated by this technique within a very small depth region. This problem is exacerbated if the fringe stripes are wide. In that case, the high-frequency harmonics cannot be properly suppressed through defocusing, making it almost impossible to generate reasonable quality sinusoids. To alleviate this problem associated with high-frequency harmonics, an optimal pulse width modulation (OPWM) method, developed in power electronics, is proposed to improve the fringe pattern quality. Instead of projecting squared binary structures, the patterns are optimized, in one dimension perpendicular to the fringe stripes, by selectively eliminating the undesired harmonics which affect the phase quality the most. Both simulation and experimental data demonstrate that the OPWM method can substantially improve the squared binary defocusing technique when the fringe periods are between 30-300 pixels. With this technique, a multi-frequency phase-shifting algorithm is realized that enables the development of a 556-Hz 3D shape measurement system capable of capturing multiple rapidly moving objects. The OPWM technique is proved successful when the fringe stripe widths are within a certain range, yet it fails to achieve higher-quality fringe patterns when the desired fringe period goes beyond the optimal range. To further improve the binary defocusing technique, binary dithering techniques are proposed. Unlike the OPWM method, the dithering technique optimizes the patterns in both x and y dimensions, and thus can achieve higher-quality fringe patterns. This research demonstrates the superiority of this technique over all aforementioned binary defocusing techniques for high-quality 3D shape measurement even when the projector is nearly focused and the fringe stripes are wide. The second challenge this dissertation addresses is accurately calibrating the DFP system with an out-of-focus projector. The binary defocusing technique generates quasi-sinusoidal patterns through defocusing, and thus the projector cannot be perfectly in focus. In the meantime, state-of-the-art DFP system calibration assumes that the projector is always in focus. To address this problem, a novel calibration method is proposed that directly relates depth z with the phase pixel by pixel without the requirement of projector calibration. By this means, very high accuracy depth measurement is achieved: for a depth measurement range of 100 mm, the root-mean-squared (rms) error is approximately 70 &mu m. The third challenge this dissertation addresses is associated with the hardware limitation for the superfast 3D shape measurement technique. The high refresh rate of the digital micro-mirror device (DMD) has enabled superfast 3D shape measurement, yet a hardware limitation has been found once the speeds go beyond a certain range. This is because the DMD cannot completely turn on/off between frames, leading to coupling problems associated with the transient response of the DMD chip. The coupling effect causes substantial measurement error during high-speed measurement. Fortunately, since this type of error is systematic, this research finds that such error can be reduced to a negligible level by properly controlling the timing of the projector and the camera. The superfast 3D shape measurement platform developed in this research could benefit numerous applications. This research applies the developed platform to the measurement of the cardiac motion of live, beating rabbit hearts. The 3D geometric motion of the live, beating rabbit hearts can be successfully captured if the measurement speed is sufficiently fast (i.e. 200 Hz or higher for normal beating rabbit hearts). This research also finds that, due to the optical properties of live tissue, caution should be given in selecting the spectrum of light in order to properly measure the heart surface. In summary, the improved binary defocusing techniques are overwhelmingly advantageous compared to the conventional sinusoidal projection method or the squared binary defocusing technique. We believe that the superfast 3D shape measurement platform we have developed has the potential to broadly impact many more academic studies and industrial practices, especially those where understanding the high-speed 3D phenomena is critical

Automatic Look-Up Table Based Real-Time Phase Unwrapping for Phase Measuring Profilometry and Optimal Reference Frequency Selection

For temporal phase unwrapping in phase measuring profilometry, it has recently been reported that two phases with co-prime frequencies can be absolutely unwrapped using a look-up table; however, frequency selection and table construction has been performed manually without optimization. In this paper, a universal phase unwrapping method is proposed to unwrap phase flexibly and automatically by using geometric analysis, and thus we can programmatically build a one-dimensional or two-dimensional look-up table for arbitrary two co-prime frequencies to correctly unwrap phases in real time. Moreover, a phase error model related to the defocus effect is derived to figure out an optimal reference frequency co-prime to the principal frequency. Experimental results verify the correctness and computational efficiency of the proposed method

Learning Wavefront Coding for Extended Depth of Field Imaging

Depth of field is an important factor of imaging systems that highly affects the quality of the acquired spatial information. Extended depth of field (EDoF) imaging is a challenging ill-posed problem and has been extensively addressed in the literature. We propose a computational imaging approach for EDoF, where we employ wavefront coding via a diffractive optical element (DOE) and we achieve deblurring through a convolutional neural network. Thanks to the end-to-end differentiable modeling of optical image formation and computational post-processing, we jointly optimize the optical design, i.e., DOE, and the deblurring through standard gradient descent methods. Based on the properties of the underlying refractive lens and the desired EDoF range, we provide an analytical expression for the search space of the DOE, which is instrumental in the convergence of the end-to-end network. We achieve superior EDoF imaging performance compared to the state of the art, where we demonstrate results with minimal artifacts in various scenarios, including deep 3D scenes and broadband imaging

Computational structured illumination for high-content fluorescent and phase microscopy

High-content biological microscopy targets high-resolution imaging across large fields-of-view (FOVs). Recent works have demonstrated that computational imaging can provide efficient solutions for high-content microscopy. Here, we use speckle structured illumination microscopy (SIM) as a robust and cost-effective solution for high-content fluorescence microscopy with simultaneous high-content quantitative phase (QP). This multi-modal compatibility is essential for studies requiring cross-correlative biological analysis. Our method uses laterally-translated Scotch tape to generate high-resolution speckle illumination patterns across a large FOV. Custom optimization algorithms then jointly reconstruct the sample's super-resolution fluorescent (incoherent) and QP (coherent) distributions, while digitally correcting for system imperfections such as unknown speckle illumination patterns, system aberrations and pattern translations. Beyond previous linear SIM works, we achieve resolution gains of 4x the objective's diffraction-limited native resolution, resulting in 700 nm fluorescence and 1.2 um QP resolution, across a FOV of 2x2.7 mm^2, giving a space-bandwidth product (SBP) of 60 megapixels