Micro Fourier Transform Profilometry (μ\muFTP): 3D shape measurement at 10,000 frames per second

Recent advances in imaging sensors and digital light projection technology have facilitated a rapid progress in 3D optical sensing, enabling 3D surfaces of complex-shaped objects to be captured with improved resolution and accuracy. However, due to the large number of projection patterns required for phase recovery and disambiguation, the maximum fame rates of current 3D shape measurement techniques are still limited to the range of hundreds of frames per second (fps). Here, we demonstrate a new 3D dynamic imaging technique, Micro Fourier Transform Profilometry ($\mu$FTP), which can capture 3D surfaces of transient events at up to 10,000 fps based on our newly developed high-speed fringe projection system. Compared with existing techniques, $\mu$FTP has the prominent advantage of recovering an accurate, unambiguous, and dense 3D point cloud with only two projected patterns. Furthermore, the phase information is encoded within a single high-frequency fringe image, thereby allowing motion-artifact-free reconstruction of transient events with temporal resolution of 50 microseconds. To show $\mu$FTP's broad utility, we use it to reconstruct 3D videos of 4 transient scenes: vibrating cantilevers, rotating fan blades, bullet fired from a toy gun, and balloon's explosion triggered by a flying dart, which were previously difficult or even unable to be captured with conventional approaches.Comment: This manuscript was originally submitted on 30th January 1

Temporal phase unwrapping using deep learning

The multi-frequency temporal phase unwrapping (MF-TPU) method, as a classical phase unwrapping algorithm for fringe projection profilometry (FPP), is capable of eliminating the phase ambiguities even in the presence of surface discontinuities or spatially isolated objects. For the simplest and most efficient case, two sets of 3-step phase-shifting fringe patterns are used: the high-frequency one is for 3D measurement and the unit-frequency one is for unwrapping the phase obtained from the high-frequency pattern set. The final measurement precision or sensitivity is determined by the number of fringes used within the high-frequency pattern, under the precondition that the phase can be successfully unwrapped without triggering the fringe order error. Consequently, in order to guarantee a reasonable unwrapping success rate, the fringe number (or period number) of the high-frequency fringe patterns is generally restricted to about 16, resulting in limited measurement accuracy. On the other hand, using additional intermediate sets of fringe patterns can unwrap the phase with higher frequency, but at the expense of a prolonged pattern sequence. Inspired by recent successes of deep learning techniques for computer vision and computational imaging, in this work, we report that the deep neural networks can learn to perform TPU after appropriate training, as called deep-learning based temporal phase unwrapping (DL-TPU), which can substantially improve the unwrapping reliability compared with MF-TPU even in the presence of different types of error sources, e.g., intensity noise, low fringe modulation, and projector nonlinearity. We further experimentally demonstrate for the first time, to our knowledge, that the high-frequency phase obtained from 64-period 3-step phase-shifting fringe patterns can be directly and reliably unwrapped from one unit-frequency phase using DL-TPU

Novel Approaches in Structured Light Illumination

Among the various approaches to 3-D imaging, structured light illumination (SLI) is widely spread. SLI employs a pair of digital projector and digital camera such that the correspondences can be found based upon the projecting and capturing of a group of designed light patterns. As an active sensing method, SLI is known for its robustness and high accuracy. In this dissertation, I study the phase shifting method (PSM), which is one of the most employed strategy in SLI. And, three novel approaches in PSM have been proposed in this dissertation. First, by regarding the design of patterns as placing points in an N-dimensional space, I take the phase measuring profilometry (PMP) as an example and propose the edge-pattern strategy which achieves maximum signal to noise ratio (SNR) for the projected patterns. Second, I develop a novel period information embedded pattern strategy for fast, reliable 3-D data acquisition and reconstruction. The proposed period coded phase shifting strategy removes the depth ambiguity associated with traditional phase shifting patterns without reducing phase accuracy or increasing the number of projected patterns. Thus, it can be employed for high accuracy realtime 3-D system. Then, I propose a hybrid approach for high quality 3-D reconstructions with only a small number of illumination patterns by maximizing the use of correspondence information from the phase, texture, and modulation data derived from multi-view, PMP-based, SLI images, without rigorously synchronizing the cameras and projectors and calibrating the device gammas. Experimental results demonstrate the advantages of the proposed novel strategies for 3-D SLI systems

Acquisition of 3D shapes of moving objects using fringe projection profilometry

Three-dimensional (3D) shape measurement for object surface reconstruction has potential applications in many areas, such as security, manufacturing and entertainment. As an effective non-contact technique for 3D shape measurements, fringe projection profilometry (FPP) has attracted significant research interests because of its high measurement speed, high measurement accuracy and ease to implement. Conventional FPP analysis approaches are applicable to the calculation of phase differences for static objects. However, 3D shape measurement for dynamic objects remains a challenging task, although they are highly demanded in many applications. The study of this thesis work aims to enhance the measurement accuracy of the FPP techniques for the 3D shape of objects subject to movement in the 3D space. The 3D movement of objects changes not only the position of the object but also the height information with respect to the measurement system, resulting in motion-induced errors with the use of existing FPP technology. The thesis presents the work conducted for solutions of this challenging problem

State-of-the-art active optical techniques for three-dimensional surface metrology: a review [Invited]

This paper reviews recent developments of non-contact three-dimensional (3D) surface metrology using an active structured optical probe. We focus primarily on those active non-contact 3D surface measurement techniques that could be applicable to the manufacturing industry. We discuss principles of each technology, and its advantageous characteristics as well as limitations. Towards the end, we discuss our perspectives on the current technological challenges in designing and implementing these methods in practical applications.Purdue Universit

Real-time 3D surface-shape measurement using fringe projection and system-geometry constraints

Optical three-dimensional (3D) surface-shape measurement has diverse applications in engineering, computer vision and medical science. Fringe projection profilometry (FPP), uses a camera-projector system to permit high-accuracy full-field 3D surface-shape measurement by projecting fringe patterns onto an object surface, capturing images of the deformed patterns, and computing the 3D surface geometry. A wrapped phase map can be computed from the camera images by phase analysis techniques. Phase-unwrapping can solve the phase ambiguity of the wrapped phase map and permit determination of camera-projector correspondences. The object surface geometry can then be reconstructed by stereovision techniques after system calibration. For real-time 3D measurement, geometry-constraint based methods may be a preferred technique over other phase-unwrapping methods, since geometry-constraint methods can handle surface discontinuities, which are problematic for spatial phase unwrapping, and they do not require additional patterns, which are needed in temporal phase unwrapping. However, the fringe patterns used in geometry-constraint based methods are usually designed with a low frequency in order to maximize the reliability of correspondence determination. Although using high-frequency fringe patterns have proven to be effective in increasing the measurement accuracy by suppressing the phase error, high-frequency fringe patterns may reduce the reliability and thus are not commonly used. To address the limitations of current geometry-constraint based methods, a new fringe projection method for surface-shape measurement was developed using modulation of background and amplitude intensities of the fringe patterns to permit identification of the fringe order, and thus unwrap the phase, for high-frequency fringe patterns. Another method was developed with background modulation only, using four high-frequency phase-shifted fringe patterns. The pattern frequency is determined using a new fringe-wavelength geometry-constraint model that allows only two point candidates in the measurement volume. The correct corresponding point is selected with high reliability using a binary pattern computed from the background intensity. Equations of geometry-constraint parameters permit parameter calculation prior to measurement, thus reducing computational cost during measurement. In a further development, a new real-time 3D measurement method was devised using new background-modulated modified Fourier transform profilometry (FTP) fringe patterns and geometry constraints. The new method reduced the number of fringe patterns required for 3D surface reconstruction to two. A short camera-projector baseline allows reliable corresponding-point selection, even with high-frequency fringe patterns, and a new calibration approach reduces error induced by the short baseline. Experiments demonstrated the ability of the methods to perform real-time 3D measurement for a surface with geometric discontinuity, and for spatially isolated objects. Although multi-image FPP techniques can achieve higher accuracy than single-image methods, they suffer from motion artifacts when measuring dynamic object surfaces that are either moving or deforming. To reduce the motion-induced measurement error for multi-image FPP techniques, a new method was developed to first estimate the motion-induced phase shift errors by computing the differences between phase maps over a multiple measurement sequence. Then, a phase map with reduced motion-induced error is computed using the estimated phase shift errors. This motion-induced error compensation is computed pixel-wise for non-homogeneous surface motion. Experiments demonstrated the ability of the method to reduce motion-induced error in real-time, for real-time shape measurement of surfaces with high depth variation, and moving and deforming surfaces

Correction of the fringe order errors for fringe projection profilometry

Non-contact three-dimensional (abbreviated as 3D) Fringe projection profilometry (abbreviated as FPP) counts as a method of reconstructing the shape of object surface. This technique has been extensively used in many areas, e.g. computer vision, biomedical research, industrial applications, and virtual reality. Using a FPP, sinusoidal patterns are projected on the object surface by mean of a digital projector, and subsequently a camera captures the reflected patterns deformed by the object surface. As the shape information of the object surface is carried by the deformed patterns, the 3D profile can be retrieved through analysing these patterns. The phase unwrapping is a primary issue bound by the existing phase unwrapping techniques in FPP, aiming to recover the absolute phase from wrapped phase. The temporal phase unwrapping with multi-frequency fringe pattern was proposed, prominently advantaged by none-error propagation. Furthermore, the fringe order is deemed as the critical property to retrieve the absolute phase