A 3D Sensor Based on a Profilometrical Approach

An improved method which considers the use of Fourier and wavelet transform based analysis to infer and extract 3D information from an object by fringe projection on it is presented. This method requires a single image which contains a sinusoidal white light fringe pattern projected on it, and this pattern has a known spatial frequency and its information is used to avoid any discontinuities in the fringes with high frequency. Several computer simulations and experiments have been carried out to verify the analysis. The comparison between numerical simulations and experiments has proved the validity of this proposed method

Fringe Projection Techniques: Whither we are?

During recent years, the use of fringe projection techniques for generating three-dimensional (3D) surface information has become one of the most active research areas in optical metrol-ogy. Its applications range from measuring the 3D shape of MEMS components to the measurement of flatness of large panels (2.5 m ×.45 m). The technique has found various ap-plications in diverse fields: biomedical applications such as 3D intra-oral dental measurements [1], non-invasive 3D imag-ing and monitoring of vascular wall deformations [2], human body shape measurement for shape guided radiotherapy treat-ment [3, 4], lower back deformation measurement [5], detection and monitoring of scoliosis [6], inspection of wounds [7, 8] and skin topography measurement for use in cosmetology [9, 10, 11]; industrial and scientific applications such as char-acterization of MEMS components [12, 13], vibration analy

3D optical metrology by digital moiré: Pixel-wise calibration refinement, grid removal, and temporal phase unwrapping

Fast, accurate three dimensional (3D) optical metrology has diverse applications in object and environment modelling. Structured-lighting techniques allow non-contacting 3D surface-shape measurement by projecting patterns of light onto an object surface, capturing images of the deformed patterns, and computing the 3D surface geometry from the captured 2D images. However, motion artifacts can still be a problem with high-speed surface-motion especially with increasing demand for higher measurement resolution and accuracy. To avoid motion artifacts, fast 2D image acquisition of projected patterns is required. Fast multi-pattern projection and minimization of the number of projected patterns are two approaches for dynamic object measurement. To achieve a higher rate of switching frames, fast multi-pattern projection techniques require costly projector hardware modification or new designs of projection systems to increase the projection rate beyond the capabilities of off-the-shelf projectors. Even if these disadvantages were acceptable (higher cost, complex hardware), and even if the rate of acquisition achievable with current systems were fast enough to avoid errors, minimization of the number of captured frames required will still contribute to reduce further the effect of object motion on measurement accuracy and to enable capture of higher object dynamics. Development of an optical 3D metrology method that minimizes the number of projected patterns while maintaining accurate 3D surface-shape measurement of objects with continuous and discontinuous surface geometry has remained a challenge. Capture of a single image-frame instead of multiple frames would be advantageous for measuring moving or deforming objects. Since accurate measurement generally requires multiple phase-shifted images, imbedding multiple patterns into a single projected composite pattern is one approach to achieve accurate single-frame 3D surface-shape measurement. The main limitations of existing single-frame methods based on composite patterns are poor resolution, small range of gray-level intensity due to collection of multiple patterns in one image, and degradation of the extracted patterns because of modulation and demodulation processes on the captured composite pattern image. To benefit from the advantages of multi-pattern projection of phase-shifted fringes and single-frame techniques, without combining phase-shifted patterns into one frame, digital moiré was used. Moiré patterns are generated by projecting a grid pattern onto the object, capturing a single frame, and in a post-process, superimposing a synthetic grid of the same frequency as in the captured image. Phase-shifting is carried out as a post-process by digitally shifting the synthetic grid across the captured image. The useful moiré patterns, which contain object shape information, are contaminated with a high-frequency grid lines that must be removed. After performing grid removal, computation of a phase map, and phase-to-height mapping, 3D object shape can be computed. The advantage of digital moiré provides an opportunity to decrease the number of projected patterns. However, in previous attempts to apply digital phase-shifting moiré to perform 3D surface-shape measurement, there have been significant limitations. To address the limitation of previous system-calibration techniques based on direct measurement of optical-setup parameters, a moiré-wavelength based phase-to-height mapping system-calibration method was developed. The moiré-wavelength refinement performs pixel-wise computation of the moiré wavelength based on the measured height (depth). In measurement of a flat plate at different depths, the range of root-mean-square (RMS) error was reduced from 0.334 to 0.828 mm using a single global wavelength across all pixels, to 0.204 to 0.261 mm using the new pixel-wise moiré-wavelength refinement. To address the limitations of previous grid removal techniques (precise mechanical grid translation, multiple-frame capture, moiré-pattern blurring, and measurement artifacts), a new grid removal technique was developed for single-frame digital moiré using combined stationary wavelet and Fourier transforms (SWT-FFT). This approach removes high frequency grid both straight and curved lines, without moiré-pattern artifacts, blurring, and degradation, and was an improvement compared to previous techniques. To address the limitations of the high number of projected patterns and captured images of temporal phase unwrapping (TPU) in fringe projection, and the low signal-to-noise ratio of the extended phase map of TPU in digital moiré, improved methods using two-image and three-image TPU in digital phase-shifting moiré were developed. For measurement of a pair of hemispherical objects with true radii 50.80 mm by two-image TPU digital moiré, least-squares fitted spheres to the measured 3D point clouds had errors of 0.03 mm and 0.06 mm, respectively (sphere fitting standard deviations 0.15 mm and 0.14 mm), and the centre-to-centre distance measurement between hemispheres had an error of 0.19 mm. The number of captured images required by this new method is one third that for three-wavelength heterodyne temporal phase unwrapping by fringe projection techniques, which would be advantageous in measuring dynamic objects, either moving or deforming

Spatial Fringe Analysis Methods and their Application to Holographic Interferometry and Fringe Projection Techniques

To date, no fringe analysis technique has the capability to provide simultaneous and direct estimation of the continuous distributions corresponding to the interference phase and its first and second-order derivatives within the framework of a single interferometric configuration. Achieving this task would provide a significant advancement in the field of optical metrology as it allows for the measurement of displacement, strain, and curvature of a deformed object and avoids the necessity of using filtering and unwrapping procedures, multiple analysis techniques, and multiple interferometric configurations. Developing such a spatial fringe analysis method with the added advantage of having less computational complexity would open up avenues for making real-time measurements such as in the study of temporal evolution of deformation and/or strain. This thesis presents a novel approach based on piecewise polynomial phase approximation as an elegant all-in-one solution to the problems mentioned above. This approach has given birth to several advanced fringe analysis methods such as discrete-chirp-Fourier transform method, high-order instantaneous moments method, and cubic-phase function method. Significant advancements brought in the field by these methods are made evident by both theoretical analysis (simulation results) and by experimental demonstrations such as the measurement of displacement, strain and curvature in digital holographic interferometry and the measurement of 3D shape, temporal evolution of deformation and/or strain in fringe projection techniques

A real-time multi-sensor 3D surface shape measurement system using fringe analysis

This thesis presents a state-of-the-art multi-sensor, 3D surface shape measurement system that is based upon fringe projection/analysis and which operates at speeds approaching real-time. The research programme was carried out as part of MEGURATH (www.megurath.org), a collaborative research project with the aim of improving the treatment of cancer by radiotherapy. The aim of this research programme was to develop a real-time, multi-sensor 3D surface shape measurement system that is based on fringe analysis, which provides the flexibility to choose from amongst several different fringe profilometry methods and to manipulate their settings interactively. The system has been designed specifically to measure dynamic 3D human body surface shape and to act as an enabling technology for the purpose of performing Metrology Guided Radiotherapy (MGRT). However, the system has a wide variety of other potential applications, including 3D modelling and visualisation, verbatim replication, reverse engineering and industrial inspection. It can also be used as a rapid prototyping tool for algorithm development and testing, within the field of fringe pattern profilometry. The system that has been developed provides single, or multi-sensor, measurement modes that are adaptable to the specific requirements of a desired application. The multi-sensor mode can be useful for covering a larger measurement area, by providing a multi-viewpoint measurement. The overall measurement accuracy of the system is better than O.5mm, with measurement speeds of up to 3 million XYZ points/second using the single-sensor mode and rising to up to 4.6 million XYZ points/second when measuring in parallel using the three sensor multi-sensor mode. In addition the system provides a wide-ranging catalogue of fringe profilometry methods and techniques, that enables the reconstruction of 3D information through an interactive user selection of 183 possible different paths of main combinations. The research aspects behind the development of the system are presented in this thesis, along with the author's contribution to this field of research, which has included the provision of a comprehensive framework for producing such a novel optical profilometry system, and the specific techniques that were developed to fulfil the aims of this research programme. This mainly included the following advanced methods: a transversal calibration method for the optical system, an adaptive filtering technique for the Fourier Transform Profilometry (FTP) method, and a method to synthetically restore the locations of the triangulation spots. Similarly, potential applications for the system have been presented and feasibility and accuracy analyses have been conducted, presenting both qualitative and quantitative measurement results. To this end, the high robustness levels exhibited by the system have been demonstrated (in terms of adaptability, accuracy and measurement capability) by performing extensive real experiments and laboratory testing. Finally, a number of potential future system developments are described, with the intention of further extending the system capabilities

A Review and Analysis of Automatic Optical Inspection and Quality Monitoring Methods in Electronics Industry

Electronics industry is one of the fastest evolving, innovative, and most competitive industries. In order to meet the high consumption demands on electronics components, quality standards of the products must be well-maintained. Automatic optical inspection (AOI) is one of the non-destructive techniques used in quality inspection of various products. This technique is considered robust and can replace human inspectors who are subjected to dull and fatigue in performing inspection tasks. A fully automated optical inspection system consists of hardware and software setups. Hardware setup include image sensor and illumination settings and is responsible to acquire the digital image, while the software part implements an inspection algorithm to extract the features of the acquired images and classify them into defected and non-defected based on the user requirements. A sorting mechanism can be used to separate the defective products from the good ones. This article provides a comprehensive review of the various AOI systems used in electronics, micro-electronics, and opto-electronics industries. In this review the defects of the commonly inspected electronic components, such as semiconductor wafers, flat panel displays, printed circuit boards and light emitting diodes, are first explained. Hardware setups used in acquiring images are then discussed in terms of the camera and lighting source selection and configuration. The inspection algorithms used for detecting the defects in the electronic components are discussed in terms of the preprocessing, feature extraction and classification tools used for this purpose. Recent articles that used deep learning algorithms are also reviewed. The article concludes by highlighting the current trends and possible future research directions.Framework of the IQONIC Project; European Union’s Horizon 2020 Research and Innovation Program

Absolute surface topography measurement with polarisation sensitive coherence scanning interferometry

Traditionally, surface topography measurement was in the domain of quality control of engineering parts. With the advancement of manufacturing technology and affordable computational costs, different types of surfaces are produced with varied shapes and surface textures. These pose significant measurement problems, therefore, surface topography research is gaining momentum to achieve a better control of the surface. Coherence scanning interferometry (CSI) is one of the most common techniques used for measurement of surface topography. It is preferred over tactile and other non-contact techniques since it provides fast and accurate measurement with high vertical (~ 1 nm) and lateral (~1 μm) resolutions over larger areas without any damage to the surface. Essentially, CSI is treated as one dimensional (1D) superposition of the light waves from an object and a reference that generates a three dimensional (3D) interferogram. Secondly, despite the advantages, there is no standard configuration of CSI that can provide absolute surface topography measurement of an engineering part with multiple materials. An effective solution to this problem will be particularly useful in the field of semiconductor and bio-related industries where chips and instruments are made of many materials. In this Thesis, first, the CSI technique is analysed in terms of a wider theoretical framework of 3D linear filtering technique which shows the similarities among other seemingly disparate techniques such as confocal and optical coherence tomography. Due consideration to the spectral characteristic of the source and the effect of numerical aperture are given and important parameters such as vertical and lateral resolutions are computed to compare this theory with standard analysis methods. Additionally, it is shown that the 3D fringe pattern can be considered to be a superposition of a reference field and the scattered field from the top foil-like layer on the top the object. The scattered field from this foil object is dependent on the normal Fresnel reflection coefficients. Therefore, it explains the phase offset and the proportional height offset introduced by different materials, especially, metals. In an object, where multiple materials are present, each material introduces different phase to the fringe pattern and therefore, the surface topography of the entire object is altered. To overcome this problem, the optical polarising properties of the material are exploited. A novel configuration of polarisation sensitive CSI is presented where interferograms with orthogonal circular polarisations are recorded and analysed. The configuration, initially, needs to be calibrated with a material and after that at each point on the object, the refractive index and height offset can be calculated. Therefore, it can be dually used to identify unknown materials present on the object and also to compensate for the height offset introduced by each material to produce absolute surface topography of the entire object. The configuration provides good agreement with ellipsometric results for metals. Additionally, it retains the advantages of high vertical and lateral resolution same as other standard coherence scanning interferometers

Development of Machine Learning Based Analytical Tools for Pavement Performance Assessment and Crack Detection

Pavement Management System (PMS) analytical tools mainly consist of pavement condition investigation and evaluation tools, pavement condition rating and assessment tools, pavement performance prediction tools, treatment prioritizations and implementation tools. The effectiveness of a PMS highly depends on the efficiency and reliability of its pavement condition evaluation tools. Traditionally, pavement condition investigation and evaluation practices are based on manual distress surveys and performance level assessments, which have been blamed for low efficiency low reliability. Those kinds of manually surveys are labor intensive and unsafe due to proximity to live traffic conditions. Meanwhile, the accuracy can be lower due to the subjective nature of the evaluators. Considering these factors, semiautomated and automated pavement condition evaluation tools had been developed for several years. In current years, it is undoubtable that highly advanced computerized technologies have resulted successful applications in diverse engineering fields. Therefore, these techniques can be successfully incorporated into pavement condition evaluation distress detection, the analytical tools can improve the performance of existing PMSs. Hence, this research aims to bridge the gaps between highly advanced Machine Learning Techniques (MLTs) and the existing analytical tools of current PMSs. The research outputs intend to provide pavement condition evaluation tools that meet the requirement of high efficiency, accuracy, and reliability. To achieve the objectives of this research, six pavement damage condition and performance evaluation methodologies are developed. The roughness condition of pavement surface directly influences the riding quality of the users. International Roughness Index (IRI) is used worldwide by research institutions, pavement condition evaluation and management agencies to evaluate the roughness condition of the pavement. IRI is a time-dependent variable which generally tends to increase with the increase of the pavement service life. In this consideration, a multi-granularity fuzzy time series analysis based IRI prediction model is developed. Meanwhile, Particle Swarm Optimization (PSO) method is used for model optimization to obtain satisfactory IRI prediction results. Historical IRI data extracted from the InfoPave website have been used for training and testing the model. Experiment results proved the effectiveness of this method. Automated pavement condition evaluation tools can provide overall performance indices, which can then be used for treatment planning. The calculations of those performance indices are required for surface distress level and roughness condition evaluations. However, pavement surface roughness conditions are hard to obtain from surface image indicators. With this consideration, an image indicators-based pavement roughness and the overall performance prediction tools are developed. The state-of-the-art machine learning technique, XGBoost, is utilized as the main method in model training, validating and testing. In order to find the dominant image indicators that influence the pavement roughness condition and the overall performance conditions, the comprehensive pavement performance evaluation data collected by ARAN 900 are analyzed. Back Propagation Neural Network (BPNN) is used to develop the performance prediction models. On this basis, the mean important values (MIVs) for each input factor are calculated to evaluate the contributions of the input indicators. It has been observed that indicators of the wheel path cracking have the highest MIVs, which emphasizes the importance of cracking-focused maintenance treatments. The same issue is also found that current automated pavement condition evaluation systems only include the analysis of pavement surface distresses, without considering the structural capacity of the actual pavement. Hence, the structural performance analysis-based pavement performance prediction tools are developed using the Support Vector Machines (SVMs). To guarantee the overall performance of the proposed methodologies, heuristic methods including Genetic Algorithm (GA) and Particle Swarm Optimization (PSO) are selected to optimize the model. The experiments results show a promising future of machine learning based pavement structural performance prediction. Automated pavement condition analyzers usually detect pavement surface distress through the collected pavement surface images. Then, distress types, severities, quantities, and other parameters are calculated for the overall performance index calculation. Cracks are one of the most important pavement surface distresses that should be quantified. Traditional approaches are less accurate and efficient in locating, counting and quantifying various types of cracks initialed on the pavement surface. An integrated Crack Deep Net (CrackDN) is developed based on deep learning technologies. Through model training, validation and testing, it has proved that CrackDN can detect pavement surface cracks on complex background with high accuracy. Moreover, the combination of box-level pavement crack locating, and pixel-level crack calculation can achieve comprehensive crack analysis. Thereby, more effective maintenance treatments can be assigned. Hence, a methodology regarding pixel-level crack detection which is called CrackU-net, is proposed. CrackU-net is composed of several convolutional, maxpooling, and up-convolutional layers. The model is developed based on the innovations of deep learning-based segmentation. Pavement crack data are collected by multiple devices, including automated pavement condition survey vehicles, smartphones, and action cameras. The proposed CrackU-net is tested on a separate crack image set which has not been used for training the model. The results demonstrate a promising future of use in the PMSs. Finally, the proposed toolboxes are validated through comparative experiments in terms of accuracy (precision, recall, and F-measure) and error levels. The accuracies of all those models are higher than 0.9 and the errors are lower than 0.05. Meanwhile, the findings of this research suggest that the wheel path cracking should be a priority when conducting maintenance activity planning. Benefiting from the highly advanced machine learning technologies, pavement roughness condition and the overall performance levels have a promising future of being predicted by extraction of the image indicators. Moreover, deep learning methods can be utilized to achieve both box-level and pixel-level pavement crack detection with satisfactory performance. Therefore, it is suggested that those state-of-the-art toolboxes be integrated into current PMSs to upgrade their service levels