Data-Driven Radiometric Photo-Linearization

In computer vision and computer graphics, a photograph is often considered a photometric representation of a scene. However, for most camera models, the relation between recorded pixel value and the amount of light received on the sensor is not linear. This non-linear relationship is modeled by the camera response function which maps the scene radiance to the image brightness. This non-linear transformation is unknown, and it can only be recovered via a rigorous radiometric calibration process. Classic radiometric calibration methods typically estimate a camera response function from an exposure stack (i.e., an image sequence captured with different exposures from the same viewpoint and time). However, for photographs in large image collections for which we do not have control over the capture process, traditional radiometric calibration methods cannot be applied. This thesis details two novel data-driven radiometric photo-linearization methods suit- able for photographs captured with unknown camera settings and under uncontrolled conditions. First, a novel example-based radiometric linearization method is pro- posed, that takes as input a radiometrically linear photograph of a scene (i.e., exemplar), and a standard (radiometrically uncalibrated) image of the same scene potentially from a different viewpoint and/or under different lighting, and which produces a radiometrically linear version of the latter. Key to this method is the observation that for many patches, their change in appearance (from different viewpoints and lighting) forms a 1D linear subspace. This observation allows the problem to be reformulated in a form similar to classic radiometric calibration from an exposure stack. In addition, practical solutions are proposed to automatically select and align the best matching patches/correspondences between the two photographs, and to robustly reject outliers/unreliable matches. Second, CRF-net (or Camera Response Function net), a robust single image radiometric calibration method based on convolutional neural net- works (CNNs) is presented. The proposed network takes as input a single photograph, and outputs an estimate of the camera response function in the form of the 11 PCA coefficients for the EMoR camera response model. CRF-net is able to accurately recover the camera response function from a single photograph under a wide range of conditions

OSPC: Online Sequential Photometric Calibration

Photometric calibration is essential to many computer vision applications. One of its key benefits is enhancing the performance of Visual SLAM, especially when it depends on a direct method for tracking, such as the standard KLT algorithm. Another advantage could be in retrieving the sensor irradiance values from measured intensities, as a pre-processing step for some vision algorithms, such as shape-from-shading. Current photometric calibration systems rely on a joint optimization problem and encounter an ambiguity in the estimates, which can only be resolved using ground truth information. We propose a novel method that solves for photometric parameters using a sequential estimation approach. Our proposed method achieves high accuracy in estimating all parameters; furthermore, the formulations are linear and convex, which makes the solution fast and suitable for online applications. Experiments on a Visual Odometry system validate the proposed method and demonstrate its advantages

Shadow Estimation Method for "The Episolar Constraint: Monocular Shape from Shadow Correspondence"

Recovering shadows is an important step for many vision algorithms. Current approaches that work with time-lapse sequences are limited to simple thresholding heuristics. We show these approaches only work with very careful tuning of parameters, and do not work well for long-term time-lapse sequences taken over the span of many months. We introduce a parameter-free expectation maximization approach which simultaneously estimates shadows, albedo, surface normals, and skylight. This approach is more accurate than previous methods, works over both very short and very long sequences, and is robust to the effects of nonlinear camera response. Finally, we demonstrate that the shadow masks derived through this algorithm substantially improve the performance of sun-based photometric stereo compared to earlier shadow mask estimation

Defect and thickness inspection system for cast thin films using machine vision and full-field transmission densitometry

Quick mass production of homogeneous thin film material is required in paper, plastic, fabric, and thin film industries. Due to the high feed rates and small thicknesses, machine vision and other nondestructive evaluation techniques are used to ensure consistent, defect-free material by continuously assessing post-production quality. One of the fastest growing inspection areas is for 0.5-500 micrometer thick thin films, which are used for semiconductor wafers, amorphous photovoltaics, optical films, plastics, and organic and inorganic membranes. As a demonstration application, a prototype roll-feed imaging system has been designed to inspect high-temperature polymer electrolyte membrane (PEM), used for fuel cells, after being die cast onto a moving transparent substrate. The inspection system continuously detects thin film defects and classifies them with a neural network into categories of holes, bubbles, thinning, and gels, with a 1.2% false alarm rate, 7.1% escape rate, and classification accuracy of 96.1%. In slot die casting processes, defect types are indicative of a misbalance in the mass flow rate and web speed; so, based on the classified defects, the inspection system informs the operator of corrective adjustments to these manufacturing parameters. Thickness uniformity is also critical to membrane functionality, so a real-time, full-field transmission densitometer has been created to measure the bi-directional thickness profile of the semi-transparent PEM between 25-400 micrometers. The local thickness of the 75 mm x 100 mm imaged area is determined by converting the optical density of the sample to thickness with the Beer-Lambert law. The PEM extinction coefficient is determined to be 1.4 D/mm and the average thickness error is found to be 4.7%. Finally, the defect inspection and thickness profilometry systems are compiled into a specially-designed graphical user interface for intuitive real-time operation and visualization.M.S.Committee Chair: Tequila Harris; Committee Member: Levent Degertekin; Committee Member: Wayne Dale

Pre-Flight Calibration of the Mars 2020 Rover Mastcam Zoom (Mastcam-Z) Multispectral, Stereoscopic Imager

The NASA Perseverance rover Mast Camera Zoom (Mastcam-Z) system is a pair of zoomable, focusable, multi-spectral, and color charge-coupled device (CCD) cameras mounted on top of a 1.7 m Remote Sensing Mast, along with associated electronics and two calibration targets. The cameras contain identical optical assemblies that can range in focal length from 26 mm (25.5∘×19.1∘ FOV) to 110 mm (6.2∘×4.2∘ FOV) and will acquire data at pixel scales of 148-540 μm at a range of 2 m and 7.4-27 cm at 1 km. The cameras are mounted on the rover’s mast with a stereo baseline of 24.3±0.1 cm and a toe-in angle of 1.17±0.03∘ (per camera). Each camera uses a Kodak KAI-2020 CCD with 1600×1200 active pixels and an 8 position filter wheel that contains an IR-cutoff filter for color imaging through the detectors’ Bayer-pattern filters, a neutral density (ND) solar filter for imaging the sun, and 6 narrow-band geology filters (16 total filters). An associated Digital Electronics Assembly provides command data interfaces to the rover, 11-to-8 bit companding, and JPEG compression capabilities. Herein, we describe pre-flight calibration of the Mastcam-Z instrument and characterize its radiometric and geometric behavior. Between April 26thth and May 9thth, 2019, ∼45,000 images were acquired during stand-alone calibration at Malin Space Science Systems (MSSS) in San Diego, CA. Additional data were acquired during Assembly Test and Launch Operations (ATLO) at the Jet Propulsion Laboratory and Kennedy Space Center. Results of the radiometric calibration validate a 5% absolute radiometric accuracy when using camera state parameters investigated during testing. When observing using camera state parameters not interrogated during calibration (e.g., non-canonical zoom positions), we conservatively estimate the absolute uncertainty to be 0.2 design requirement. We discuss lessons learned from calibration and suggest tactical strategies that will optimize the quality of science data acquired during operation at Mars. While most results matched expectations, some surprises were discovered, such as a strong wavelength and temperature dependence on the radiometric coefficients and a scene-dependent dynamic component to the zero-exposure bias frames. Calibration results and derived accuracies were validated using a Geoboard target consisting of well-characterized geologic samples

Measuring atmospheric scattering from digital images of urban scenery using temporal polarization-based vision

Suspended atmospheric particles (particulate matter) are a form of air pollution that visually degrades urban scenery and is hazardous to human health and the environment. Current environmental monitoring devices are limited in their capability of measuring average particulate matter (PM) over large areas. Quantifying the visual effects of haze in digital images of urban scenery and correlating these effects to PM levels is a vital step in more practically monitoring our environment. Current image haze extraction algorithms remove all the haze from the scene and hence produce unnatural scenes for the sole purpose of enhancing vision. We present two algorithms which bridge the gap between image haze extraction and environmental monitoring. We provide a means of measuring atmospheric scattering from images of urban scenery by incorporating temporal knowledge. In doing so, we also present a method of recovering an accurate depthmap of the scene and recovering the scene without the visual effects of haze. We compare our algorithm to three known haze removal methods from the perspective of measuring atmospheric scattering, measuring depth and dehazing. The algorithms are composed of an optimization over a model of haze formation in images and an optimization using the constraint of constant depth over a sequence of images taken over time. These algorithms not only measure atmospheric scattering, but also recover a more accurate depthmap and dehazed image. The measurements of atmospheric scattering this research produces, can be directly correlated to PM levels and therefore pave the way to monitoring the health of the environment by visual means. Accurate atmospheric sensing from digital images is a challenging and under-researched problem. This work provides an important step towards a more practical and accurate visual means of measuring PM from digital images

Registration for Optical Multimodal Remote Sensing Images Based on FAST Detection,Window Selection, and Histogram Specification

In recent years, digital frame cameras have been increasingly used for remote sensing applications. However, it is always a challenge to align or register images captured with different cameras or different imaging sensor units. In this research, a novel registration method was proposed. Coarse registration was first applied to approximately align the sensed and reference images. Window selection was then used to reduce the search space and a histogram specification was applied to optimize the grayscale similarity between the images. After comparisons with other commonly-used detectors, the fast corner detector, FAST (Features from Accelerated Segment Test), was selected to extract the feature points. The matching point pairs were then detected between the images, the outliers were eliminated, and geometric transformation was performed. The appropriate window size was searched and set to one-tenth of the image width. The images that were acquired by a two-camera system, a camera with five imaging sensors, and a camera with replaceable filters mounted on a manned aircraft, an unmanned aerial vehicle, and a ground-based platform, respectively, were used to evaluate the performance of the proposed method. The image analysis results showed that, through the appropriate window selection and histogram specification, the number of correctly matched point pairs had increased by 11.30 times, and that the correct matching rate had increased by 36%, compared with the results based on FAST alone. The root mean square error (RMSE) in the x and y directions was generally within 0.5 pixels. In comparison with the binary robust invariant scalable keypoints (BRISK), curvature scale space (CSS), Harris, speed up robust features (SURF), and commercial software ERDAS and ENVI, this method resulted in larger numbers of correct matching pairs and smaller, more consistent RMSE. Furthermore, it was not necessary to choose any tie control points manually before registration. The results from this study indicate that the proposed method can be effective for registering optical multimodal remote sensing images that have been captured with different imaging sensors