High Dynamic Range Spectral Imaging Pipeline For Multispectral Filter Array Cameras

Spectral filter arrays imaging exhibits a strong similarity with color filter arrays. This permits us to embed this technology in practical vision systems with little adaptation of the existing solutions. In this communication, we define an imaging pipeline that permits high dynamic range (HDR)-spectral imaging, which is extended from color filter arrays. We propose an implementation of this pipeline on a prototype sensor and evaluate the quality of our implementation results on real data with objective metrics and visual examples. We demonstrate that we reduce noise, and, in particular we solve the problem of noise generated by the lack o

Deep Learning Methods for Calibrated Photometric Stereo and Beyond

Photometric stereo recovers the surface normals of an object from multiple images with varying shading cues, i.e., modeling the relationship between surface orientation and intensity at each pixel. Photometric stereo prevails in superior per-pixel resolution and fine reconstruction details. However, it is a complicated problem because of the non-linear relationship caused by non-Lambertian surface reflectance. Recently, various deep learning methods have shown a powerful ability in the context of photometric stereo against non-Lambertian surfaces. This paper provides a comprehensive review of existing deep learning-based calibrated photometric stereo methods. We first analyze these methods from different perspectives, including input processing, supervision, and network architecture. We summarize the performance of deep learning photometric stereo models on the most widely-used benchmark data set. This demonstrates the advanced performance of deep learning-based photometric stereo methods. Finally, we give suggestions and propose future research trends based on the limitations of existing models.Comment: 19 pages, 11 figures, 4 table

Evaluation and optimal design of spectral sensitivities for digital color imaging

The quality of an image captured by color imaging system primarily depends on three factors: sensor spectral sensitivity, illumination and scene. While illumination is very important to be known, the sensitivity characteristics is critical to the success of imaging applications, and is necessary to be optimally designed under practical constraints. The ultimate image quality is judged subjectively by human visual system. This dissertation addresses the evaluation and optimal design of spectral sensitivity functions for digital color imaging devices. Color imaging fundamentals and device characterization are discussed in the first place. For the evaluation of spectral sensitivity functions, this dissertation concentrates on the consideration of imaging noise characteristics. Both signal-independent and signal-dependent noises form an imaging noise model and noises will be propagated while signal is processed. A new colorimetric quality metric, unified measure of goodness (UMG), which addresses color accuracy and noise performance simultaneously, is introduced and compared with other available quality metrics. Through comparison, UMG is designated as a primary evaluation metric. On the optimal design of spectral sensitivity functions, three generic approaches, optimization through enumeration evaluation, optimization of parameterized functions, and optimization of additional channel, are analyzed in the case of the filter fabrication process is unknown. Otherwise a hierarchical design approach is introduced, which emphasizes the use of the primary metric but the initial optimization results are refined through the application of multiple secondary metrics. Finally the validity of UMG as a primary metric and the hierarchical approach are experimentally tested and verified

Color image-based shape reconstruction of multi-color objects under general illumination conditions

Humans have the ability to infer the surface reflectance properties and three-dimensional shape of objects from two-dimensional photographs under simple and complex illumination fields. Unfortunately, the reported algorithms in the area of shape reconstruction require a number of simplifying assumptions that result in poor performance in uncontrolled imaging environments. Of all these simplifications, the assumptions of non-constant surface reflectance, globally consistent illumination, and multiple surface views are the most likely to be contradicted in typical environments. In this dissertation, three automatic algorithms for the recovery of surface shape given non-constant reflectance using a single-color image acquired are presented. In addition, a novel method for the identification and removal of shadows from simple scenes is discussed.In existing shape reconstruction algorithms for surfaces of constant reflectance, constraints based on the assumed smoothness of the objects are not explicitly used. Through Explicit incorporation of surface smoothness properties, the algorithms presented in this work are able to overcome the limitations of the previously reported algorithms and accurately estimate shape in the presence of varying reflectance. The three techniques developed for recovering the shape of multi-color surfaces differ in the method through which they exploit the surface smoothness property. They are summarized below:• Surface Recovery using Pre-Segmentation - this algorithm pre-segments the image into distinct color regions and employs smoothness constraints at the color-change boundaries to constrain and recover surface shape. This technique is computationally efficient and works well for images with distinct color regions, but does not perform well in the presence of high-frequency color textures that are difficult to segment.iv• Surface Recovery via Normal Propagation - this approach utilizes local gradient information to propagate a smooth surface solution from points of known orientation. While solution propagation eliminates the need for color-based image segmentation, the quality of the recovered surface can be degraded by high degrees of image noise due to reliance on local information.• Surface Recovery by Global Variational Optimization - this algorithm utilizes a normal gradient smoothness constraint in a non-linear optimization strategy, to iteratively solve for the globally optimal object surface. Because of its global nature, this approach is much less sensitive to noise than the normal propagation is, but requires significantly more computational resources.Results acquired through application of the above algorithms to various synthetic and real image data sets are presented for qualitative evaluation. A quantitative analysis of the algorithms is also discussed for quadratic shapes. The robustness of the three approaches to factors such as segmentation error and random image noise is also explored

Estimation of illuminants from color signals of illuminated objects

Color constancy is the ability of the human visual systems to discount the effect of the illumination and to assign approximate constant color descriptions to objects. This ability has long been studied and widely applied to many areas such as color reproduction and machine vision, especially with the development of digital color processing. This thesis work makes some improvements in illuminant estimation and computational color constancy based on the study and testing of existing algorithms. During recent years, it has been noticed that illuminant estimation based on gamut comparison is efficient and simple to implement. Although numerous investigations have been done in this field, there are still some deficiencies. A large part of this thesis has been work in the area of illuminant estimation through gamut comparison. Noting the importance of color lightness in gamut comparison, and also in order to simplify three-dimensional gamut calculation, a new illuminant estimation method is proposed through gamut comparison at separated lightness levels. Maximum color separation is a color constancy method which is based on the assumption that colors in a scene will obtain the largest gamut area under white illumination. The method was further derived and improved in this thesis to make it applicable and efficient. In addition, some intrinsic questions in gamut comparison methods, for example the relationship between the color space and the application of gamut or probability distribution, were investigated. Color constancy methods through spectral recovery have the limitation that there is no effective way to confine the range of object spectral reflectance. In this thesis, a new constraint on spectral reflectance based on the relative ratios of the parameters from principal component analysis (PCA) decomposition is proposed. The proposed constraint was applied to illuminant detection methods as a metric on the recovered spectral reflectance. Because of the importance of the sensor sensitivities and their wide variation, the influence from the sensor sensitivities on different kinds of illuminant estimation methods was also studied. Estimation method stability to wrong sensor information was tested, suggesting the possible solution to illuminant estimation on images with unknown sources. In addition, with the development of multi-channel imaging, some research on illuminant estimation for multi-channel images both on the correlated color temperature (CCT) estimation and the illuminant spectral recovery was performed in this thesis. All the improvement and new proposed methods in this thesis are tested and compared with those existing methods with best performance, both on synthetic data and real images. The comparison verified the high efficiency and implementation simplicity of the proposed methods

Practical Spectral Diffraction Imaging

Spectral imaging has proved to be an invaluable tool in many industries and research areas. Spectral images can be used for material identification, food quality monitoring, as an aid in cancer diagnostics, the analysis of paintings, and land cover type detection, to name a few tasks. Despite the high utility of spectral images, spectral imaging devices are often expensive or technically impaired for many use cases and hence they are not used as widely as they could. The computed tomography imaging spectrometer (CTIS) is a special type of spectral imaging device capable of capturing snapshot spectral imaging using a commodity digital camera, together with a modified objective lens containing a diffraction grating. This offers a cost effective and fast way of spectral imaging that is potentially accessible to a large audience. In this thesis, we introduce the diffraction imaging add-on, an alternative design to the CTIS, and several algorithms for spectral image production and device calibration. We make the following four contributions: First, we discuss the salient aspects of the design of a diffraction imaging add-on together with two design examples for two cameras, which can be 3D printed with a commodity-grade 3D printer. Second, we present three use cases for diffraction imaging, which also discuss related practical issues of spectral imaging by diffraction imaging. Third, we introduce calibration procedures needed to produce physically correct spectra estimates. Calibration procedures are supported from the perspective of the three use cases. Lastly, we discuss possible shortcomings of the methods presented in the use cases and present improvement suggestions.Spektrikuvaus on osoittautunut korvaamattomaksi työkaluksi monilla toimialoilla ja tutkimusalueilla. Spektrikuvia voidaan esimerkiksi käyttää materiaalin tunnistamiseen, elintarvikkeiden laadun seurantaan, apuna syövän diagnostiikassa, maalausten analysoinnissa ja maanpeitetyypin havaitsemisessa. Vaikka spektrikuvien hyöty on kiistaton, niitä ei käytetä niin laajalti kuin voisi. Tämä johtuu spektrikuvantamislaitteiden korkeasta hankintahinnasta ja niiden teknisistä rajoitteista monissa käyttötapauksissa. Spektrikuvia voidaan muodostaa myös hyödyntämällä tietokonetomografiaa (computed tomography imaging spectrometer, CTIS), eli yhdistelmää, jossa kuvantaminen suoritetaan digitaalikameran avulla yhdessä modifioidun objektiivilinssin kanssa, joka sisältää diffraktiohilan ja itse spektri muodostetaan laskennallisilla menetelmillä. Tämä tarjoaa kustannustehokkaan ja nopean tavan spektrikuvaukseen, joka on mahdollisesti suuren yleisön käytettävissä. Työssä käsittelemme diffraktiokuvantamisen lisäosaa, vaihtoehtoista fyysistä mallia CTIS:lle sekä useita algoritmeja spektrikuvan tuottamiseen ja laitekalibrointiin. Ensiksi käsittelemme diffraktiokuvauslisäosan suunnittelun keskeisiä näkökohtia sekä kahta suunnitteluesimerkkiä kahdelle kameralle, jotka voidaan 3D-tulostaa kuluttajille suunnatuilla 3D-tulostimilla. Toiseksi esittelemme kolme diffraktiokuvauksen käyttötapausta, joissa käsitellään myös diffraktiokuvantamisen spektrikuvauksen käytännön kysymyksiä. Kolmanneksi esittelemme kalibrointimenettelyt, joita tarvitaan fyysisesti oikeiden spektriestimaattien tuottamiseksi. Kalibrointimenettelyjä tuetaan kolmen käyttötapauksen näkökulmasta. Esitetyt menetelmät on osoitettu toimiksi rajoitetuissa olosuhteissa, joten lopuksi pohditaan käyttötapauksissa esitettyjen menetelmien saattamista käytännössä toimiviksi eri käyttötarkoituksiin ja olosuhteisiin

Height from Photometric Ratio with Model-based Light Source Selection

In this paper, we present a photometric stereo algorithm for estimating surface height. We follow recent work that uses photometric ratios to obtain a linear formulation relating surface gradients and image intensity. Using smoothed finite difference approximations for the surface gradient, we are able to express surface height recovery as a linear least squares problem that is large but sparse. In order to make the method practically useful, we combine it with a model-based approach that excludes observations which deviate from the assumptions made by the image formation model. Despite its simplicity, we show that our algorithm provides surface height estimates of a high quality even for objects with highly non-Lambertian appearance. We evaluate the method on both synthetic images with ground truth and challenging real images that contain strong specular reflections and cast shadows