Deep Demosaicing for Polarimetric Filter Array Cameras

Polarisation Filter Array (PFA) cameras allow the analysis of light polarisation state in a simple and cost-effective manner. Such filter arrays work as the Bayer pattern for colour cameras, sharing similar advantages and drawbacks. Among the others, the raw image must be demosaiced considering the local variations of the PFA and the characteristics of the imaged scene. Non-linear effects, like the cross-talk among neighbouring pixels, are difficult to explicitly model and suggest the potential advantage of a data-driven learning approach. However, the PFA cannot be removed from the sensor, making it difficult to acquire the ground-truth polarization state for training. In this work we propose a novel CNN-based model which directly demosaics the raw camera image to a per-pixel Stokes vector. Our contribution is twofold. First, we propose a network architecture composed by a sequence of Mosaiced Convolutions operating coherently with the local arrangement of the different filters. Second, we introduce a new method, employing a consumer LCD screen, to effectively acquire real-world data for training. The process is designed to be invariant by monitor gamma and external lighting conditions. We extensively compared our method against algorithmic and learning-based demosaicing techniques, obtaining a consistently lower error especially in terms of polarisation angle

Differentiable Display Photometric Stereo

Photometric stereo leverages variations in illumination conditions to reconstruct per-pixel surface normals. The concept of display photometric stereo, which employs a conventional monitor as an illumination source, has the potential to overcome limitations often encountered in bulky and difficult-to-use conventional setups. In this paper, we introduce Differentiable Display Photometric Stereo (DDPS), a method designed to achieve high-fidelity normal reconstruction using an off-the-shelf monitor and camera. DDPS addresses a critical yet often neglected challenge in photometric stereo: the optimization of display patterns for enhanced normal reconstruction. We present a differentiable framework that couples basis-illumination image formation with a photometric-stereo reconstruction method. This facilitates the learning of display patterns that leads to high-quality normal reconstruction through automatic differentiation. Addressing the synthetic-real domain gap inherent in end-to-end optimization, we propose the use of a real-world photometric-stereo training dataset composed of 3D-printed objects. Moreover, to reduce the ill-posed nature of photometric stereo, we exploit the linearly polarized light emitted from the monitor to optically separate diffuse and specular reflections in the captured images. We demonstrate that DDPS allows for learning display patterns optimized for a target configuration and is robust to initialization. We assess DDPS on 3D-printed objects with ground-truth normals and diverse real-world objects, validating that DDPS enables effective photometric-stereo reconstruction

Imaging Polarimetry with Polarization-Sensitive Focal Plane Arrays

Polarization is an intrinsic property of light, like frequency or coherence. Humans have long benefited from our ability to distinguish light of different frequency based on its color. However, our eyes are not sensitive to the polarization of light. Devices to measure polarization are relatively rare and expertise in polarimetry even more so. Polarization sensors based on micropolarizer arrays appear to be the first devices capable of bringing polarimetric capability to a wide range of applications. Whereas previous polarimeters were built to perform very specific measurements, the same micropolarizer-based camera can be used on a telescope, a microscope, or with a conventional camera lens. In this work, I investigate the operating principles of micropolarizer arrays using high resolution 3D simulations and describe several strategies to fabricate and characterize micropolarizer-based imaging polarimeters. Furthermore, I show how to incorporate the device characterization into a calibrated demodulation procedure to extract polarimetric quantities from the raw pixel intensities. As part of this effort, I show how the measured sensor properties, like pixel throughput and contrast ratio, can be used to construct a software model to produce synthetic observations of various scenes. These synthetic data are a powerful tool to study the many effects which can give rise to systematic and/or random errors during the data analysis process. Finally, I present the polarimetry performed on several astronomical sources using the RIT Polarization Imaging Camera and compare my results to previous measurements made with conventional polarimeters. Using the current calibration of the RIT Polarization Imaging Camera, I was able to achieve a polarimetric accuracy of ~0.3% in images of extended objects and unresolved sources

Polarisation vision: overcoming challenges of working with a property of light we barely see.

In recent years, the study of polarisation vision in animals has seen numerous breakthroughs, not just in terms of what is known about the function of this sensory ability, but also in the experimental methods by which polarisation can be controlled, presented and measured. Once thought to be limited to only a few animal species, polarisation sensitivity is now known to be widespread across many taxonomic groups, and advances in experimental techniques are, in part, responsible for these discoveries. Nevertheless, its study remains challenging, perhaps because of our own poor sensitivity to the polarisation of light, but equally as a result of the slow spread of new practices and methodological innovations within the field. In this review, we introduce the most important steps in designing and calibrating polarised stimuli, within the broader context of areas of current research and the applications of new techniques to key questions. Our aim is to provide a constructive guide to help researchers, particularly those with no background in the physics of polarisation, to design robust experiments that are free from confounding factors

Random Transformations Of Optical Fields And Applications

The interaction of optical waves with material systems often results in complex, seemingly random fields. Because the fluctuations of such fields are typically difficult to analyze, they are regarded as noise to be suppressed. Nevertheless, in many cases the fluctuations of the field result from a linear and deterministic, albeit complicated, interaction between the optical field and the scattering system. As a result, linear systems theory (LST) can be used to frame the scattering problem and highlight situations in which useful information can be extracted from the fluctuations of the scattered field. Three fundamental problems can be posed in LST regardless of the nature of the system: one direct and two inverse problems. The direct problem attempts to predict the response of a known system to a known input. The problem may be simple enough to admit analytical solutions as in the case of homogeneous materials, phase and amplitude screens, and weakly scattering materials; or the problem may require the use of numerical techniques. This dissertation will focus on the two inverse problems, namely the determination of either the excitation field or the scattering system. Traditionally, the excitation determination problem has relied on designing optical systems that respond to the property of interest in a simple, easily quantified way. For example, gratings can be used to map wavelength onto direction of propagation while waveplates and polarizers can map polarization properties onto intensity. The primary difficulty with directly applying the concepts of LST to scattering systems iv is that, while the outputs are still combinations of the inputs, they are not ``simple\u27\u27 combinations such as Fourier transforms or spatially dispersed spectral components of the input spectrum. Instead, the scattered field can be thought of as a massive sampling and mixing of the excitation field. This dissertation will show that such complicated sampling functions can be characterized and that the corresponding scattering medium can then be used as an optical device such as a lens, polarimeter, or spectrometer. The second inverse problem, system determination, is often more difficult because the problem itself may be ill-posed. For scattering systems that are dominated by low-order scattering, the statistical properties of the scattered light may serve as a fingerprint for material discrimination; however, in many situations, the statistical properties of the output do not depend on the material properties. Rather than analyzing the scattered field from one realization of the random interaction, several measurement techniques have been developed that attempt to extract information about the material system from modifications of the scattered field in response to changes in either the excitation or the intrinsic dynamics of the medium itself. One such technique is dynamic light scattering. This dissertation includes an extension to this method that allows for a polarimetric measurement of the scattered light using a reference beam with controllable polarization. Another system determination problem relates to imaging the reflectivity of a target that is being randomly illuminated. It will be demonstrated that an approach based on the correlation between the integrated scattered intensity and the corresponding illumination intensity distribution can prove superior to standard imaging microscop