Image Quality Is Not All You Want: Task-Driven Lens Design for Image Classification

In computer vision, it has long been taken for granted that high-quality images obtained through well-designed camera lenses would lead to superior results. However, we find that this common perception is not a "one-size-fits-all" solution for diverse computer vision tasks. We demonstrate that task-driven and deep-learned simple optics can actually deliver better visual task performance. The Task-Driven lens design approach, which relies solely on a well-trained network model for supervision, is proven to be capable of designing lenses from scratch. Experimental results demonstrate the designed image classification lens (``TaskLens'') exhibits higher accuracy compared to conventional imaging-driven lenses, even with fewer lens elements. Furthermore, we show that our TaskLens is compatible with various network models while maintaining enhanced classification accuracy. We propose that TaskLens holds significant potential, particularly when physical dimensions and cost are severely constrained.Comment: Use an image classification network to supervise the lens design from scratch. The final designs can achieve higher accuracy with fewer optical element

Computational Spectral Imaging: A Contemporary Overview

Spectral imaging collects and processes information along spatial and spectral coordinates quantified in discrete voxels, which can be treated as a 3D spectral data cube. The spectral images (SIs) allow identifying objects, crops, and materials in the scene through their spectral behavior. Since most spectral optical systems can only employ 1D or maximum 2D sensors, it is challenging to directly acquire the 3D information from available commercial sensors. As an alternative, computational spectral imaging (CSI) has emerged as a sensing tool where the 3D data can be obtained using 2D encoded projections. Then, a computational recovery process must be employed to retrieve the SI. CSI enables the development of snapshot optical systems that reduce acquisition time and provide low computational storage costs compared to conventional scanning systems. Recent advances in deep learning (DL) have allowed the design of data-driven CSI to improve the SI reconstruction or, even more, perform high-level tasks such as classification, unmixing, or anomaly detection directly from 2D encoded projections. This work summarises the advances in CSI, starting with SI and its relevance; continuing with the most relevant compressive spectral optical systems. Then, CSI with DL will be introduced, and the recent advances in combining the physical optical design with computational DL algorithms to solve high-level tasks

Snapshot Multispectral Imaging Using a Diffractive Optical Network

Multispectral imaging has been used for numerous applications in e.g., environmental monitoring, aerospace, defense, and biomedicine. Here, we present a diffractive optical network-based multispectral imaging system trained using deep learning to create a virtual spectral filter array at the output image field-of-view. This diffractive multispectral imager performs spatially-coherent imaging over a large spectrum, and at the same time, routes a pre-determined set of spectral channels onto an array of pixels at the output plane, converting a monochrome focal plane array or image sensor into a multispectral imaging device without any spectral filters or image recovery algorithms. Furthermore, the spectral responsivity of this diffractive multispectral imager is not sensitive to input polarization states. Through numerical simulations, we present different diffractive network designs that achieve snapshot multispectral imaging with 4, 9 and 16 unique spectral bands within the visible spectrum, based on passive spatially-structured diffractive surfaces, with a compact design that axially spans ~72 times the mean wavelength of the spectral band of interest. Moreover, we experimentally demonstrate a diffractive multispectral imager based on a 3D-printed diffractive network that creates at its output image plane a spatially-repeating virtual spectral filter array with 2x2=4 unique bands at terahertz spectrum. Due to their compact form factor and computation-free, power-efficient and polarization-insensitive forward operation, diffractive multispectral imagers can be transformative for various imaging and sensing applications and be used at different parts of the electromagnetic spectrum where high-density and wide-area multispectral pixel arrays are not widely available.Comment: 24 Pages, 9 Figure

On design of hybrid diffractive optics for achromatic extended depth-of-field (EDoF) RGB imaging

A hybrid imaging system is a simultaneous physical arrangement of a refractive lens and a multilevel phase mask (MPM) as a diffractive optical element (DOE). The favorable properties of the hybrid setup are improved extended-depth-of-field (EDoF) imaging and low chromatic aberrations. We built a fully differentiable image formation model in order to use neural network techniques to optimize imaging. At the first stage, the design framework relies on the model-based approach with numerical simulation and end-to-end joint optimization of both MPM and imaging algorithms. In the second stage, MPM is fixed as found at the first stage, and the image processing is optimized experimentally using the CNN learning-based approach with MPM implemented by a spatial light modulator. The paper is concentrated on a comparative analysis of imaging accuracy and quality for design with various basic optical parameters: aperture size, lens focal length, and distance between MPM and sensor. We point out that the varying aperture size, lens focal length, and distance between MPM and sensor are for the first time considered for end-to-end optimization of EDoF. We numerically and experimentally compare the designs for visible wavelength interval [400-700]nm and the following EDoF ranges: [0.5-100]m for simulations and [0.5-1.9]m for experimental tests. This study concerns an application of hybrid optics for compact cameras with aperture [5-9] mm and distance between MPM and sensor [3-10]mm.Comment: 16 pages, 11 figures, 1 tabl

Power-Balanced Hybrid Optics Boosted Design for Achromatic Extended-Depth-of-Field Imaging via Optimized Mixed OTF

The power-balanced hybrid optical imaging system is a special design of a diffractive computational camera, introduced in this paper, with image formation by a refractive lens and Multilevel Phase Mask (MPM). This system provides a long focal depth with low chromatic aberrations thanks to MPM and a high energy light concentration due to the refractive lens. We introduce the concept of optical power balance between the lens and MPM which controls the contribution of each element to modulate the incoming light. Additional unique features of our MPM design are the inclusion of quantization of the MPM's shape on the number of levels and the Fresnel order (thickness) using a smoothing function. To optimize optical power-balance as well as the MPM, we build a fully-differentiable image formation model for joint optimization of optical and imaging parameters for the proposed camera using Neural Network techniques. Additionally, we optimize a single Wiener-like optical transfer function (OTF) invariant to depth to reconstruct a sharp image. We numerically and experimentally compare the designed system with its counterparts, lensless and just-lens optical systems, for the visible wavelength interval (400-700)nm and the depth-of-field range (0.5-$\infty$m for numerical and 0.5-2m for experimental). The attained results demonstrate that the proposed system equipped with the optimal OTF overcomes its counterparts (even when they are used with optimized OTF) in terms of reconstruction quality for off-focus distances. The simulation results also reveal that optimizing the optical power-balance, Fresnel order, and the number of levels parameters are essential for system performance attaining an improvement of up to 5dB of PSNR using the optimized OTF compared with its counterpart lensless setup.Comment: 18 pages, 14 figure

On design of hybrid diffractive optics for achromatic extended depth-of-field (EDoF) RGB imaging

A hybrid imaging system is a simultaneous physical arrangement of a refractive lens and a multilevel phase mask (MPM) as a diffractive optical element (DOE). The favorable properties of the hybrid setup are improved extended-depth-of-field (EDoF) imaging and low chromatic aberrations. We built a fully differentiable image formation model in order to use neural network techniques to optimize imaging. At the first stage, the design framework relies on the model-based approach with numerical simulation and end-to-end joint optimization of both MPM and imaging algorithms. In the second stage, MPM is fixed as found at the first stage, and the image processing is optimized experimentally using the CNN learning-based approach with MPM implemented by a spatial light modulator. The paper is concentrated on a comparative analysis of imaging accuracy and quality for design with various basic optical parameters: aperture size, lens focal length, and distance between MPM and sensor. We point out that the varying aperture size, lens focal length, and distance between MPM and sensor are for the first time considered for end-to-end optimization of EDoF. We numerically and experimentally compare the designs for visible wavelength interval [400-700] nm and the following EDoF ranges: [0.5-100] m for simulations and [0.5-1.9] m for experimental tests. This study concerns an application of hybrid optics for compact cameras with aperture [5-9] mm and distance between MPM and sensor [3-10] mm.publishedVersionPeer reviewe

Compact Snapshot Hyperspectral Imaging with Diffracted Rotation

Traditional snapshot hyperspectral imaging systems include various optical elements: a dispersive optical element (prism), a coded aperture, several relay lenses, and an imaging lens, resulting in an impractically large form factor. We seek an alternative, minimal form factor of snapshot spectral imaging based on recent advances in diffractive optical technology. We thereupon present a compact, diffraction-based snapshot hyperspectral imaging method, using only a novel diffractive optical element (DOE) in front of a conventional, bare image sensor. Our diffractive imaging method replaces the common optical elements in hyperspectral imaging with a single optical element. To this end, we tackle two main challenges: First, the traditional diffractive lenses are not suitable for color imaging under incoherent illumination due to severe chromatic aberration because the size of the point spread function (PSF) changes depending on the wavelength. By leveraging this wavelength-dependent property alternatively for hyperspectral imaging, we introduce a novel DOE design that generates an anisotropic shape of the spectrally-varying PSF. The PSF size remains virtually unchanged, but instead the PSF shape rotates as the wavelength of light changes. Second, since there is no dispersive element and no coded aperture mask, the ill-posedness of spectral reconstruction increases significantly. Thus, we propose an end-to-end network solution based on the unrolled architecture of an optimization procedure with a spatial-spectral prior, specifically designed for deconvolution-based spectral reconstruction. Finally, we demonstrate hyperspectral imaging with a fabricated DOE attached to a conventional DSLR sensor. Results show that our method compares well with other state-of-the-art hyperspectral imaging methods in terms of spectral accuracy and spatial resolution, while our compact, diffraction-based spectral imaging method uses only a single optical element on a bare image sensor.11Nsciescopu