Fast Two-step Blind Optical Aberration Correction

The optics of any camera degrades the sharpness of photographs, which is a key visual quality criterion. This degradation is characterized by the point-spread function (PSF), which depends on the wavelengths of light and is variable across the imaging field. In this paper, we propose a two-step scheme to correct optical aberrations in a single raw or JPEG image, i.e., without any prior information on the camera or lens. First, we estimate local Gaussian blur kernels for overlapping patches and sharpen them with a non-blind deblurring technique. Based on the measurements of the PSFs of dozens of lenses, these blur kernels are modeled as RGB Gaussians defined by seven parameters. Second, we remove the remaining lateral chromatic aberrations (not contemplated in the first step) with a convolutional neural network, trained to minimize the red/green and blue/green residual images. Experiments on both synthetic and real images show that the combination of these two stages yields a fast state-of-the-art blind optical aberration compensation technique that competes with commercial non-blind algorithms.Comment: 28 pages, 20 figures, accepted at ECCV'22 as a poste

AAM: An Assessment Metric of Axial Chromatic Aberration

Knowledge of lens specifications is important to identify the best lens for a given capture scenario and application. Lens manufacturers provide many specifications in their data sheets, and multiple initiatives for testing and comparing different lenses can be found online. However, due to the lack of a suitable metric or technique, no evaluation of axial chromatic aberration is available. In this paper, we propose a metric, Axial Aberration Magnitude or AAM, that assesses the degree of axial chromatic aberration of a given lens. Our metric is generalizable to multispectral acquisition systems and is very simple and cheap to compute. We present the entire procedure and algorithm for computing the AAM metric, and evaluate it for two spectral systems and two consumer lenses

Development of freeform optical systems

One of the essential properties of the freeform surface is that its asymmetric and locally variant surface profile breaks the symmetry of an optical system, thus provoking unique issues that have never been considered in rotationally symmetric optical systems. This thesis focuses on developing an optimization algorithm that automatically eliminates the obscuration in the non-rotationally symmetric reflective optical system, as well as defining and computing the generalized chromatic aberrations in the non-rotationally symmetric refractive optical system. Furthermore, a comprehensive model for the tolerancing of freeform surface is put forward. When optimizing a three-dimensional (3D) reflective optical system by tilting the mirrors, the mirrors can block the ray path and, in consequence, reduce the image brightness and contrast. To take the degree of obscuration into consideration, an error function that mathematically describes all the obscuration cases in 3D reflective systems is proposed. In order to analyze the generalized chromatic aberrations in 3D refractive systems, the reference axis and reference plane are clarified to figure out the precise definition of the generalized chromatic aberrations. Both ray-based and wavefront-based methods are proposed to calculate the generalized chromatic aberrations surface-by-surface. In addition, the influence of pupil aberration is discussed to improve calculation accuracy. The manufacturing error of the freeform surface can be transferred into the frequency domain by Fourier transform. The autocorrelation function (ACF) of the phase pattern is computed in different frequency ranges. By characterizing the width of ACF, the boundary frequency between the deterministic and statistic errors can be found. A comprehensive model representing different types of surface errors is proposed to construct a synthetic freeform surface. By performing the Monte-Carlo simulation, tolerancing of the freeform system can be realized

Robust Joint Image Reconstruction from Color and Monochrome Cameras

International audienceRecent years have seen an explosion of the number of camera modules integratedinto individual consumer mobile devices, including configurations that contain multi-ple different types of image sensors. One popular configuration is to combine an RGBcamera for color imaging with a monochrome camera that has improved performancein low-light settings, as well as some sensitivity in the infrared. In this work we in-troduce a method to combine simultaneously captured images from such a two-camerastereo system to generate a high-quality, noise reduced color image. To do so, pixel-to-pixel alignment has to be constructed between the two captured monochrome and colorimages, which however, is prone to artifacts due to parallax. The joint image recon-struction is made robust by introducing a novel artifact-robust optimization formulation.We provide extensive experimental results based on the two-camera configuration of a commercially available cell phone

Treatise on Hearing: The Temporal Auditory Imaging Theory Inspired by Optics and Communication

A new theory of mammalian hearing is presented, which accounts for the auditory image in the midbrain (inferior colliculus) of objects in the acoustical environment of the listener. It is shown that the ear is a temporal imaging system that comprises three transformations of the envelope functions: cochlear group-delay dispersion, cochlear time lensing, and neural group-delay dispersion. These elements are analogous to the optical transformations in vision of diffraction between the object and the eye, spatial lensing by the lens, and second diffraction between the lens and the retina. Unlike the eye, it is established that the human auditory system is naturally defocused, so that coherent stimuli do not react to the defocus, whereas completely incoherent stimuli are impacted by it and may be blurred by design. It is argued that the auditory system can use this differential focusing to enhance or degrade the images of real-world acoustical objects that are partially coherent. The theory is founded on coherence and temporal imaging theories that were adopted from optics. In addition to the imaging transformations, the corresponding inverse-domain modulation transfer functions are derived and interpreted with consideration to the nonuniform neural sampling operation of the auditory nerve. These ideas are used to rigorously initiate the concepts of sharpness and blur in auditory imaging, auditory aberrations, and auditory depth of field. In parallel, ideas from communication theory are used to show that the organ of Corti functions as a multichannel phase-locked loop (PLL) that constitutes the point of entry for auditory phase locking and hence conserves the signal coherence. It provides an anchor for a dual coherent and noncoherent auditory detection in the auditory brain that culminates in auditory accommodation. Implications on hearing impairments are discussed as well.Comment: 603 pages, 131 figures, 13 tables, 1570 reference

Panoramic, large-screen, 3-D flight display system design

The report documents and summarizes the results of the required evaluations specified in the SOW and the design specifications for the selected display system hardware. Also included are the proposed development plan and schedule as well as the estimated rough order of magnitude (ROM) cost to design, fabricate, and demonstrate a flyable prototype research flight display system. The thrust of the effort was development of a complete understanding of the user/system requirements for a panoramic, collimated, 3-D flyable avionic display system and the translation of the requirements into an acceptable system design for fabrication and demonstration of a prototype display in the early 1997 time frame. Eleven display system design concepts were presented to NASA LaRC during the program, one of which was down-selected to a preferred display system concept. A set of preliminary display requirements was formulated. The state of the art in image source technology, 3-D methods, collimation methods, and interaction methods for a panoramic, 3-D flight display system were reviewed in depth and evaluated. Display technology improvements and risk reductions associated with maturity of the technologies for the preferred display system design concept were identified

Machine Learning Advances for Practical Problems in Computer Vision

Convolutional neural networks (CNN) have become the de facto standard for computer vision tasks, due to their unparalleled performance and versatility. Although deep learning removes the need for extensive hand engineered features for every task, real world applications of CNNs still often require considerable engineering effort to produce usable results. In this thesis, we explore solutions to problems that arise in practical applications of CNNs. We address a rarely acknowledged weakness of CNN object detectors: the tendency to emit many excess detection boxes per object, which must be pruned by non maximum suppression (NMS). This practice relies on the assumption that highly overlapping boxes are excess, which is problematic when objects are occluding overlapping detections are actually required. Therefore we propose a novel loss function that incentivises a CNN to emit exactly one detection per object, making NMS unnecessary. Another common problem when deploying a CNN in the real world is domain shift - CNNs can be surprisingly vulnerable to sometimes quite subtle differences between the images they encounter at deployment and those they are trained on. We investigate the role that texture plays in domain shift, and propose a novel data augmentation technique using style transfer to train CNNs that are more robust against shifts in texture. We demonstrate that this technique results in better domain transfer on several datasets, without requiring any domain specific knowledge. In collaboration with AstraZeneca, we develop an embedding space for cellular images collected in a high throughput imaging screen as part of a drug discovery project. This uses a combination of techniques to embed the images in 2D space such that similar images are nearby, for the purpose of visualization and data exploration. The images are also clustered automatically, splitting the large dataset into a smaller number of clusters that display a common phenotype. This allows biologists to quickly triage the high throughput screen, selecting a small subset of promising phenotypes for further investigation. Finally, we investigate an unusual form of domain bias that manifested in a real-world visual binary classification project for counterfeit detection. We confirm that CNNs are able to ``cheat'' the task by exploiting a strong correlation between class label and the specific camera that acquired the image, and show that this reliably occurs when the correlation is present. We also investigate the question of how exactly the CNN is able to infer camera type from image pixels, given that this is impossible to the human eye. The contributions in this thesis are of practical value to deep learning practitioners working on a variety of problems in the field of computer vision