Image enhancement methods and applications in computational photography

Computational photography is currently a rapidly developing and cutting-edge topic in applied optics, image sensors and image processing fields to go beyond the limitations of traditional photography. The innovations of computational photography allow the photographer not only merely to take an image, but also, more importantly, to perform computations on the captured image data. Good examples of these innovations include high dynamic range imaging, focus stacking, super-resolution, motion deblurring and so on. Although extensive work has been done to explore image enhancement techniques in each subfield of computational photography, attention has seldom been given to study of the image enhancement technique of simultaneously extending depth of field and dynamic range of a scene. In my dissertation, I present an algorithm which combines focus stacking and high dynamic range (HDR) imaging in order to produce an image with both extended depth of field (DOF) and dynamic range than any of the input images. In this dissertation, I also investigate super-resolution image restoration from multiple images, which are possibly degraded by large motion blur. The proposed algorithm combines the super-resolution problem and blind image deblurring problem in a unified framework. The blur kernel for each input image is separately estimated. I also do not make any restrictions on the motion fields among images; that is, I estimate dense motion field without simplifications such as parametric motion. While the proposed super-resolution method uses multiple images to enhance spatial resolution from multiple regular images, single image super-resolution is related to techniques of denoising or removing blur from one single captured image. In my dissertation, space-varying point spread function (PSF) estimation and image deblurring for single image is also investigated. Regarding the PSF estimation, I do not make any restrictions on the type of blur or how the blur varies spatially. Once the space-varying PSF is estimated, space-varying image deblurring is performed, which produces good results even for regions where it is not clear what the correct PSF is at first. I also bring image enhancement applications to both personal computer (PC) and Android platform as computational photography applications

Multi-plane super-resolution microscopy

Understanding cell functions is the major goal of molecular biology, which intends to elucidate the interactions between biomolecules at a subcellular level. One of the widely used techniques in molecular biology is fluorescence microscopy, which offers high specificity and sensitivity at the submicrometer spatial scale but is limited by diffraction to about 200nm lateral resolution, which is insufficient for the observation of many molecular processes. During the last two decades several super-resolution techniques overcoming the diffraction limit have been developed. However, imaging samples in three dimensions (3D) at high speed remains a challenging and not yet resolved task. This thesis focuses on enhancing super-resolution imaging towards fast, live-cell and 3D imaging. Super-resolution optical fluctuation imaging (SOFI) is a technique based on the stochastic fluctuations of photoswitchable fluorescent markers. It possesses several unique features such as background reduction, capability of increased pixel grid generation, i.e. spatial oversampling, as well as tolerance and robustness to a wide range of photoswitching conditions. In this thesis SOFI was extended to perform 3D analysis. As a result, the resolution in all three spatial dimensions can be improved and the depth sampling increased. We present a novel design of a 3D fluorescence microscope capable of acquiring images of eight depth planes simultaneously. This design incorporates an image-splitting prism, a single optical element allowing to achieve in-depth image separation. The optical performance of the 3D microscope was described and experimentally verified. The simultaneous depth plane acquisition allows to fully exploit the 3D capabilities of SOFI while generating additional virtual depth planes. An algorithm for the extraction of switching kinetics of fluorescent markers is presented. Using appropriate imaging conditions, we demonstrate the applications of 3D SOFI on several examples of fixed and living cells. We also present the potential of the 3D microscope for phase retrieval in transparent samples

Plasmonic photoconductive terahertz focal-plane array with pixel super-resolution

Imaging systems operating in the terahertz part of the electromagnetic spectrum are in great demand because of the distinct characteristics of terahertz waves in penetrating many optically-opaque materials and providing unique spectral signatures of various chemicals. However, the use of terahertz imagers in real-world applications has been limited by the slow speed, large size, high cost, and complexity of the existing imaging systems. These limitations are mainly imposed due to the lack of terahertz focal-plane arrays (THz-FPAs) that can directly provide the frequency-resolved and/or time-resolved spatial information of the imaged objects. Here, we report the first THz-FPA that can directly provide the spatial amplitude and phase distributions, along with the ultrafast temporal and spectral information of an imaged object. It consists of a two-dimensional array of ~0.3 million plasmonic photoconductive nanoantennas optimized to rapidly detect broadband terahertz radiation with a high signal-to-noise ratio. As the first proof-of-concept, we utilized the multispectral nature of the amplitude and phase data captured by these plasmonic nanoantennas to realize pixel super-resolution imaging of objects. We successfully imaged and super-resolved etched patterns in a silicon substrate and reconstructed both the shape and depth of these structures with an effective number of pixels that exceeds 1-kilo pixels. By eliminating the need for raster scanning and spatial terahertz modulation, our THz-FPA offers more than a 1000-fold increase in the imaging speed compared to the state-of-the-art. Beyond this proof-of-concept super-resolution demonstration, the unique capabilities enabled by our plasmonic photoconductive THz-FPA offer transformative advances in a broad range of applications that use hyperspectral and three-dimensional terahertz images of objects for a wide range of applications.Comment: 62 page

Acoustical structured illumination for super-resolution ultrasound imaging.

Structured illumination microscopy is an optical method to increase the spatial resolution of wide-field fluorescence imaging beyond the diffraction limit by applying a spatially structured illumination light. Here, we extend this concept to facilitate super-resolution ultrasound imaging by manipulating the transmitted sound field to encode the high spatial frequencies into the observed image through aliasing. Post processing is applied to precisely shift the spectral components to their proper positions in k-space and effectively double the spatial resolution of the reconstructed image compared to one-way focusing. The method has broad application, including the detection of small lesions for early cancer diagnosis, improving the detection of the borders of organs and tumors, and enhancing visualization of vascular features. The method can be implemented with conventional ultrasound systems, without the need for additional components. The resulting image enhancement is demonstrated with both test objects and ex vivo rat metacarpals and phalanges

The Devil is in the Decoder: Classification, Regression and GANs

Many machine vision applications, such as semantic segmentation and depth prediction, require predictions for every pixel of the input image. Models for such problems usually consist of encoders which decrease spatial resolution while learning a high-dimensional representation, followed by decoders who recover the original input resolution and result in low-dimensional predictions. While encoders have been studied rigorously, relatively few studies address the decoder side. This paper presents an extensive comparison of a variety of decoders for a variety of pixel-wise tasks ranging from classification, regression to synthesis. Our contributions are: (1) Decoders matter: we observe significant variance in results between different types of decoders on various problems. (2) We introduce new residual-like connections for decoders. (3) We introduce a novel decoder: bilinear additive upsampling. (4) We explore prediction artifacts

Light Field Super-Resolution Via Graph-Based Regularization

Light field cameras capture the 3D information in a scene with a single exposure. This special feature makes light field cameras very appealing for a variety of applications: from post-capture refocus, to depth estimation and image-based rendering. However, light field cameras suffer by design from strong limitations in their spatial resolution, which should therefore be augmented by computational methods. On the one hand, off-the-shelf single-frame and multi-frame super-resolution algorithms are not ideal for light field data, as they do not consider its particular structure. On the other hand, the few super-resolution algorithms explicitly tailored for light field data exhibit significant limitations, such as the need to estimate an explicit disparity map at each view. In this work we propose a new light field super-resolution algorithm meant to address these limitations. We adopt a multi-frame alike super-resolution approach, where the complementary information in the different light field views is used to augment the spatial resolution of the whole light field. We show that coupling the multi-frame approach with a graph regularizer, that enforces the light field structure via nonlocal self similarities, permits to avoid the costly and challenging disparity estimation step for all the views. Extensive experiments show that the new algorithm compares favorably to the other state-of-the-art methods for light field super-resolution, both in terms of PSNR and visual quality.Comment: This new version includes more material. In particular, we added: a new section on the computational complexity of the proposed algorithm, experimental comparisons with a CNN-based super-resolution algorithm, and new experiments on a third datase