Super resolution and dynamic range enhancement of image sequences

Camera producers try to increase the spatial resolution of a camera by reducing size of sites on sensor array. However, shot noise causes the signal to noise ratio drop as sensor sites get smaller. This fact motivates resolution enhancement to be performed through software. Super resolution (SR) image reconstruction aims to combine degraded images of a scene in order to form an image which has higher resolution than all observations. There is a demand for high resolution images in biomedical imaging, surveillance, aerial/satellite imaging and high-definition TV (HDTV) technology. Although extensive research has been conducted in SR, attention has not been given to increase the resolution of images under illumination changes. In this study, a unique framework is proposed to increase the spatial resolution and dynamic range of a video sequence using Bayesian and Projection onto Convex Sets (POCS) methods. Incorporating camera response function estimation into image reconstruction allows dynamic range enhancement along with spatial resolution improvement. Photometrically varying input images complicate process of projecting observations onto common grid by violating brightness constancy. A contrast invariant feature transform is proposed in this thesis to register input images with high illumination variation. Proposed algorithm increases the repeatability rate of detected features among frames of a video. Repeatability rate is increased by computing the autocorrelation matrix using the gradients of contrast stretched input images. Presented contrast invariant feature detection improves repeatability rate of Harris corner detector around %25 on average. Joint multi-frame demosaicking and resolution enhancement is also investigated in this thesis. Color constancy constraint set is devised and incorporated into POCS framework for increasing resolution of color-filter array sampled images. Proposed method provides fewer demosaicking artifacts compared to existing POCS method and a higher visual quality in final image

Adaptive Wiener Filter Super-Resolution of Color Filter Array Images

Digital color cameras using a single detector array with a Bayer color filter array (CFA) require interpolation or demosaicing to estimate missing color information and provide full-color images. However, demosaicing does not specifically address fundamental undersampling and aliasing inherent in typical camera designs. Fast non-uniform interpolation based super-resolution (SR) is an attractive approach to reduce or eliminate aliasing and its relatively low computational load is amenable to real-time applications. The adaptive Wiener filter (AWF) SR algorithm was initially developed for grayscale imaging and has not previously been applied to color SR demosaicing. Here, we develop a novel fast SR method for CFA cameras that is based on the AWF SR algorithm and uses global channel-to-channel statistical models. We apply this new method as a stand-alone algorithm and also as an initialization image for a variational SR algorithm. This paper presents the theoretical development of the color AWF SR approach and applies it in performance comparisons to other SR techniques for both simulated and real data

Recent Advances in Image Restoration with Applications to Real World Problems

In the past few decades, imaging hardware has improved tremendously in terms of resolution, making widespread usage of images in many diverse applications on Earth and planetary missions. However, practical issues associated with image acquisition are still affecting image quality. Some of these issues such as blurring, measurement noise, mosaicing artifacts, low spatial or spectral resolution, etc. can seriously affect the accuracy of the aforementioned applications. This book intends to provide the reader with a glimpse of the latest developments and recent advances in image restoration, which includes image super-resolution, image fusion to enhance spatial, spectral resolution, and temporal resolutions, and the generation of synthetic images using deep learning techniques. Some practical applications are also included

A Gaussian Process Model for Color Camera Characterization: Assessment in Outdoor Levantine Rock Art Scenes

[EN] In this paper, we propose a novel approach to undertake the colorimetric camera characterization procedure based on a Gaussian process (GP). GPs are powerful and flexible nonparametric models for multivariate nonlinear functions. To validate the GP model, we compare the results achieved with a second-order polynomial model, which is the most widely used regression model for characterization purposes. We applied the methodology on a set of raw images of rock art scenes collected with two different Single Lens Reflex (SLR) cameras. A leave-one-out cross-validation (LOOCV) procedure was used to assess the predictive performance of the models in terms of CIE XYZ residuals and Delta E-ab* color differences. Values of less than 3 CIELAB units were achieved for Delta E-ab*. The output sRGB characterized images show that both regression models are suitable for practical applications in cultural heritage documentation. However, the results show that colorimetric characterization based on the Gaussian process provides significantly better results, with lower values for residuals and Delta E-ab*. We also analyzed the induced noise into the output image after applying the camera characterization. As the noise depends on the specific camera, proper camera selection is essential for the photogrammetric work.This research is partly funded by the Research and Development Aid Program PAID-01-16 of the Universitat Politecnica de Valencia, through FPI-UPV-2016 Sub 1 grant.Molada-Tebar, A.; Riutort-Mayol, G.; Marqués-Mateu, Á.; Lerma, JL. (2019). A Gaussian Process Model for Color Camera Characterization: Assessment in Outdoor Levantine Rock Art Scenes. Sensors. 19(21):1-22. https://doi.org/10.3390/s19214610S1221921Ruiz, J. F., & Pereira, J. (2014). The colours of rock art. Analysis of colour recording and communication systems in rock art research. Journal of Archaeological Science, 50, 338-349. doi:10.1016/j.jas.2014.06.023Gaiani, M., Apollonio, F., Ballabeni, A., & Remondino, F. (2017). Securing Color Fidelity in 3D Architectural Heritage Scenarios. Sensors, 17(11), 2437. doi:10.3390/s17112437Robert, E., Petrognani, S., & Lesvignes, E. (2016). Applications of digital photography in the study of Paleolithic cave art. Journal of Archaeological Science: Reports, 10, 847-858. doi:10.1016/j.jasrep.2016.07.026Fernández-Lozano, J., Gutiérrez-Alonso, G., Ruiz-Tejada, M. Á., & Criado-Valdés, M. (2017). 3D digital documentation and image enhancement integration into schematic rock art analysis and preservation: The Castrocontrigo Neolithic rock art (NW Spain). Journal of Cultural Heritage, 26, 160-166. doi:10.1016/j.culher.2017.01.008López-Menchero Bendicho, V. M., Marchante Ortega, Á., Vincent, M., Cárdenas Martín-Buitrago, Á. J., & Onrubia Pintado, J. (2017). Uso combinado de la fotografía digital nocturna y de la fotogrametría en los procesos de documentación de petroglifos: el caso de Alcázar de San Juan (Ciudad Real, España). Virtual Archaeology Review, 8(17), 64. doi:10.4995/var.2017.6820Hong, G., Luo, M. R., & Rhodes, P. A. (2000). A study of digital camera colorimetric characterization based on polynomial modeling. Color Research & Application, 26(1), 76-84. doi:10.1002/1520-6378(200102)26:13.0.co;2-3Hung, P.-C. (1993). Colorimetric calibration in electronic imaging devices using a look-up-table model and interpolations. Journal of Electronic Imaging, 2(1), 53. doi:10.1117/12.132391Vrhel, M. J., & Trussell, H. J. (1992). Color correction using principal components. Color Research & Application, 17(5), 328-338. doi:10.1002/col.5080170507Bianco, S., Gasparini, F., Russo, A., & Schettini, R. (2007). A New Method for RGB to XYZ Transformation Based on Pattern Search Optimization. IEEE Transactions on Consumer Electronics, 53(3), 1020-1028. doi:10.1109/tce.2007.4341581Finlayson, G. D., Mackiewicz, M., & Hurlbert, A. (2015). Color Correction Using Root-Polynomial Regression. IEEE Transactions on Image Processing, 24(5), 1460-1470. doi:10.1109/tip.2015.2405336Connah, D., Westland, S., & Thomson, M. G. A. (2001). Recovering spectral information using digital camera systems. Coloration Technology, 117(6), 309-312. doi:10.1111/j.1478-4408.2001.tb00080.xLiang, J., & Wan, X. (2017). Optimized method for spectral reflectance reconstruction from camera responses. Optics Express, 25(23), 28273. doi:10.1364/oe.25.028273Heikkinen, V. (2018). Spectral Reflectance Estimation Using Gaussian Processes and Combination Kernels. IEEE Transactions on Image Processing, 27(7), 3358-3373. doi:10.1109/tip.2018.2820839Molada-Tebar, A., Lerma, J. L., & Marqués-Mateu, Á. (2017). Camera characterization for improving color archaeological documentation. Color Research & Application, 43(1), 47-57. doi:10.1002/col.22152Durmus, A., Moulines, É., & Pereyra, M. (2018). Efficient Bayesian Computation by Proximal Markov Chain Monte Carlo: When Langevin Meets Moreau. SIAM Journal on Imaging Sciences, 11(1), 473-506. doi:10.1137/16m1108340Ruppert, D., Wand, M. P., & Carroll, R. J. (2009). Semiparametric regression during 2003–2007. Electronic Journal of Statistics, 3(0), 1193-1256. doi:10.1214/09-ejs525Rock Art of the Mediterranean Basin on the Iberian Peninsulahttp://whc.unesco.org/en/list/874Direct Image Sensor Sigma SD15http://www.sigma-sd.com/SD15/technology-colorsensor.htmlRamanath, R., Snyder, W. E., Yoo, Y., & Drew, M. S. (2005). Color image processing pipeline. IEEE Signal Processing Magazine, 22(1), 34-43. doi:10.1109/msp.2005.1407713Stone, M. (1974). Cross-Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statistical Society: Series B (Methodological), 36(2), 111-133. doi:10.1111/j.2517-6161.1974.tb00994.xVazquez-Corral, J., Connah, D., & Bertalmío, M. (2014). Perceptual Color Characterization of Cameras. Sensors, 14(12), 23205-23229. doi:10.3390/s141223205Sharma, G., Wu, W., & Dalal, E. N. (2004). The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations. Color Research & Application, 30(1), 21-30. doi:10.1002/col.20070Lebrun, M., Buades, A., & Morel, J. M. (2013). A Nonlocal Bayesian Image Denoising Algorithm. SIAM Journal on Imaging Sciences, 6(3), 1665-1688. doi:10.1137/120874989Colom, M., Buades, A., & Morel, J.-M. (2014). Nonparametric noise estimation method for raw images. Journal of the Optical Society of America A, 31(4), 863. doi:10.1364/josaa.31.000863Sur, F., & Grédiac, M. (2015). Measuring the Noise of Digital Imaging Sensors by Stacking Raw Images Affected by Vibrations and Illumination Flickering. SIAM Journal on Imaging Sciences, 8(1), 611-643. doi:10.1137/140977035Zhang, Y., Wang, G., & Xu, J. (2018). Parameter Estimation of Signal-Dependent Random Noise in CMOS/CCD Image Sensor Based on Numerical Characteristic of Mixed Poisson Noise Samples. Sensors, 18(7), 2276. doi:10.3390/s18072276Naveed, K., Ehsan, S., McDonald-Maier, K. D., & Ur Rehman, N. (2019). A Multiscale Denoising Framework Using Detection Theory with Application to Images from CMOS/CCD Sensors. Sensors, 19(1), 206. doi:10.3390/s19010206Riutort-Mayol, G., Marqués-Mateu, Á., Seguí, A. E., & Lerma, J. L. (2012). Grey Level and Noise Evaluation of a Foveon X3 Image Sensor: A Statistical and Experimental Approach. Sensors, 12(8), 10339-10368. doi:10.3390/s120810339Marqués-Mateu, Á., Lerma, J. L., & Riutort-Mayol, G. (2013). Statistical grey level and noise evaluation of Foveon X3 and CFA image sensors. Optics & Laser Technology, 48, 1-15. doi:10.1016/j.optlastec.2012.09.034Chou, Y.-F., Luo, M. R., Li, C., Cheung, V., & Lee, S.-L. (2013). Methods for designing characterisation targets for digital cameras. Coloration Technology, 129(3), 203-213. doi:10.1111/cote.12022Shen, H.-L., Cai, P.-Q., Shao, S.-J., & Xin, J. H. (2007). Reflectance reconstruction for multispectral imaging by adaptive Wiener estimation. Optics Express, 15(23), 15545. doi:10.1364/oe.15.015545Molada-Tebar, A., Marqués-Mateu, Á., & Lerma, J. (2019). Camera Characterisation Based on Skin-Tone Colours for Rock Art Recording. Proceedings, 19(1), 12. doi:10.3390/proceedings201901901

Bio-Inspired Multi-Spectral Image Sensor and Augmented Reality Display for Near-Infrared Fluorescence Image-Guided Surgery

Background: Cancer remains a major public health problem worldwide and poses a huge economic burden. Near-infrared (NIR) fluorescence image-guided surgery (IGS) utilizes molecular markers and imaging instruments to identify and locate tumors during surgical resection. Unfortunately, current state-of-the-art NIR fluorescence imaging systems are bulky, costly, and lack both fluorescence sensitivity under surgical illumination and co-registration accuracy between multimodal images. Additionally, the monitor-based display units are disruptive to the surgical workflow and are suboptimal at indicating the 3-dimensional position of labeled tumors. These major obstacles have prevented the wide acceptance of NIR fluorescence imaging as the standard of care for cancer surgery. The goal of this dissertation is to enhance cancer treatment by developing novel image sensors and presenting the information using holographic augmented reality (AR) display to the physician in intraoperative settings. Method: By mimicking the visual system of the Morpho butterfly, several single-chip, color-NIR fluorescence image sensors and systems were developed with CMOS technologies and pixelated interference filters. Using a holographic AR goggle platform, an NIR fluorescence IGS display system was developed. Optoelectronic evaluation was performed on the prototypes to evaluate the performance of each component, and small animal models and large animal models were used to verify the overall effectiveness of the integrated systems at cancer detection. Result: The single-chip bio-inspired multispectral logarithmic image sensor I developed has better main performance indicators than the state-of-the-art NIR fluorescence imaging instruments. The image sensors achieve up to 140 dB dynamic range. The sensitivity under surgical illumination achieves 6108 V/(mW/cm2), which is up to 25 times higher. The signal-to-noise ratio is up to 56 dB, which is 11 dB greater. These enable high sensitivity fluorescence imaging under surgical illumination. The pixelated interference filters enable temperature-independent co-registration accuracy between multimodal images. Pre-clinical trials with small animal model demonstrate that the sensor can achieve up to 95% sensitivity and 94% specificity with tumor-targeted NIR molecular probes. The holographic AR goggle provides the physician with a non-disruptive 3-dimensional display in the clinical setup. This is the first display system that co-registers a virtual image with human eyes and allows video rate image transmission. The imaging system is tested in the veterinary science operating room on canine patients with naturally occurring cancers. In addition, a time domain pulse-width-modulation address-event-representation multispectral image sensor and a handheld multispectral camera prototype are developed. Conclusion: The major problems of current state-of-the-art NIR fluorescence imaging systems are successfully solved. Due to enhanced performance and user experience, the bio-inspired sensors and augmented reality display system will give medical care providers much needed technology to enable more accurate value-based healthcare