An investigation into high dynamic range imaging technologies

This thesis studies high dynamic range imaging (HDR) technologies. It covers techniques for creating HDR radiance map from photographs, tone mapping HDR radiance map for display and the evaluation of the quality of tone-mapped images. The influential technique introduced by Debevec and Malik has become the de facto standard for recovering high dynamic range radiance maps from photographs and has been widely used in research and commercial systems for over a decade. However, we have discovered an important defect in the original algorithm that will make this technique often fail to produce reasonable results in the extremely bright or dark regions of a scene. Therefore, we introduce a novel technique to correct this defect. Instead of the original algorithm where only pixel values from the photographs are used to guide the synthesis of the high dynamic range radiance map, we explicitly incorporate the shutter speed information of the camera. At each spatial pixel location, we estimate a “suitable shutter” that will make that location best exposed. A pixel’s contribution to the high dynamic range radiance value is not only a function of its value but also depends on the difference between shutter speed used to take the pixel and the estimated “suitable shutter” of that pixel. We also show that this new idea can be successfully used to directly fuse differently exposed photographs into a single low dynamic range image for display in conventional low dynamic range devices. Then, we present a novel tone mapping framework. In this framework, firstly we introduce a tone mapping fidelity principle which explicitly stipulates that tone-mapped image data should not only be visually enhanced but should also stay faithful to the original image. Second, this principle naturally translates tone mapping into a constrained optimization problem where a two-term cost function, one measures the difference between the tone-mapped image and a visually enhanced version of the image, and the other measures the difference between the tone-mapped image and the original image, is optimized. The relative weightings of the two terms in the cost function not only offers an insightful and simple mechanism to control the appearance of the tone-mapped image but also enables the introduction of spatially varying or uniform weighting functions thus unifying local and global tone mapping in a single framework. The HDR image is not directly viewable and dynamic range compression will unavoidably loose information. A saliency map analyses the visual importance of the regions and can therefore direct the tone mapping operators to preserve the visual conspicuity of the regions that should more likely attract visual attention. Therefore, we present a novel tone mapping method - Saliency Modulated High Dynamic Range Image Tone Mapping (SMTM). In SMTM, we have developed a very fast algorithm to first compute the visual saliency map of the high dynamic range radiance map and then directly use the saliency of the local regions to control the local tone mapping curve such that highly salient regions will have their details and contrast better protected so as to remain salient and attract visual attention in the tone-mapped display. We present experimental results to show that SMTM provides competitive performances to state of the art tone mapping techniques in rending visually pleasing low dynamic range displays. We also show that SMTM is better able to preserve the visual saliency of the HDR image and that SMTM renders high saliency regions to stand out to attract observers’ attention. Finally, to solve a difficult problem of evaluating tone mapping algorithms, we introduce a novel approach - pair comparison using Web 2.0 Technology. In this evaluation approach, we have developed a Web2.0 style system that enables Internet users from anywhere to evaluate tone-mapped HDR photos at any time. We adopt a simple paired comparison protocol, Internet users are presented a pair of tone-mapped images and are simply asked to select the one that they think is better or click a “no difference” button. These user inputs are collected in the web server and analysed by a rank aggregation algorithm which ranks the tone-mapped photos according to the votes they received. The advantages of this approach include the potential of collecting large user inputs under a variety of viewing environments rather than limited user participation under controlled laboratory environments thus enabling more robust and reliable quality assessment. We also present data analysis to correlate user generated qualitative indices with quantitative image statistics which may provide useful guidance for developing better tone mapping operators

Fully-automatic inverse tone mapping algorithm based on dynamic mid-level tone mapping

High Dynamic Range (HDR) displays can show images with higher color contrast levels and peak luminosities than the common Low Dynamic Range (LDR) displays. However, most existing video content is recorded and/or graded in LDR format. To show LDR content on HDR displays, it needs to be up-scaled using a so-called inverse tone mapping algorithm. Several techniques for inverse tone mapping have been proposed in the last years, going from simple approaches based on global and local operators to more advanced algorithms such as neural networks. Some of the drawbacks of existing techniques for inverse tone mapping are the need for human intervention, the high computation time for more advanced algorithms, limited low peak brightness, and the lack of the preservation of the artistic intentions. In this paper, we propose a fully-automatic inverse tone mapping operator based on mid-level mapping capable of real-time video processing. Our proposed algorithm allows expanding LDR images into HDR images with peak brightness over 1000 nits, preserving the artistic intentions inherent to the HDR domain. We assessed our results using the full-reference objective quality metrics HDR-VDP-2.2 and DRIM, and carrying out a subjective pair-wise comparison experiment. We compared our results with those obtained with the most recent methods found in the literature. Experimental results demonstrate that our proposed method outperforms the current state-of-the-art of simple inverse tone mapping methods and its performance is similar to other more complex and time-consuming advanced techniques

Efficient and effective objective image quality assessment metrics

Acquisition, transmission, and storage of images and videos have been largely increased in recent years. At the same time, there has been an increasing demand for high quality images and videos to provide satisfactory quality-of-experience for viewers. In this respect, high dynamic range (HDR) imaging with higher than 8-bit depth has been an interesting approach in order to capture more realistic images and videos. Objective image and video quality assessment plays a significant role in monitoring and enhancing the image and video quality in several applications such as image acquisition, image compression, multimedia streaming, image restoration, image enhancement and displaying. The main contributions of this work are to propose efficient features and similarity maps that can be used to design perceptually consistent image quality assessment tools. In this thesis, perceptually consistent full-reference image quality assessment (FR-IQA) metrics are proposed to assess the quality of natural, synthetic, photo-retouched and tone-mapped images. In addition, efficient no-reference image quality metrics are proposed to assess JPEG compressed and contrast distorted images. Finally, we propose a perceptually consistent color to gray conversion method, perform a subjective rating and evaluate existing color to gray assessment metrics. Existing FR-IQA metrics may have the following limitations. First, their performance is not consistent for different distortions and datasets. Second, better performing metrics usually have high complexity. We propose in this thesis an efficient and reliable full-reference image quality evaluator based on new gradient and color similarities. We derive a general deviation pooling formulation and use it to compute a final quality score from the similarity maps. Extensive experimental results verify high accuracy and consistent performance of the proposed metric on natural, synthetic and photo retouched datasets as well as its low complexity. In order to visualize HDR images on standard low dynamic range (LDR) displays, tone-mapping operators are used in order to convert HDR into LDR. Given different depth bits of HDR and LDR, traditional FR-IQA metrics are not able to assess the quality of tone-mapped images. The existing full-reference metric for tone-mapped images called TMQI converts both HDR and LDR to an intermediate color space and measure their similarity in the spatial domain. We propose in this thesis a feature similarity full-reference metric in which local phase of HDR is compared with the local phase of LDR. Phase is an important information of images and previous studies have shown that human visual system responds strongly to points in an image where the phase information is ordered. Experimental results on two available datasets show the very promising performance of the proposed metric. No-reference image quality assessment (NR-IQA) metrics are of high interest because in the most present and emerging practical real-world applications, the reference signals are not available. In this thesis, we propose two perceptually consistent distortion-specific NR-IQA metrics for JPEG compressed and contrast distorted images. Based on edge statistics of JPEG compressed images, an efficient NR-IQA metric for blockiness artifact is proposed which is robust to block size and misalignment. Then, we consider the quality assessment of contrast distorted images which is a common distortion. Higher orders of Minkowski distance and power transformation are used to train a low complexity model that is able to assess contrast distortion with high accuracy. For the first time, the proposed model is used to classify the type of contrast distortions which is very useful additional information for image contrast enhancement. Unlike its traditional use in the assessment of distortions, objective IQA can be used in other applications. Examples are the quality assessment of image fusion, color to gray image conversion, inpainting, background subtraction, etc. In the last part of this thesis, a real-time and perceptually consistent color to gray image conversion methodology is proposed. The proposed correlation-based method and state-of-the-art methods are compared by subjective and objective evaluation. Then, a conclusion is made on the choice of the objective quality assessment metric for the color to gray image conversion. The conducted subjective ratings can be used in the development process of quality assessment metrics for the color to gray image conversion and to test their performance

Contemplation of tone mapping operators in high dynamic range imaging

The technique of tone mapping has found widespread popularity in the modern era owing to its applications in the digital world. There are a considerable number of tone mapping techniques that have been developed so far. One method may be better than the other in some cases which is determined by the requirement of the user. In this paper, some of the techniques for tone mapping/tone reproduction of high dynamic range images have been contemplated. The classification of tone mapping operators has also been given. However, it has been found that these techniques lack in providing quality of service visualization of high dynamic range images. This paper has tried to highlight the drawbacks in the existing traditional methods so that the tone-mapped techniques can be enhanced