Efficient HDR Reconstruction from Real-World Raw Images

High dynamic range (HDR) imaging is still a significant yet challenging problem due to the limited dynamic range of generic image sensors. Most existing learning-based HDR reconstruction methods take a set of bracketed-exposure sRGB images to extend the dynamic range, and thus are computational- and memory-inefficient by requiring the Image Signal Processor (ISP) to produce multiple sRGB images from the raw ones. In this paper, we propose to broaden the dynamic range from the raw inputs and perform only one ISP processing for the reconstructed HDR raw image. Our key insights are threefold: (1) we design a new computational raw HDR data formation pipeline and construct the first real-world raw HDR dataset, RealRaw-HDR; (2) we develop a lightweight-efficient HDR model, RepUNet, using the structural re-parameterization technique; (3) we propose a plug-and-play motion alignment loss to mitigate motion misalignment between short- and long-exposure images. Extensive experiments demonstrate that our approach achieves state-of-the-art performance in both visual quality and quantitative metrics

HDR Video Reconstruction with a Large Dynamic Dataset in Raw and sRGB Domains

High dynamic range (HDR) video reconstruction is attracting more and more attention due to the superior visual quality compared with those of low dynamic range (LDR) videos. The availability of LDR-HDR training pairs is essential for the HDR reconstruction quality. However, there are still no real LDR-HDR pairs for dynamic scenes due to the difficulty in capturing LDR-HDR frames simultaneously. In this work, we propose to utilize a staggered sensor to capture two alternate exposure images simultaneously, which are then fused into an HDR frame in both raw and sRGB domains. In this way, we build a large scale LDR-HDR video dataset with 85 scenes and each scene contains 60 frames. Based on this dataset, we further propose a Raw-HDRNet, which utilizes the raw LDR frames as inputs. We propose a pyramid flow-guided deformation convolution to align neighboring frames. Experimental results demonstrate that 1) the proposed dataset can improve the HDR reconstruction performance on real scenes for three benchmark networks; 2) Compared with sRGB inputs, utilizing raw inputs can further improve the reconstruction quality and our proposed Raw-HDRNet is a strong baseline for raw HDR reconstruction. Our dataset and code will be released after the acceptance of this paper

Text2Light: Zero-Shot Text-Driven HDR Panorama Generation

High-quality HDRIs(High Dynamic Range Images), typically HDR panoramas, are one of the most popular ways to create photorealistic lighting and 360-degree reflections of 3D scenes in graphics. Given the difficulty of capturing HDRIs, a versatile and controllable generative model is highly desired, where layman users can intuitively control the generation process. However, existing state-of-the-art methods still struggle to synthesize high-quality panoramas for complex scenes. In this work, we propose a zero-shot text-driven framework, Text2Light, to generate 4K+ resolution HDRIs without paired training data. Given a free-form text as the description of the scene, we synthesize the corresponding HDRI with two dedicated steps: 1) text-driven panorama generation in low dynamic range(LDR) and low resolution, and 2) super-resolution inverse tone mapping to scale up the LDR panorama both in resolution and dynamic range. Specifically, to achieve zero-shot text-driven panorama generation, we first build dual codebooks as the discrete representation for diverse environmental textures. Then, driven by the pre-trained CLIP model, a text-conditioned global sampler learns to sample holistic semantics from the global codebook according to the input text. Furthermore, a structure-aware local sampler learns to synthesize LDR panoramas patch-by-patch, guided by holistic semantics. To achieve super-resolution inverse tone mapping, we derive a continuous representation of 360-degree imaging from the LDR panorama as a set of structured latent codes anchored to the sphere. This continuous representation enables a versatile module to upscale the resolution and dynamic range simultaneously. Extensive experiments demonstrate the superior capability of Text2Light in generating high-quality HDR panoramas. In addition, we show the feasibility of our work in realistic rendering and immersive VR.Comment: SIGGRAPH Asia 2022; Project Page https://frozenburning.github.io/projects/text2light/ Codes are available at https://github.com/FrozenBurning/Text2Ligh

デバイスの限界を超えた正確な撮像を可能にする深層学習

Tohoku University博士（情報科学）thesi

COMPRESSIVE IMAGING AND DUAL MOIRE´ LASER INTERFEROMETER AS METROLOGY TOOLS

Metrology is the science of measurement and deals with measuring different physical aspects of objects. In this research the focus has been on two basic problems that metrologists encounter. The first problem is the trade-off between the range of measurement and the corresponding resolution; measurement of physical parameters of a large object or scene accompanies by losing detailed information about small regions of the object. Indeed, instruments and techniques that perform coarse measurements are different from those that make fine measurements. This problem persists in the field of surface metrology, which deals with accurate measurement and detailed analysis of surfaces. For example, laser interferometry is used for fine measurement (in nanometer scale) while to measure the form of in object, which lies in the field of coarse measurement, a different technique like moire technique is used. We introduced a new technique to combine measurement from instruments with better resolution and smaller measurement range with those with coarser resolution and larger measurement range. We first measure the form of the object with coarse measurement techniques and then make some fine measurement for features in regions of interest. The second problem is the measurement conditions that lead to difficulties in measurement. These conditions include low light condition, large range of intensity variation, hyperspectral measurement, etc. Under low light condition there is not enough light for detector to detect light from object, which results in poor measurements. Large range of intensity variation results in a measurement with some saturated regions on the camera as well as some dark regions. We use compressive sampling based imaging systems to address these problems. Single pixel compressive imaging uses a single detector instead of array of detectors and reconstructs a complete image after several measurements. In this research we examined compressive imaging for different applications including low light imaging, high dynamic range imaging and hyperspectral imaging

Learning a Practical SDR-to-HDRTV Up-conversion using New Dataset and Degradation Models

In media industry, the demand of SDR-to-HDRTV up-conversion arises when users possess HDR-WCG (high dynamic range-wide color gamut) TVs while most off-the-shelf footage is still in SDR (standard dynamic range). The research community has started tackling this low-level vision task by learning-based approaches. When applied to real SDR, yet, current methods tend to produce dim and desaturated result, making nearly no improvement on viewing experience. Different from other network-oriented methods, we attribute such deficiency to training set (HDR-SDR pair). Consequently, we propose new HDRTV dataset (dubbed HDRTV4K) and new HDR-to-SDR degradation models. Then, it's used to train a luminance-segmented network (LSN) consisting of a global mapping trunk, and two Transformer branches on bright and dark luminance range. We also update assessment criteria by tailored metrics and subjective experiment. Finally, ablation studies are conducted to prove the effectiveness. Our work is available at: https://github.com/AndreGuo/HDRTVDM.Comment: Accepted by CVPR202

Performances of a portable Fourier transform hyperspectral imaging camera for rapid investigation of paintings

Abstract: Scientific investigation in the cultural heritage field is generally aimed at the characterization of the constituent materials and the conservation status of artworks. Since the 1990s, reflectance spectral imaging proved able to map pigments, reveal hidden details and evaluate the presence of restorations in paintings. Over the past two decades, hyperspectral imaging has further improved our understanding of paints and of its changes in time. In this work, we present an innovative hyperspectral camera, based on the Fourier transform approach, utilising an ultra-stable interferometer and we describe its advantages and drawbacks with respect to the commonly used line- and spectral-scanning methods. To mitigate the weaknesses of the Fourier transform hyperspectral imaging, we propose a strategy based on the virtual extension of the dynamic range of the camera and on the design of an illumination system with a balanced emission throughout the spectral range of interest. The hyperspectral camera was employed for the analysis of a painting from the “Album of Nasir al-din Shah”. By applying analysis routines based on supervised spectral unmixing, we demonstrate the effectiveness of our camera for pigment mapping. This work shows how the proposed hyperspectral imaging camera based on the Fourier transform is a promising technique for robust and compact in situ investigation of artistic objects in conditions compatible with museum and archaeological sites. Graphic abstract: [Figure not available: see fulltext.

Deep Reflectance Maps

Undoing the image formation process and therefore decomposing appearance into its intrinsic properties is a challenging task due to the under-constraint nature of this inverse problem. While significant progress has been made on inferring shape, materials and illumination from images only, progress in an unconstrained setting is still limited. We propose a convolutional neural architecture to estimate reflectance maps of specular materials in natural lighting conditions. We achieve this in an end-to-end learning formulation that directly predicts a reflectance map from the image itself. We show how to improve estimates by facilitating additional supervision in an indirect scheme that first predicts surface orientation and afterwards predicts the reflectance map by a learning-based sparse data interpolation. In order to analyze performance on this difficult task, we propose a new challenge of Specular MAterials on SHapes with complex IllumiNation (SMASHINg) using both synthetic and real images. Furthermore, we show the application of our method to a range of image-based editing tasks on real images.Comment: project page: http://homes.esat.kuleuven.be/~krematas/DRM