This publication describes systems and techniques directed to enhancing High Dynamic Range (HDR) imaging by identifying and understanding the lighting environment of a scene for image capture.  Natural light (e.g., outdoor sky, sunlight) is identified and differentiated from artificial light (e.g., light-emitting diode (LED), fluorescent, halogen, incandescent lighting) for advanced metering and optimal exposure control.  Exposure is adjusted relative to the differentiated lighting for final image capture.  Regions of the scene are differentiated by mapping different weights to the dynamic range detected.  This comprehensive understanding of the scene captures the most important region of interest (e.g., from the viewer’s perspective) within a good exposure value

Gao, Jinglun

Hung, Szepo Robert

Sun, Gang

Velarde, Ruben

English

Technical Disclosure Common

Technical Disclosure Commons Defensive Publications Series April 2020 Scene Metering and Exposure Control for Enhancing High Dynamic Range Imaging Jinglun Gao Szepo Robert Hung Ruben Velarde Gang Sun Follow this and additional works at: https://www.tdcommons.org/dpubs_series Recommended Citation Gao, Jinglun; Hung, Szepo Robert; Velarde, Ruben; and Sun, Gang, "Scene Metering and Exposure Control for Enhancing High Dynamic Range Imaging", Technical Disclosure Commons, (April 01, 2020) https://www.tdcommons.org/dpubs_series/3092 This work is licensed under a Creative Commons Attribution 4.0 License. This Article is brought to you for free and open access by Technical Disclosure Commons. It has been accepted for inclusion in Defensive Publications Series by an authorized administrator of Technical Disclosure Commons. Scene Metering and Exposure Control for Enhancing High Dynamic Range Imaging Abstract: This publication describes systems and techniques directed to enhancing High Dynamic Range (HDR) imaging by identifying and understanding the lighting environment of a scene for image capture.  Natural light (e.g., outdoor sky, sunlight) is identified and differentiated from artificial light (e.g., light-emitting diode (LED), fluorescent, halogen, incandescent lighting) for advanced metering and optimal exposure control.  Exposure is adjusted relative to the differentiated lighting for final image capture.  Regions of the scene are differentiated by mapping different weights to the dynamic range detected.  This comprehensive understanding of the scene captures the most important region of interest (e.g., from the viewer’s perspective) within a good exposure value.  Keywords: Digital imaging, photography, high dynamic range imaging, HDR, HDR pipeline, exposure, exposure control, luminance, illuminance, lighting, frequency, artificial light, natural light, sky, outdoor, indoor, smartphone, camera, highlight, metering Background: In digital imaging and photography, dynamic range refers to the luminance range (e.g., the ratio of light to dark) of a scene, or the luminance range that a camera can capture.  High Dynamic Range (HDR) imaging is a method used to increase the dynamic range in captured images (e.g., digital images and photographs).  HDR images present a greater range of luminance levels for improved visual perception of the scene.  In aspects, HDR imaging may be performed by 2Gao et al.: Scene Metering and Exposure Control for Enhancing High Dynamic RaPublished by Technical Disclosure Commons, 2020combining the best parts of multiple separate images of the same scene, with the images being taken at different exposure levels. It is generally desirable to capture more dynamic range in a scene.  For example, when an outdoor (natural light) scene is captured, HDR imaging may adjust the exposure such that a sky area and a ground area are both well-exposed.  Without HDR imaging, the sky area may end up overexposed (too bright and washed out) as the camera increases exposure to capture an accurate luminance and color image of the ground area.  Similarly, when an indoor (artificial light) scene is captured, HDR imaging may adjust the exposure such that the indoor room area and the artificial light sources are all well-exposed.  Without HDR imaging, the artificial light sources may end up overexposed as the camera increases exposure to capture an accurate luminance and color image of the other areas of the rooms.   Although the use of HDR imaging is becoming more popular, it may be difficult to achieve without extensive and time-consuming image processing requirements.  HDR imaging may also underexpose important regions of interest in a scene, causing those regions to be darker than desired.  Additionally, even using HDR imaging, an overexposed light source in an image may end up being considered acceptable by a person if, in fact, the region of interest is well-exposed, simply because of the difficulty or lack of technology for easily enabling a better overall exposed image. Description: This publication describes systems and techniques directed to enhancing High Dynamic Range (HDR) imaging by identifying and understanding the lighting environment of a scene for image capture.  Natural light (e.g., outdoor sky, sunlight) is identified and differentiated from artificial light (e.g., LED, fluorescent, halogen, incandescent lighting) for advanced metering and 3Defensive Publications Series, Art. 3092 [2020]https://www.tdcommons.org/dpubs_series/3092optimal exposure control.  Exposure is adjusted relative to the differentiated lighting for final image capture.  Regions of the scene are differentiated by mapping different weights to the dynamic range detected.  This comprehensive understanding of the scene captures the most important region of interest (e.g., from the viewer’s perspective) within a good exposure value.  Detecting and differentiating light sources is critical for understanding scene content and for providing enhanced imaging.  Accordingly, this disclosure uses an image-based light source region detection method in combination with a spectral sensor-based light source region detection method.  Together these help identify and differentiate light sources, highlight regions (e.g., areas of increased luminance), and regions of interest within a scene.  For example, this method detects and identifies whether a highlight region is from a natural light source or an artificial light source.  Based on the light source identification, a statistical map is generated (mapping different weights to the dynamic range detected) to enable improved exposure control.  This method and technology is adaptable for use in digital cameras, smartphones, and other digital imaging devices, and may be easily integrated into the exposure control algorithm in the HDR imaging pipeline.   Because highlight regions of a scene may be close to saturation, not as much information is preserved in the captured image for visual perception.  Accordingly, underexposing highlight regions is imperative during image capture so that the highlight region can be well-exposed.  However, a well-exposed highlight region typically causes other non-highlighted regions to be dark (underexposed).  For example, well-exposing a highlighted outdoor sky region typically causes the ground region to be underexposed and dark.  This makes it difficult to retrieve relevant image data from the underexposed region. Accordingly, for image-based light source region detection (e.g., to detect natural sky light), a number of exposure algorithms may be considered.  For example, one method incorporates 4Gao et al.: Scene Metering and Exposure Control for Enhancing High Dynamic RaPublished by Technical Disclosure Commons, 2020scene descriptors into an auto-exposure database.  Other methods may include using a convolutional neural network (CNN) on an image pixel-level perception, attempting to specifically detect small light source regions, or addressing frame artifacts (e.g., noise, banding effect).  On the other hand, a spectral sensor-based light source region detection method may be used to detect artificial light flicker, environmental luminance (lux range), and color identification.  A reliable and efficient algorithm for leveraging the spectral sensor considers critical lighting condition information, including the spectral power distribution and light source frequency. Different light sources have different spectral power distributions that lead to different color effects for imaging sensors.  Figure 1 illustrates different light spectrum power distributions for six example light sources:  circadian, daylight, correlated color temperature (CCT) for LEDs, incandescent, fluorescent, and metal halide.  Ultraviolet light (UV) is indicated on the left side of each spectrum, RGB is in the middle, and Infrared (IR) is indicated on the right.  Figure 1 5Defensive Publications Series, Art. 3092 [2020]https://www.tdcommons.org/dpubs_series/3092Because artificial light is powered by electricity, it generates a fixed frequency of magnitude.  Figure 2 illustrates an example of light frequency (light source energy wave) resulting from artificial light sources (based on the power supply wave).  Figure 2 The spectral sensor is calibrated, using this spectral power distribution and light frequency information, to identify if the highlight region of a scene is natural light or artificial light.  A spectral sensor can provide two types of data.  One type of data is the ambient light data (natural or artificial), which uses the color pixels (R/G/B) and IR, UV, or clear channel (no color coding), to represent the color spectrum.  The other type of data is artificial light-flicker effect data.  This light-flicker effect data is obtained by the spectral sensor exposing in a high frequency (e.g., up to 8K Hz), which reflects the flickering effect of an artificial light source.  With these two types of data, the following cases can be differentiated: 1. Artificial light sources: these present flickering data, aligned with a power cycle, or triggered internally, so can be detected using single-channel flicker data. 6Gao et al.: Scene Metering and Exposure Control for Enhancing High Dynamic RaPublished by Technical Disclosure Commons, 20202. Different light scenes:  these present different color spectrums, and ambient light sensor (ALS) data can be used to differentiate them, or a combination of multi-channel data can be used. 3. Different CCT: sunny outdoor conditions, or shadow-casting lighting conditions, can cause different CCT, which can be detected by the ALS data as well. In this context, an example algorithm for an improved understanding of scene lighting conditions includes: 1. Compute light flicker for frequency (Lf) and magnitude (Lm). 2. Compute the whole light spectrum magnitude (Sm). 3. Fit the whole light spectrum with the sunlight spectrum, and fit the rest of the spectrum with the artificial light magnitude. These criteria provide the ratio of natural light to artificial light in a scene for differentiating the light sources.  Additionally, the sky can be detected if the region takes a certain part of the frame, or a convolutional neural network may be leveraged if deemed feasible. Once the light source is understood and differentiated to identify whether it is natural light or artificial light, different weights are mapped to the dynamic range detected, and the information is integrated into the exposure control algorithm in the HDR imaging pipeline.  For efficiency, exposure metering and control algorithms generally accept a very small resolution of statistics as input.  Thus, regions that are close to saturation are easily identified, and their strength is measured.  In addition, a histogram of the middle region of the image frame may be identified, which is usually the main region of interest.  All these factors affect the decisions for the improved exposure control algorithm. 7Defensive Publications Series, Art. 3092 [2020]https://www.tdcommons.org/dpubs_series/3092In one example, this luminance information is integrated into the exposure control algorithm using an example-based method, including leveraging a scene descriptor for control matching.  The new luminance information is incorporated as an additional feature in the descriptor.  The algorithm takes input from the image frame statistics, in addition to the illuminance understanding, e.g., the light source condition and scene perception (such as sky detection), to evaluate the scene dynamic range as well as the weight of the dynamic range.  The exposure control can also be modeled as a machine learning problem, e.g., using reinforcement learning, to make it more general-purpose if desired.  When the input has the statistics of each image sub-block, and the overall ratio of artificial and natural light, along with an indicator of the sky, the learning model can be trained with the ground truth of the most pleasant exposure decision.  In this context, false decisions can be the far neighbors of the correct decision, when the model is large, and it can implicitly learn the other semantic (contextual) meaning embedded in the frame. Figure 3 illustrates an example method of scene metering and exposure control for enhanced HDR imaging according to principles of this disclosure. 8Gao et al.: Scene Metering and Exposure Control for Enhancing High Dynamic RaPublished by Technical Disclosure Commons, 2020 Figure 4 illustrates examples of imaging devices that may utilize the scene metering and exposure control method and technology of this disclosure.  In the example of Figure 4, the imaging device is illustrated as being a camera, a smartphone, a video recorder, or a tablet computer.  The imaging device may include one or more optical elements (e.g., a lens, a mechanical shutter, an electrical shutter, an aperture).  The imaging device includes an image sensor (e.g., a complementary metal-oxide-semiconductor (CMOS) image sensor, or a charged-couple device (CCD) image sensor) for detecting information used to make an image.  The imaging device also includes a spectral sensor to detect artificial light flicker, environmental luminance (lux range), and color identification.  The imaging device further includes at least one processor (e.g., a central processing unit, an image processor for processing images).  The imaging device 9Defensive Publications Series, Art. 3092 [2020]https://www.tdcommons.org/dpubs_series/3092may also include a display for displaying a user interface.  The user interface may be configured to receive input from a user of the imaging device.  The user interface may include one or more of a touchscreen, a button, a dial, or a keypad.  The imaging device also includes executable instructions that perform functions as described in this disclosure, including to identify and differentiate light sources, affect exposure control, and incorporate image adjustments into the HDR imaging pipeline.  These instructions may be implemented in hardware, or on the computer-readable media of the imaging device (as shown), which may include any suitable memory or storage device such as random-access memory (RAM), read-only (ROM), flash memory, or a solid-state drive (SSD). 10Gao et al.: Scene Metering and Exposure Control for Enhancing High Dynamic RaPublished by Technical Disclosure Commons, 2020 To summarize, spectral sensor-based (whole scene illuminance) and image-based detection results (localization-based information) are leveraged for an improved method of detecting scene content, including luminance highlights.  This detection identifies if the highlight region is natural light (e.g., sky, or a high reflection of natural light) or an artificial light region.  A statistical map is generated to mark the highlight region(s).  The highlight region(s) are then marked with different priorities (weights).  For example, artificial light regions may be marked to overexpose, and the 11Defensive Publications Series, Art. 3092 [2020]https://www.tdcommons.org/dpubs_series/3092sky may be marked to be well-exposed or balanced with other exposure control metrics.  With this solution, light and scene understanding achieve improved exposure control and enhanced HDR imaging. References: [1] Whitson Gordon.  “What is HDR and When Should I Use It In My Photos?”  Lifehacker. June 12, 2019. https://lifehacker.com/what-is-hdr-and-when-should-i-use-it-in-my-photos-5991508 [2] Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy.  2016.  Burst photography for high dynamic range and low-light imaging on mobile cameras.  ACM Trans. Graph. 35, 6, Article 192 (November 2016), 12 pages.  https://doi.org/10.1145/2980179.2980254. [3] Wikipedia contributors, ‘Reinforcement Learning,’ Wikipedia, The Free Encyclopedia, https://en.wikipedia.org/wiki/Reinforcement_learning.  [4] Astrophotography with Night Sight on Pixel Phones, Florian Kainz et. al. November 26, 2019.  https://ai.googleblog.com/2019/11/astrophotography-with-night-sight-on.html. [5] Patent Publication: US 20120002074 A1; Image processing apparatus, signal processing apparatus, and program; Priority Date: January 29, 2010. [6] Patent Publication: US 20190130630 A1; Quotidian scene reconstruction engine; Priority Date: April 12, 2016.  [7] Patent Publication: US 20120051635 A1; Light detection, color appearance models, and modifying dynamic range for image display; Priority Date: May 11, 2009.  12Gao et al.: Scene Metering and Exposure Control for Enhancing High Dynamic RaPublished by Technical Disclosure Commons, 2020

Scene Metering and Exposure Control for Enhancing High Dynamic Range Imaging

https://www.tdcommons.org/cgi/viewcontent.cgi?article=4164&amp;context=dpubs_series

Scene Metering and Exposure Control for Enhancing High Dynamic Range Imaging

Abstract

Similar works

Full text

Available Versions

Technical Disclosure Common