Robust estimation of exposure ratios in multi-exposure image stacks

Merging multi-exposure image stacks into a high dynamic range (HDR) image requires knowledge of accurate exposure times. When exposure times are inaccurate, for example, when they are extracted from a camera's EXIF metadata, the reconstructed HDR images reveal banding artifacts at smooth gradients. To remedy this, we propose to estimate exposure ratios directly from the input images. We derive the exposure time estimation as an optimization problem, in which pixels are selected from pairs of exposures to minimize estimation error caused by camera noise. When pixel values are represented in the logarithmic domain, the problem can be solved efficiently using a linear solver. We demonstrate that the estimation can be easily made robust to pixel misalignment caused by camera or object motion by collecting pixels from multiple spatial tiles. The proposed automatic exposure estimation and alignment eliminates banding artifacts in popular datasets and is essential for applications that require physically accurate reconstructions, such as measuring the modulation transfer function of a display. The code for the method is available.Comment: 11 pages, 11 figures, journa

Perceptual model for adaptive local shading and refresh rate

When the rendering budget is limited by power or time, it is necessary to find the combination of rendering parameters, such as resolution and refresh rate, that could deliver the best quality. Variable-rate shading (VRS), introduced in the last generations of GPUs, enables fine control of the rendering quality, in which each 16×16 image tile can be rendered with a different ratio of shader executions. We take advantage of this capability and propose a new method for adaptive control of local shading and refresh rate. The method analyzes texture content, on-screen velocities, luminance, and effective resolution and suggests the refresh rate and a VRS state map that maximizes the quality of animated content under a limited budget. The method is based on the new content-adaptive metric of judder, aliasing, and blur, which is derived from the psychophysical models of contrast sensitivity. To calibrate and validate the metric, we gather data from literature and also collect new measurements of motion quality under variable shading rates, different velocities of motion, texture content, and display capabilities, such as refresh rate, persistence, and angular resolution. The proposed metric and adaptive shading method is implemented as a game engine plugin. Our experimental validation shows a substantial increase in preference of our method over rendering with a fixed resolution and refresh rate, and an existing motion-adaptive techniqu

A perceptual model of motion quality for rendering with adaptive refresh-rate and resolution

Limited GPU performance budgets and transmission bandwidths mean that real-time rendering often has to compromise on the spatial resolution or temporal resolution (refresh rate). A common practice is to keep either the resolution or the refresh rate constant and dynamically control the other variable. But this strategy is non-optimal when the velocity of displayed content varies. To find the best trade-off between the spatial resolution and refresh rate, we propose a perceptual visual model that predicts the quality of motion given an object velocity and predictability of motion. The model considers two motion artifacts to establish an overall quality score: non-smooth (juddery) motion, and blur. Blur is modeled as a combined effect of eye motion, finite refresh rate and display resolution. To fit the free parameters of the proposed visual model, we measured eye movement for predictable and unpredictable motion, and conducted psychophysical experiments to measure the quality of motion from 50 Hz to 165 Hz. We demonstrate the utility of the model with our on-the-fly motion-adaptive rendering algorithm that adjusts the refresh rate of a G-Sync-capable monitor based on a given rendering budget and observed object motion. Our psychophysical validation experiments demonstrate that the proposed algorithm performs better than constant-refresh-rate solutions, showing that motion-adaptive rendering is an attractive technique for driving variable-refresh-rate displays.</jats:p

A visual model for predicting chromatic banding artifacts

Quantization of images containing low texture regions, such as sky, water or skin, can produce banding artifacts. As the bit-depth of each color channel is decreased, smooth image gradients are transformed into perceivable, wide, discrete bands. Commonly used quality metrics cannot reliably measure the visibility of such artifacts. In this paper we introduce a visual model for predicting the visibility of both luminance and chrominance banding artifacts in image gradients spanning between two arbitrary points in a color space. The model analyzes the error introduced by quantization in the Fourier space, and employs a purpose-built spatio-chromatic contrast sensitivity function to predict its visibility. The output of the model is a detection probability, which can be then used to compute the minimum bit-depth for which banding artifacts are just-noticeable. We demonstrate that the model can accurately predict the results of our psychophysical experiments

Suprathreshold contrast matching between different luminance levels

We investigated how perceived achromatic and chromatic contrast changes with luminance. The experiment consisted of test and reference displays viewed haploscopically, where each eye sees one of the displays. Test stimuli presented on the test display on a background of varying luminance levels (0.02, 2, 20, 200, 2000 cd/m2 ) were matched in perceived contrast to reference stimuli presented on a background at a fixed 200 cd/m2 luminance level. We found that approximate contrast constancy holds at photopic luminance levels (20 cd/m2 and above), that is, test stimuli presented at these luminance backgrounds matched when their physical contrasts were the same magnitude as the reference stimulus for most conditions. For lower background luminances, covering an extensive range of 5 log units, much higher physical contrast was required to achieve a match with the reference. This deviation from constancy was larger for lower spatial frequencies and lower pedestal suprathreshold contrasts. Our data provides the basis for new contrast retargeting models for matching appearances across luminance levels

Stereoscopic Depth Perception Through Foliage

Abstract: Both humans and computational methods struggle to discriminate the depth of objects hidden under foliage. However, such discrimination becomes feasible when we combine computational optical synthetic aperture sensing with human’s ability to fuse stereoscopic images. For object identification tasks, as required in search and rescue, wildlife observation, surveillance, or early wildfire detection, depth provides an additional hint to differentiate between true and false findings, such as people, animals, or vehicles vs. sun-heated patches on the ground surfaces or the tree crowns, or ground fires vs. tree trunks. We used video captured by a drone above dense forest to test user’s ability to discriminate depth. We found that discriminating the depth of objects is infeasible when inspecting monoscopic video and relying on motion parallax. This was also impossible for stereoscopic video because of the occlusions from the foliage. However, when the occlusions were reduced with synthetic aperture sensing and disparity-scaled stereoscopic video was presented, human observers were successful in the depth discrimination. At the same time, computational (stereoscopic matching) methods were unsuccessful. This shows the potential of systems which use the synergy of computational methods and human vision to perform tasks that are infeasible for either of them alone

Stereoscopic Depth Perception Through Foliage

Abstract: Both humans and computational methods struggle to discriminate the depth of objects hidden under foliage. However, such discrimination becomes feasible when we combine computational optical synthetic aperture sensing with human’s ability to fuse stereoscopic images. For object identification tasks, as required in search and rescue, wildlife observation, surveillance, or early wildfire detection, depth provides an additional hint to differentiate between true and false findings, such as people, animals, or vehicles vs. sun-heated patches on the ground surfaces or the tree crowns, or ground fires vs. tree trunks. We used video captured by a drone above dense forest to test user’s ability to discriminate depth. We found that discriminating the depth of objects is infeasible when inspecting monoscopic video and relying on motion parallax. This was also impossible for stereoscopic video because of the occlusions from the foliage. However, when the occlusions were reduced with synthetic aperture sensing and disparity-scaled stereoscopic video was presented, human observers were successful in the depth discrimination. At the same time, computational (stereoscopic matching) methods were unsuccessful. This shows the potential of systems which use the synergy of computational methods and human vision to perform tasks that are infeasible for either of them alone

Stereoscopic depth perception through foliage

Acknowledgements: This research was funded by the Austrian Science Fund (FWF) and German Research Foundation (DFG) under grant numbers P32185-NBL and I 6046-N, as well as by the State of Upper Austria and the Austrian Federal Ministry of Education, Science and Research via the LIT-Linz Institute of Technology under grant number LIT2019-8-SEE114.Both humans and computational methods struggle to discriminate the depths of objects hidden beneath foliage. However, such discrimination becomes feasible when we combine computational optical synthetic aperture sensing with the human ability to fuse stereoscopic images. For object identification tasks, as required in search and rescue, wildlife observation, surveillance, and early wildfire detection, depth assists in differentiating true from false findings, such as people, animals, or vehicles vs. sun-heated patches at the ground level or in the tree crowns, or ground fires vs. tree trunks. We used video captured by a drone above dense woodland to test users’ ability to discriminate depth. We found that this is impossible when viewing monoscopic video and relying on motion parallax. The same was true with stereoscopic video because of the occlusions caused by foliage. However, when synthetic aperture sensing was used to reduce occlusions and disparity-scaled stereoscopic video was presented, whereas computational (stereoscopic matching) methods were unsuccessful, human observers successfully discriminated depth. This shows the potential of systems which exploit the synergy between computational methods and human vision to perform tasks that neither can perform alone

Stereoscopic Depth Perception Through Foliage

&lt;p&gt;&lt;strong&gt;Abstract:&lt;/strong&gt;&lt;/p&gt;&lt;p&gt;Both humans and computational methods struggle to discriminate the depths of objects hidden beneath foliage. However, such discrimination becomes feasible when we combine computational optical synthetic aperture sensing with the human ability to fuse stereoscopic images. For object identification tasks, as required in search and rescue, wildlife observation, surveillance, and early wildfire detection, depth assists in differentiating true from false findings, such as people, animals, or vehicles vs. sun-heated patches at the ground level or in the tree crowns, or ground fires vs. tree trunks. We used video captured by a drone above dense woodland to test users' ability to discriminate depth. We found that this is impossible when viewing monoscopic video and relying on motion parallax. The same was true with stereoscopic video because of the occlusions caused by foliage. However, when synthetic aperture sensing was used to reduce occlusions and disparity-scaled stereoscopic video was presented, whereas computational (stereoscopic matching) methods were unsuccessful, human observers successfully discriminated depth. This shows the potential of systems which exploit the synergy between computational methods and human vision to perform tasks that neither can perform alone.&lt;/p&gt