Making Video Quality Assessment Models Robust to Bit Depth

We introduce a novel feature set, which we call HDRMAX features, that when included into Video Quality Assessment (VQA) algorithms designed for Standard Dynamic Range (SDR) videos, sensitizes them to distortions of High Dynamic Range (HDR) videos that are inadequately accounted for by these algorithms. While these features are not specific to HDR, and also augment the equality prediction performances of VQA models on SDR content, they are especially effective on HDR. HDRMAX features modify powerful priors drawn from Natural Video Statistics (NVS) models by enhancing their measurability where they visually impact the brightest and darkest local portions of videos, thereby capturing distortions that are often poorly accounted for by existing VQA models. As a demonstration of the efficacy of our approach, we show that, while current state-of-the-art VQA models perform poorly on 10-bit HDR databases, their performances are greatly improved by the inclusion of HDRMAX features when tested on HDR and 10-bit distorted videos.Comment: Published in IEEE Signal Processing Letters 202

Subjective and objective quality assessment for advanced videos

The surge of video streaming services, particularly for high motion content such as sporting events, necessitates advanced techniques to maintain video quality, facing challenges such as capture artifacts and distortions during coding and transmission. The advent of High Dynamic Range (HDR) content, offering a broader and more accurate representation of brightness and color, poses additional complexities due to increased data volume. The critical need for robust Video Quality Assessment (VQA) models arises from these challenges. To meet this need, we conducted three substantial subjective quality studies and constructed corresponding databases. The Laboratory for Image and Video Engineering (LIVE) Livestream Database comprises 315 videos of 45 source sequences from 33 original contents impaired by six types of distortions. This database facilitated the gathering of over 12,000 human opinions from 40 subjects. The LIVE HDR Database, the first of its kind dedicated to HDR10 videos, includes 310 videos from 31 distinct source sequences, processed with ten different compression and resolution combinations. This resource was instrumental in amassing over 20,000 human quality judgments under two different illumination conditions. An additional LIVE HDR AQ was developed with 400 videos from 40 unique source sequences. These videos were processed using varied compression, resolution combinations, and AQ-mode settings, to study the effects of adaptive quantization (AQ) and rate-distortion optimization techniques on HDR video perceptual quality. Building on these invaluable databases, we developed two innovative objective quality models: HDRMAX and HDRGREED. HDRMAX, a pioneering framework designed to create HDR quality-sensitive features, augments the widely-deployed Video Multimethod Assessment Fusion (VMAF) model, yielding significantly improved performance on both HDR and SDR videos. HDRGREED, a novel model leveraging localized histogram equalization and Difference of Gaussian filters, employs the Generalized Gaussian Distribution to model the bandpass responses and measure the entropy variations between reference and distorted videos. This model is particularly sensitive to banding and blocking artifacts introduced by inappropriate AQ settings. In conclusion, the comprehensive subjective quality studies and databases, along with the state-of-the-art objective quality models, HDRMAX and HDRGREED, significantly contribute to the advancement of future VQA models. These tools cater specifically to challenges posed by live streaming and HDR content, providing critical resources for the development, testing, and comparison of future VQA models. These databases, publicly available for research purposes, and the innovative models offer valuable insights to improve and control the perceptual quality of streamed videos.Electrical and Computer Engineerin

The economic transition and migration of Vietnam and the Mekong Delta region

Relationship between economic transition and migration has long attracted increasing attention of both policy-makers and researchers. Migration is seen as a response of changes during the economic transition in a country, because labour is an important production factor in the market, in which labourers have a desire to move to a place of better working conditions rather than going to a disadvantaged conditions (De Haas, 2010; Todaro, 1980).In this paper I extend this discussion by examining how effects of economic transition on internal migration since the late 1980s. This idea aims at gaining a broader insight into the relationship between economic transition and migration during the renovation processEconomic transition, migration

The quality of experience of emerging display technologies

As new display technologies emerge and become part of everyday life, the understanding of the visual experience they provide becomes more relevant. The cognition of perception is the most vital component of visual experience; however, it is not the only cognition that contributes to the complex overall experience of the end-user. Expectations can create significant cognitive bias that may even override what the user genuinely perceives. Even if a visualization technology is somewhat novel, expectations can be fuelled by prior experiences gained from using similar displays and, more importantly, even a single word or an acronym may induce serious preconceptions, especially if such word suggests excellence in quality. In this interdisciplinary Ph.D. thesis, the effect of minimal, one-word labels on the Quality of Experience (QoE) is investigated in a series of subjective tests. In the studies carried out on an ultra-high-definition (UHD) display, UHD video contents were directly compared to their HD counterparts, with and without labels explicitly informing the test participants about the resolution of each stimulus. The experiments on High Dynamic Range (HDR) visualization addressed the effect of the word “premium” on the quality aspects of HDR video, and also how this may affect the perceived duration of stalling events. In order to support the findings, additional tests were carried out comparing the stalling detection thresholds of HDR video with conventional Low Dynamic Range (LDR) video. The third emerging technology addressed by this thesis is light field visualization. Due to its novel nature and the lack of comprehensive, exhaustive research on the QoE of light field displays and content parameters at the time of this thesis, instead of investigating the labeling effect, four phases of subjective studies were performed on light field QoE. The first phases started with fundamental research, and the experiments progressed towards the concept and evaluation of the dynamic adaptive streaming of light field video, introduced in the final phase

User generated HDR gaming video streaming : dataset, codec comparison and challenges

Gaming video streaming services have grown tremendously in the past few years, with higher resolutions, higher frame rates and HDR gaming videos getting increasingly adopted among the gaming community. Since gaming content as such is different from non-gaming content, it is imperative to evaluate the performance of the existing encoders to help understand the bandwidth requirements of such services, as well as further improve the compression efficiency of such encoders. Towards this end, we present in this paper GamingHDRVideoSET, a dataset consisting of eighteen 10-bit UHD-HDR gaming videos and encoded video sequences using four different codecs, together with their objective evaluation results. The dataset is available online at [to be added after paper acceptance]. Additionally, the paper discusses the codec compression efficiency of most widely used practical encoders, i.e., x264 (H.264/AVC), x265 (H.265/HEVC) and libvpx (VP9), as well the recently proposed encoder libaom (AV1), on 10-bit, UHD-HDR content gaming content. Our results show that the latest compression standard AV1 results in the best compression efficiency, followed by HEVC, H.264, and VP9.Comment: 14 pages, 8 figures, submitted to IEEE journa

Automatic assessment and enhancement of streaming video quality under bandwidth and dynamic range limitations

The explosion in the amount of video content being streamed over the internet in recent years has accelerated the demand for effective and efficient methods for assessing and improving the perceptual quality of images and videos while adhering to internet bandwidth and display dynamic range limitations. Objective models of perceptual quality have found extensive use in optimizing video compression and enhancement parameters to achieve desirable streaming fidelity. In this dissertation, we develop a variety of quality modeling and quality enhancement methods targeting the streaming of standard and high dynamic range (SDR/HDR) videos over the internet, subjected to compression and tone mapping. The Visual Multimethod Assessment Fusion (VMAF) algorithm has recently emerged as a state-of-the-art approach to video quality prediction, that now pervades the streaming and social media industry. However, since VMAF requires the evaluation of a heterogeneous set of quality models, it is computationally expensive. Given other advances in hardware-accelerated encoding, quality assessment is emerging as a significant bottleneck in video compression pipelines. Towards alleviating this burden, we first propose a novel Fusion of Unified Quality Evaluators (FUNQUE) framework, by enabling computation sharing and by using a transform that is sensitive to visual perception to boost accuracy. Further, we expand the FUNQUE framework to define a collection of improved low-complexity fused-feature models that advance the state-of-the-art of video quality performance with respect to both accuracy, by 4.2\% to 5.3\%, and computational efficiency, by factors of 3.8 to 11 times! High Dynamic Range (HDR) videos are able to represent wider ranges of contrasts and colors than Standard Dynamic Range (SDR) videos, giving more vivid experiences. Due to this, HDR videos are expected to grow into the dominant video modality of the future. However, HDR videos are incompatible with existing SDR displays, which form the majority of affordable consumer displays on the market. Because of this, HDR videos must be processed by tone-mapping them to reduced bit-depths to service a broad swath of SDR-limited video consumers. Here, we analyzed the impact of tone-mapping operators on the visual quality of streaming HDR videos by building the first large-scale subjectively annotated open-source database of compressed tone-mapped HDR videos, containing 15,000 tone-mapped sequences derived from 40 unique HDR source contents. The videos in the database were labeled with more than 750,000 subjective quality annotations, collected from more than 1,600 unique human observers. We envision that the new LIVE Tone-Mapped HDR (LIVE-TMHDR) database will enable significant progress on HDR video tone mapping and quality assessment in the future. To this end, we make the database freely available to the community at https://live.ece.utexas.edu/research/LIVE_TMHDR/index.html. Server-side tone-mapping involves automating decisions regarding the choices of tone-mapping operators (TMOs) and their parameters to yield high-fidelity outputs. Moreover, these choices must be balanced against the effects of lossy compression, which is ubiquitous in streaming scenarios. To automate this process, we developed a novel, efficient model of objective video quality named Cut-FUNQUE that is able to accurately predict the visual quality of tone-mapped and compressed HDR videos. By evaluating Cut-FUNQUE on the LIVE-TMHDR database, we show that it achieves state-of-the-art accuracy. Finally, the deep learning revolution has strongly impacted low-level image processing tasks such as style/domain transfer, enhancement/restoration, and visual quality assessments. Despite often being treated separately, the aforementioned tasks share a common theme of understanding, editing, or enhancing the appearance of input images without modifying the underlying content. We leverage this observation to develop a novel disentangled representation learning method that decomposes inputs into content and appearance features. The model is trained in a self-supervised manner and we use the learned features to develop a new quality prediction model named DisQUE. We demonstrate through extensive evaluations that DisQUE achieves state-of-the-art accuracy across quality prediction tasks and distortion types. Moreover, we demonstrate that the same features may also be used for image processing tasks such as HDR tone mapping, where the desired output characteristics may be tuned using example input-output pairs.Electrical and Computer Engineerin

Modeling Perceptual Trade-offs for Designing HDR Displays

Display technology has evolved in pursuit of perceptual pleasure by providing realism and visual impact. The endeavor of the evolution has brought HDR displays to the market. HDR displays, which have become the mainstream display technology recently, are considered not only the present but also the future of displays because of their daunting technical goals: A peak luminance of 10,000 cd/m^2 and near-monochromatic primaries. However, both positive and negative prospects in terms of perceptual aspects for future HDR displays coexist. On the positive side, it is expected that HDR displays will provide better image quality and more vivid color. On the negative side, apart from technical barriers such as production cost and power consumption, HDR displays will induce side effects, for example, observer metamerism, which refers to the phenomenon that color matches for one observer result in color mismatches for other observers. This particular side effect could be a severe issue in HDR displays as their narrow-band primaries likely worsen the color mismatches. Hence, critical to the success of future HDR displays is dealing properly with the perceptual trade-offs. In other words, future HDR display designers need to select physical specifications that maximize perceptual benefits while minimizing adverse effects. This dissertation aims at exploring both potentially positive and negative aspects of future HDR displays, using various perceptual assessments. In particular, the dissertation focuses on two physical factors of a display device: peak luminance and chromaticity color gamut, and the effects of the two factors on related human perception: image quality, observer metamerism, and colorfulness. The ultimate goal of this dissertation is to address the related human perception aroused by the physical factors and propose models to help design future HDR displays. In order to achieve the goal, the dissertation first addresses the image quality trade-off relationship between peak luminance and chromaticity color gamut. A psychophysical experiment was used to develop models to predict equivalent image quality under the trade-off between peak luminance and chromaticity gamut as a function of the perceptual attributes lightness and chroma. Second, a novel approach based on a computational evaluation to investigate potential observer metamerism in HDR displays was explored. This research shows how observer metamerism in HDR displays varies with varying peak luminance and chromaticity color gamut. This research aims at developing a straightforward model to predict observer metamerism in HDR displays based on the computational evaluation. Third, a psychophysical experiment to derive a colorfulness scale for very saturated colors is carried out. This experiment focuses on understanding how the sensitivity of the human visual system responds to highly-saturated colors that extend beyond the stimuli studied in previous research. The colorfulness scale would help both advanced lighting system and display system designers. Fourth, the dissertation suggests an evaluation tool devised based on the observer metamerism and colorfulness scale works that can be utilized to determine the physical specification of HDR displays, maximizing perceptually positive effects while minimizing perceptually negative effects at the same time