1,403 research outputs found
Blind image quality assessment: from heuristic-based to learning-based
Image quality assessment (IQA) plays an important role in numerous digital image
processing applications, including image compression, image transmission, and image
restoration, etc. The goal of objective IQA is to develop computational models that
can predict image quality in a way being consistent with human perception. Compared
with subjective quality evaluations such as psycho-visual tests, objective IQA
metrics have the advantages of predicting image quality automatically and effectively
in a timely manner.
This thesis focuses on a particular type of objective IQA – blind IQA (BIQA),
where the developed methods not only achieve objective IQA, but also are able to
assess the perceptual quality of digital images without access to their pristine reference
counterparts. Firstly, a novel blind image sharpness evaluator is introduced
in Chapter 3, which leverages the discrepancy measures of structural degradation.
Secondly, a “completely blind” quality assessment metric for gamut-mapped images
is designed in Chapter 4, which does not need subjective quality scores during the
model training. Thirdly, a general-purpose BIQA method is presented in Chapter 5,
which can evaluate the quality of digital images without prior knowledge on the types
of distortions. Finally, in Chapter 6, a deep neural network-based general-purpose
BIQA method is proposed, which is fully data driven and trained in an end-to-end
manner.
In summary, four BIQA methods are introduced in this thesis, where the first three
are heuristic-based and the last one is learning-based. Unlike heuristics-based ones,
the learning-based method does not involves manually engineered feature designs
HDR-ChipQA: No-Reference Quality Assessment on High Dynamic Range Videos
We present a no-reference video quality model and algorithm that delivers
standout performance for High Dynamic Range (HDR) videos, which we call
HDR-ChipQA. HDR videos represent wider ranges of luminances, details, and
colors than Standard Dynamic Range (SDR) videos. The growing adoption of HDR in
massively scaled video networks has driven the need for video quality
assessment (VQA) algorithms that better account for distortions on HDR content.
In particular, standard VQA models may fail to capture conspicuous distortions
at the extreme ends of the dynamic range, because the features that drive them
may be dominated by distortions {that pervade the mid-ranges of the signal}. We
introduce a new approach whereby a local expansive nonlinearity emphasizes
distortions occurring at the higher and lower ends of the {local} luma range,
allowing for the definition of additional quality-aware features that are
computed along a separate path. These features are not HDR-specific, and also
improve VQA on SDR video contents, albeit to a reduced degree. We show that
this preprocessing step significantly boosts the power of distortion-sensitive
natural video statistics (NVS) features when used to predict the quality of HDR
content. In similar manner, we separately compute novel wide-gamut color
features using the same nonlinear processing steps. We have found that our
model significantly outperforms SDR VQA algorithms on the only publicly
available, comprehensive HDR database, while also attaining state-of-the-art
performance on SDR content
Recommended from our members
Subjective and objective quality evaluation of synthetic and high dynamic range images
Recent years have seen a huge growth in the acquisition, transmission, and storage of videos. The visual data consists of both natural scenes as well as synthetic scenes, such as animated movies, cartoons and video games. In all these cases, the ultimate goal is to provide the viewers with a satisfactory quality-of-experience. In addition to the traditional 8-bit images, high dynamic range imaging is also becoming popular because of its ability to represent the real world luminances more realistically. Coming up with objective image quality assessment algorithms for these applications is an interesting research problem. In this work, I have developed a synthetic image quality database by introducing varying degrees of different types of distortions and conducted a subjective experiment in order to obtain the ground-truth data. I evaluated the performance of state-of-the-art image quality assessment algorithms (typically meant for natural images) on this database, especially no-reference algorithms that have not been applied to the domain of computer graphics images before. I identified the top-performing algorithms along with analyzing the types of distortions on which the present algorithms show a less impressive performance. For high dynamic range(HDR) images, I have designed two new full-reference image quality assessment algorithms to judge the quality of tonemapped HDR images using statistical features extracted from them. I have also conducted a massive online crowd-sourced subjective test for HDR image artifacts arising from tonemapping, multiple-exposure fusion and post processing. To the best of our knowledge, presently this is the largest HDR image database in the world involving the largest number of source images and most number of human evaluations. Based on the subjective evaluations obtained, I have also proposed machine learning based no-reference image quality assessment algorithms to predict the perceptual quality of HDR images.Electrical and Computer Engineerin
Adaptive Methods for Color Vision Impaired Users
Color plays a key role in the understanding of the information in computer environments. It
happens that about 5% of the world population is affected by color vision deficiency (CVD),
also called color blindness. This visual impairment hampers the color perception, ending up by
limiting the overall perception that CVD people have about the surrounding environment, no
matter it is real or virtual. In fact, a CVD individual may not distinguish between two different
colors, what often originates confusion or a biased understanding of the reality, including web
environments, whose web pages are plenty of media elements like text, still images, video,
sprites, and so on.
Aware of the difficulties that color-blind people may face in interpreting colored contents,
a significant number of recoloring algorithms have been proposed in the literature with the
purpose of improving the visual perception of those people somehow. However, most of those
algorithms lack a systematic study of subjective assessment, what undermines their validity, not
to say usefulness. Thus, in the sequel of the research work behind this Ph.D. thesis, the central
question that needs to be answered is whether recoloring algorithms are of any usefulness and
help for colorblind people or not.
With this in mind, we conceived a few preliminary recoloring algorithms that were published in
conference proceedings elsewhere. Except the algorithm detailed in Chapter 3, these conference
algorithms are not described in this thesis, though they have been important to engender
those presented here. The first algorithm (Chapter 3) was designed and implemented for people
with dichromacy to improve their color perception. The idea is to project the reddish hues onto
other hues that are perceived more regularly by dichromat people.
The second algorithm (Chapter 4) is also intended for people with dichromacy to improve their
perception of color, but its applicability covers the adaptation of text and image, in HTML5-
compliant web environments. This enhancement of color contrast of text and imaging in web
pages is done while keeping the naturalness of color as much as possible. Also, to the best of our
knowledge, this is the first web recoloring approach targeted to dichromat people that takes
into consideration both text and image recoloring in an integrated manner.
The third algorithm (Chapter 5) primarily focuses on the enhancement of some of the object
contours in still images, instead of recoloring the pixels of the regions bounded by such contours.
Enhancing contours is particularly suited to increase contrast in images, where we find adjacent
regions that are color indistinguishable from dichromat’s point of view. To our best knowledge,
this is one of the first algorithms that take advantage of image analysis and processing techniques
for region contours.
After accurate subjective assessment studies for color-blind people, we concluded that the CVD
adaptation methods are useful in general. Nevertheless, each method is not efficient enough to
adapt all sorts of images, that is, the adequacy of each method depends on the type of image
(photo-images, graphical representations, etc.).
Furthermore, we noted that the experience-based perceptual learning of colorblind people
throughout their lives determines their visual perception. That is, color adaptation algorithms must satisfy requirements such as color naturalness and consistency, to ensure that dichromat
people improve their visual perception without artifacts. On the other hand, CVD adaptation
algorithms should be object-oriented, instead of pixel-oriented (as typically done), to select
judiciously pixels that should be adapted. This perspective opens an opportunity window for
future research in color accessibility in the field of in human-computer interaction (HCI).A cor desempenha um papel fundamental na compreensão da informação em ambientes computacionais.
Porém, cerca de 5% da população mundial é afetada pela deficiência de visão de
cor (ou Color Vision Deficiency (CVD), do InglĂŞs), correntemente designada por daltonismo. Esta
insuficiĂŞncia visual dificulta a perceção das cores, o que limita a perceção geral que os indivĂduos
tĂŞm sobre o meio, seja real ou virtual. Efetivamente, um indivĂduo com CVD vĂŞ como iguais
cores que sĂŁo diferentes, o que origina confusĂŁo ou uma compreensĂŁo distorcida da realidade,
assim como dos ambientes web, onde existe uma abundância de conteúdos média coloridos,
como texto, imagens fixas e vĂdeo, entre outros.
Com o intuito de mitigar as dificuldades que as pessoas com CVD enfrentam na interpretação de
conteĂşdos coloridos, tem sido proposto na literatura um nĂşmero significativo de algoritmos de
recoloração, que têm como o objetivo melhorar, de alguma forma, a perceção visual de pessoas
com CVD. Porém, a maioria desses trabalhos carece de um estudo sistemático de avaliação
subjetiva, o que põe em causa a sua validação, se não mesmo a sua utilidade. Assim, a principal
questão à qual se pretende responder, como resultado do trabalho de investigação subjacente
a esta tese de doutoramento, é se os algoritmos de recoloração têm ou não uma real utilidade,
constituindo assim uma ajuda efetiva Ă s pessoas com daltonismo.
Tendo em mente esta questão, concebemos alguns algoritmos de recoloração preliminares que
foram publicados em atas de conferĂŞncias. Com exceção do algoritmo descrito no CapĂtulo 3,
esses algoritmos não são descritos nesta tese, não obstante a sua importância na conceção
daqueles descritos nesta dissertação. O primeiro algoritmo (CapĂtulo 3) foi projetado e implementado
para pessoas com dicromacia, a fim de melhorar a sua perceção da cor. A ideia consiste
em projetar as cores de matiz avermelhada em matizes que sĂŁo melhor percebidos pelas pessoas
com os tipos de daltonismo em causa.
O segundo algoritmo (CapĂtulo 4) tambĂ©m se destina a melhorar a perceção da cor por parte de
pessoas com dicromacia, porém a sua aplicabilidade abrange a adaptação de texto e imagem,
em ambientes web compatĂveis com HTML5. Isto Ă© conseguido atravĂ©s do realce do contraste
de cores em blocos de texto e em imagens, em páginas da web, mantendo a naturalidade da
cor tanto quanto possĂvel. AlĂ©m disso, tanto quanto sabemos, esta Ă© a primeira abordagem de
recoloração em ambiente web para pessoas com dicromacia, que trata o texto e a imagem de
forma integrada.
O terceiro algoritmo (CapĂtulo 5) centra-se principalmente na melhoria de alguns dos contornos
de objetos em imagens, em vez de aplicar a recoloração aos pixels das regiões delimitadas por
esses contornos. Esta abordagem Ă© particularmente adequada para aumentar o contraste em
imagens, quando existem regiões adjacentes que sĂŁo de cor indistinguĂvel sob a perspetiva dos
observadores com dicromacia. Também neste caso, e tanto quanto é do nosso conhecimento,
este é um dos primeiros algoritmos em que se recorre a técnicas de análise e processamento de
contornos de regiões.
Após rigorosos estudos de avaliação subjetiva com pessoas com daltonismo, concluiu-se que os
métodos de adaptação CVD são úteis em geral. No entanto, cada método não é suficientemente
eficiente para todos os tipo de imagens, isto é, o desempenho de cada método depende do tipo de imagem (fotografias, representações gráficas, etc.).
Além disso, notámos que a aprendizagem perceptual baseada na experiência das pessoas daltónicas
ao longo de suas vidas Ă© determinante para perceber aquilo que vĂŞem. Isto significa que os
algoritmos de adaptação de cor devem satisfazer requisitos tais como a naturalidade e a consistência
da cor, de modo a não pôr em causa aquilo que os destinatários consideram razoável
ver no mundo real. Por outro lado, a abordagem seguida na adaptação CVD deve ser orientada
aos objetos, em vez de ser orientada aos pixéis (como tem sido feito até ao momento), de
forma a possibilitar uma seleção mais criteriosa dos pixéis que deverão ser sujeitos ao processo
de adaptação. Esta perspectiva abre uma janela de oportunidade para futura investigação em
acessibilidade da cor no domĂnio da interacção humano-computador (HCI)
On Box-Cox Transformation for Image Normality and Pattern Classification
A unique member of the power transformation family is known as the Box-Cox
transformation. The latter can be seen as a mathematical operation that leads
to finding the optimum lambda ({\lambda}) value that maximizes the
log-likelihood function to transform a data to a normal distribution and to
reduce heteroscedasticity. In data analytics, a normality assumption underlies
a variety of statistical test models. This technique, however, is best known in
statistical analysis to handle one-dimensional data. Herein, this paper
revolves around the utility of such a tool as a pre-processing step to
transform two-dimensional data, namely, digital images and to study its effect.
Moreover, to reduce time complexity, it suffices to estimate the parameter
lambda in real-time for large two-dimensional matrices by merely considering
their probability density function as a statistical inference of the underlying
data distribution. We compare the effect of this light-weight Box-Cox
transformation with well-established state-of-the-art low light image
enhancement techniques. We also demonstrate the effectiveness of our approach
through several test-bed data sets for generic improvement of visual appearance
of images and for ameliorating the performance of a colour pattern
classification algorithm as an example application. Results with and without
the proposed approach, are compared using the AlexNet (transfer deep learning)
pretrained model. To the best of our knowledge, this is the first time that the
Box-Cox transformation is extended to digital images by exploiting histogram
transformation.Comment: The paper has 4 Tables and 6 Figure
Recommended from our members
Automatic assessment and enhancement of streaming video quality under bandwidth and dynamic range limitations
The explosion in the amount of video content being streamed over the internet in recent years has accelerated the demand for effective and efficient methods for assessing and improving the perceptual quality of images and videos while adhering to internet bandwidth and display dynamic range limitations. Objective models of perceptual quality have found extensive use in optimizing video compression and enhancement parameters to achieve desirable streaming fidelity. In this dissertation, we develop a variety of quality modeling and quality enhancement methods targeting the streaming of standard and high dynamic range (SDR/HDR) videos over the internet, subjected to compression and tone mapping. The Visual Multimethod Assessment Fusion (VMAF) algorithm has recently emerged as a state-of-the-art approach to video quality prediction, that now pervades the streaming and social media industry. However, since VMAF requires the evaluation of a heterogeneous set of quality models, it is computationally expensive. Given other advances in hardware-accelerated encoding, quality assessment is emerging as a significant bottleneck in video compression pipelines. Towards alleviating this burden, we first propose a novel Fusion of Unified Quality Evaluators (FUNQUE) framework, by enabling computation sharing and by using a transform that is sensitive to visual perception to boost accuracy. Further, we expand the FUNQUE framework to define a collection of improved low-complexity fused-feature models that advance the state-of-the-art of video quality performance with respect to both accuracy, by 4.2\% to 5.3\%, and computational efficiency, by factors of 3.8 to 11 times! High Dynamic Range (HDR) videos are able to represent wider ranges of contrasts and colors than Standard Dynamic Range (SDR) videos, giving more vivid experiences. Due to this, HDR videos are expected to grow into the dominant video modality of the future. However, HDR videos are incompatible with existing SDR displays, which form the majority of affordable consumer displays on the market. Because of this, HDR videos must be processed by tone-mapping them to reduced bit-depths to service a broad swath of SDR-limited video consumers. Here, we analyzed the impact of tone-mapping operators on the visual quality of streaming HDR videos by building the first large-scale subjectively annotated open-source database of compressed tone-mapped HDR videos, containing 15,000 tone-mapped sequences derived from 40 unique HDR source contents. The videos in the database were labeled with more than 750,000 subjective quality annotations, collected from more than 1,600 unique human observers. We envision that the new LIVE Tone-Mapped HDR (LIVE-TMHDR) database will enable significant progress on HDR video tone mapping and quality assessment in the future. To this end, we make the database freely available to the community at https://live.ece.utexas.edu/research/LIVE_TMHDR/index.html. Server-side tone-mapping involves automating decisions regarding the choices of tone-mapping operators (TMOs) and their parameters to yield high-fidelity outputs. Moreover, these choices must be balanced against the effects of lossy compression, which is ubiquitous in streaming scenarios. To automate this process, we developed a novel, efficient model of objective video quality named Cut-FUNQUE that is able to accurately predict the visual quality of tone-mapped and compressed HDR videos. By evaluating Cut-FUNQUE on the LIVE-TMHDR database, we show that it achieves state-of-the-art accuracy. Finally, the deep learning revolution has strongly impacted low-level image processing tasks such as style/domain transfer, enhancement/restoration, and visual quality assessments. Despite often being treated separately, the aforementioned tasks share a common theme of understanding, editing, or enhancing the appearance of input images without modifying the underlying content. We leverage this observation to develop a novel disentangled representation learning method that decomposes inputs into content and appearance features. The model is trained in a self-supervised manner and we use the learned features to develop a new quality prediction model named DisQUE. We demonstrate through extensive evaluations that DisQUE achieves state-of-the-art accuracy across quality prediction tasks and distortion types. Moreover, we demonstrate that the same features may also be used for image processing tasks such as HDR tone mapping, where the desired output characteristics may be tuned using example input-output pairs.Electrical and Computer Engineerin
Color image quality measures and retrieval
The focus of this dissertation is mainly on color image, especially on the images with lossy compression. Issues related to color quantization, color correction, color image retrieval and color image quality evaluation are addressed. A no-reference color image quality index is proposed. A novel color correction method applied to low bit-rate JPEG image is developed. A novel method for content-based image retrieval based upon combined feature vectors of shape, texture, and color similarities has been suggested. In addition, an image specific color reduction method has been introduced, which allows a 24-bit JPEG image to be shown in the 8-bit color monitor with 256-color display. The reduction in download and decode time mainly comes from the smart encoder incorporating with the proposed color reduction method after color space conversion stage. To summarize, the methods that have been developed can be divided into two categories: one is visual representation, and the other is image quality measure.
Three algorithms are designed for visual representation:
(1) An image-based visual representation for color correction on low bit-rate JPEG images. Previous studies on color correction are mainly on color image calibration among devices. Little attention was paid to the compressed image whose color distortion is evident in low bit-rate JPEG images. In this dissertation, a lookup table algorithm is designed based on the loss of PSNR in different compression ratio.
(2) A feature-based representation for content-based image retrieval. It is a concatenated vector of color, shape, and texture features from region of interest (ROI).
(3) An image-specific 256 colors (8 bits) reproduction for color reduction from 16 millions colors (24 bits). By inserting the proposed color reduction method into a JPEG encoder, the image size could be further reduced and the transmission time is also reduced. This smart encoder enables its decoder using less time in decoding.
Three algorithms are designed for image quality measure (IQM):
(1) A referenced IQM based upon image representation in very low-dimension. Previous studies on IQMs are based on high-dimensional domain including spatial and frequency domains. In this dissertation, a low-dimensional domain IQM based on random projection is designed, with preservation of the IQM accuracy in high-dimensional domain.
(2) A no-reference image blurring metric. Based on the edge gradient, the degree of image blur can be measured.
(3) A no-reference color IQM based upon colorfulness, contrast and sharpness
Intrinsic Image Transfer for Illumination Manipulation
This paper presents a novel intrinsic image transfer (IIT) algorithm for
illumination manipulation, which creates a local image translation between two
illumination surfaces. This model is built on an optimization-based framework
consisting of three photo-realistic losses defined on the sub-layers factorized
by an intrinsic image decomposition. We illustrate that all losses can be
reduced without the necessity of taking an intrinsic image decomposition under
the well-known spatial-varying illumination illumination-invariant reflectance
prior knowledge. Moreover, with a series of relaxations, all of them can be
directly defined on images, giving a closed-form solution for image
illumination manipulation. This new paradigm differs from the prevailing
Retinex-based algorithms, as it provides an implicit way to deal with the
per-pixel image illumination. We finally demonstrate its versatility and
benefits to the illumination-related tasks such as illumination compensation,
image enhancement, and high dynamic range (HDR) image compression, and show the
high-quality results on natural image datasets
The Influence of media displays and image quality attributes for HDR image reproductions
High Dynamic Range (HDR) photography has been in existence at least since the time of Ansel Adams, with his experiments using analog film and darkroom techniques for the production of black and white prints in the 1940\u27s (Ashbrook, 2010). This photographic method has the ability to provide a more accurate representation of a scene through a greater range of the light and dark areas captured in an image. In the mid-20th century HDR Photography it has continued to grow in popularity among those interested in photography wishing to optimize their resulting image beyond a more commonly used technique. Presently, the limitations of commonly available reproduction technologies can lead to unpredictable output results through media such as monitor displays and inkjet prints. The purpose of this research was to determine the influence of quality attributes and image content on the preference of display media for HDR image reproductions. To achieve this purpose, a psychophysical experiment was conducted of 38 observers with previous imaging related exposure. This part of the study consisted of HDR comparisons across both a monitor display device and inkjet prints. Through qualitative and quantitative methods, common trends were identified among observer responses. The results show that for inkjet prints are the most preferred for the output of HDR images, specifically when printed on a metallic substrate. Additionally, the content of displayed images can directly impact display preference depending on the viewer\u27s perception and relationship formed with the photographic image. When evaluating HDR images across two media platforms, quality attributes comprising of a strong influence towards preference are sharpness, naturalness, contrast and highlights while artifacts, physical qualities and shadows were found to have barely any influence. Within the attributes related to HDR, relationships between attributes are found to be significant regarding image evaluation, leading to areas of further research
Symmetry sensitivities of Derivative-of-Gaussian filters
We consider the measurement of image structure using linear filters, in particular derivative-of-Gaussian (DtG) filters, which are an important model of V1 simple cells and widely used in computer vision, and whether such measurements can determine local image symmetry. We show that even a single linear filter can be sensitive to a symmetry, in the sense that specific responses of the filter can rule it out. We state and prove a necessary and sufficient, readily computable, criterion for filter symmetry-sensitivity. We use it to show that the six filters in a second order DtG family have patterns of joint sensitivity which are distinct for 12 different classes of symmetry. This rich symmetry-sensitivity adds to the properties that make DtG filters well-suited for probing local image structure, and provides a set of landmark responses suitable to be the foundation of a nonarbitrary system of feature categories
- …