Depth-aware Neural Style Transfer using Instance Normalization

Neural Style Transfer (NST) is concerned with the artistic stylization of visual media. It can be described as the process of transferring the style of an artistic image onto an ordinary photograph. Recently, a number of studies have considered the enhancement of the depth-preserving capabilities of the NST algorithms to address the undesired effects that occur when the input content images include numerous objects at various depths. Our approach uses a deep residual convolutional network with instance normalization layers that utilizes an advanced depth prediction network to integrate depth preservation as an additional loss function to content and style. We demonstrate results that are effective in retaining the depth and global structure of content images. Three different evaluation processes show that our system is capable of preserving the structure of the stylized results while exhibiting style-capture capabilities and aesthetic qualities comparable or superior to state-of-the-art methods. Project page: https://ioannoue.github.io/depth-aware-nst-using-in.html.Comment: 8 pages, 8 figures, Computer Graphics & Visual Computing (CGVC) 202

Efficient and effective objective image quality assessment metrics

Acquisition, transmission, and storage of images and videos have been largely increased in recent years. At the same time, there has been an increasing demand for high quality images and videos to provide satisfactory quality-of-experience for viewers. In this respect, high dynamic range (HDR) imaging with higher than 8-bit depth has been an interesting approach in order to capture more realistic images and videos. Objective image and video quality assessment plays a significant role in monitoring and enhancing the image and video quality in several applications such as image acquisition, image compression, multimedia streaming, image restoration, image enhancement and displaying. The main contributions of this work are to propose efficient features and similarity maps that can be used to design perceptually consistent image quality assessment tools. In this thesis, perceptually consistent full-reference image quality assessment (FR-IQA) metrics are proposed to assess the quality of natural, synthetic, photo-retouched and tone-mapped images. In addition, efficient no-reference image quality metrics are proposed to assess JPEG compressed and contrast distorted images. Finally, we propose a perceptually consistent color to gray conversion method, perform a subjective rating and evaluate existing color to gray assessment metrics. Existing FR-IQA metrics may have the following limitations. First, their performance is not consistent for different distortions and datasets. Second, better performing metrics usually have high complexity. We propose in this thesis an efficient and reliable full-reference image quality evaluator based on new gradient and color similarities. We derive a general deviation pooling formulation and use it to compute a final quality score from the similarity maps. Extensive experimental results verify high accuracy and consistent performance of the proposed metric on natural, synthetic and photo retouched datasets as well as its low complexity. In order to visualize HDR images on standard low dynamic range (LDR) displays, tone-mapping operators are used in order to convert HDR into LDR. Given different depth bits of HDR and LDR, traditional FR-IQA metrics are not able to assess the quality of tone-mapped images. The existing full-reference metric for tone-mapped images called TMQI converts both HDR and LDR to an intermediate color space and measure their similarity in the spatial domain. We propose in this thesis a feature similarity full-reference metric in which local phase of HDR is compared with the local phase of LDR. Phase is an important information of images and previous studies have shown that human visual system responds strongly to points in an image where the phase information is ordered. Experimental results on two available datasets show the very promising performance of the proposed metric. No-reference image quality assessment (NR-IQA) metrics are of high interest because in the most present and emerging practical real-world applications, the reference signals are not available. In this thesis, we propose two perceptually consistent distortion-specific NR-IQA metrics for JPEG compressed and contrast distorted images. Based on edge statistics of JPEG compressed images, an efficient NR-IQA metric for blockiness artifact is proposed which is robust to block size and misalignment. Then, we consider the quality assessment of contrast distorted images which is a common distortion. Higher orders of Minkowski distance and power transformation are used to train a low complexity model that is able to assess contrast distortion with high accuracy. For the first time, the proposed model is used to classify the type of contrast distortions which is very useful additional information for image contrast enhancement. Unlike its traditional use in the assessment of distortions, objective IQA can be used in other applications. Examples are the quality assessment of image fusion, color to gray image conversion, inpainting, background subtraction, etc. In the last part of this thesis, a real-time and perceptually consistent color to gray image conversion methodology is proposed. The proposed correlation-based method and state-of-the-art methods are compared by subjective and objective evaluation. Then, a conclusion is made on the choice of the objective quality assessment metric for the color to gray image conversion. The conducted subjective ratings can be used in the development process of quality assessment metrics for the color to gray image conversion and to test their performance

Depth-aware neural style transfer using instance normalization

Neural Style Transfer (NST) is concerned with the artistic stylization of visual media. It can be described as the process of transferring the style of an artistic image onto an ordinary photograph. Recently, a number of studies have considered the enhancement of the depth-preserving capabilities of the NST algorithms to address the undesired effects that occur when the input content images include numerous objects at various depths. Our approach uses a deep residual convolutional network with instance normalization layers that utilizes an advanced depth prediction network to integrate depth preservation as an additional loss function to content and style. We demonstrate results that are effective in retaining the depth and global structure of content images. Three different evaluation processes show that our system is capable of preserving the structure of the stylized results while exhibiting style-capture capabilities and aesthetic qualities comparable or superior to state-of-the-art methods

Computational Aesthetics and Image Enhancements using Deep Neural Networks

Imaging devices have become ubiquitous in modern life, and many of us capture an increasing number of images every day. When we choose to share or store some of these images, our primary selection criterion is to choose the most visually pleasing ones. Yet, quantifying visual pleasantness is a challenge, as image aesthetics not only correlate with low-level image quality, such as contrast, but also high-level visual processes, like composition and context. For most users, a considerable amount of manual effort and/or professional knowledge is required to get aesthetically pleasing images. Developing automatic solutions thus benefits a large community. This thesis proposes several computational approaches to help users obtain the desired images. The first technique aims at automatically measuring the aesthetics quality, which benefits the users in selecting and ranking images. We form the aesthetics prediction problem as a regression task and train a deep neural network on a large image aesthetics dataset. The unbalanced distribution of aesthetics scores in the training set can result in bias of the trained model towards certain aesthetics levels. Therefore, we propose to add sample weights during training to overcome such bias. Moreover, we build a loss function on the histograms of user labels, thus enabling the network to predict not only the average aesthetics quality but also the difficulty of such predictions. Extensive experiments demonstrate that our model outperforms the previous state-of-the-art by a notable margin. Additionally, we propose an image cropping technique that automatically outputs aesthetically pleasing crops. Given an input image and a certain template, we first extract a sufficient amount of candidate crops. These crops are later ranked according to the scores predicted by the pre-trained aesthetics network, after which the best crop is output to the users. We conduct psychophysical experiments to validate the performance. We further present a keyword-based image color re-rendering algorithm. For this task, the colors in the input image are modified to be visually more appealing according to the keyword specified by users. Our algorithm applies local color re-rendering operations to achieve this goal. A novel weakly-supervised semantic segmentation algorithm is developed to locate the keyword-related regions where the color re-rendering operations are applied. The color re-rendering process benefits from the segmentation network in two aspects. Firstly, we achieve more accurate correlation measurements between keywords and color characteristics, contributing to better re-render rendering results of the colors. Secondly, the artifacts caused by the color re-rendering operations are significantly reduced. To avoid the need of keywords when enhancing image aesthetics, we explore generative adversarial networks (GANs) for automatic image enhancement. GANs are known for directly learning the transformations between images from the training data. To learn the image enhancement operations, we train the GANs on an aesthetics dataset with three different losses combined. The first two are standard generative losses that enforce the generated images to be natural and content-wise similar to the input images. We propose a third aesthetics loss that aims at improving the aesthetics quality of the generated images. Overall, the three losses together direct the GANs to apply appropriate image enhancement operations

Development and application of deep learning and spatial statistics within 3D bone marrow imaging

The bone marrow is a highly specialised organ, responsible for the formation of blood cells. Despite 50 years of research, the spatial organisation of the bone marrow remains an area full of controversy and contradiction. One reason for this is that imaging of bone marrow tissue is notoriously difficult. Secondly, efficient methodologies to fully extract and analyse large datasets remain the Achilles heels of imaging-based research. In this thesis I present a pipeline for generating 3D bone marrow images followed by the large-scale data extraction and spatial statistical analysis of the resulting data. Using these techniques, in the context of 3D imaging, I am able to identify and classify the location of hundreds of thousands of cells within various bone marrow samples. I then introduce a series of statistical techniques tailored to work with spatial data, resulting in a 3D statistical map of the tissue from which multi-cellular interactions can be clearly understood. As an illustration of the power of this new approach, I apply this pipeline to diseased samples of bone marrow with a particular focus on leukaemia and its interactions with CD8+ T cells. In so doing I show that this novel pipeline can be used to unravel complex multi-cellular interactions and assist researchers in understanding the processes taking place within the bone marrow.Open Acces

Sustainable photocatalytic oxidation processes for the treatment of emerging microcontaminants

This work investigates the elimination of new and emerging microcontaminants (EMs) from water by means of photochemical oxidation processes, namely heterogeneous and homogeneous photocatalysis. Representative compounds of artificial sweeteners (saccharin, SAC), endocrine disruptors (bisphenol-A, BPA), and pharmaceutica ls (antipyrine, AP) of high environmental persistence and widespread occurrence in the water cycle are used as case studies. Novel concepts that can make photochemica l oxidation a more cost-effective and environmentally benign technology are tested. In Chapter 4, the photocatalytic treatment of SAC and BPA is investigated. Novel submicronic anatase–rutile nanocomposite particles with tuned phase ratio are used as catalysts to increase the photocatalytic performance under UVA irradiation. At the best-assayed conditions (C0 = 3 mg/L, catalyst = 400 mg/L), SAC and BPA are completely degraded within 90 and 150 min of photocatalytic treatment, respectively. [variables: anatase-rutile ratio; initial substrate concentration; catalyst concentration; catalyst reuse; sonication during catalyst recovery] In Chapter 5, a UVA light-emitting diode (UVA-LED) and sunlight are used as irradiation sources to reduce energy requirements and environmental impacts of photocatalytic processes. The photocatalytic degradation of SAC and BPA is studied under UVA irradiation provided by either a UVA-LED or a conventional fluoresce nt blacklight UVA lamp (UVA-BL) and solar irradiation. At the best-assayed conditions (C0 = 2.5 mg/L, TiO2 = 250 mg/L), BPA is completely degraded within 20, 30, and 120 min under UVA-LED, solar, and UVA-BL irradiation, respectively. The treatment time required for the complete elimination of SAC is 20 min under UVA-LED and 90 min under UVA-BL irradiation. [variables: initial substrate concentration; catalyst concentration; water matrix; light source; reactor configuration] In Chapter 6, a comparative study is carried out among the photocatalytic systems of Chapters 4 and 5 in terms of EMs removal, photonic and energy efficiencies. Technica l and economic aspects of all the processes are assessed. LED-driven photocatalysis achieves the highest efficiency in terms of organic removal with the minimum energy consumption, rendering it the most sustainable technology for the treatment of EMs. In Chapter 7, olive mill wastewater (OMW) is used as an iron-chelating agent in the photo-Fenton reaction to obviate the need for water acidification at pH 2.8. Conventional, OMW- and EDDS-assisted photo-Fenton treatment is applied for SAC degradation in a solar compound parabolic collector (CPC). It was found that OMW forms iron complexes able to catalyse H2O2 decomposition and generate hydroxyl radicals. At the optimal OMW dilution (1:800), 90% of SAC is degraded within 75 min. [variables: pH; iron-chelating agent; initial SAC concentration; OMW dilution] In Chapter 8, other complexing and oxidising agents, namely oxalate and persulfate, are used for the intensification of AP degradation during UVA-LED photo-Fenton treatment. Neural networks are applied for process modelling and optimisation. At the optimal conditions (hydrogen peroxide = 100 mg/L, ferrous iron = 20 mg/L, oxalic acid = 100 mg/L), complete degradation of AP and 93% mineralisation is achieved within 2.5 and 60 min, respectively. [variables: initial concentration of hydrogen peroxide, ferrous iron, oxalic acid, persulfate] It is concluded that LED-driven photocatalysis is a sustainable technology for the elimination of EMs from water. Results from this work highlight the need for development and optimisation of engineering proper LED reactors. Furthermore, this work introduces a new concept towards the sustainable operation of photo-Fenton that is based on the use of wastewaters rich in polyphenols instead of pricey and hazardous chemicals for iron chelation. The addition of ferrioxalate complexes is proposed for the intensification of EMs mineralisation during UVA-LED photo-Fenton treatment. Finally, the findings of this work encourage the use of chemometric tools as predictive and optimisation tool

Affect-based Modeling and its Application in Multimedia Analysis Problems

The multimedia domain is undergoing a rapid development phase with transition in audio, image, and video systems such as VoIP, Telepresence, Live/On-Demand Internet Streaming, SecondLife, and many more. In such a situation, the analysis of multimedia systems, like retrieval, quality evaluation, enhancement, summarization, and re-targeting applications, from various context is becoming critical. Current methods for solving the above-mentioned analysis problems do not consider the existence of humans and their affective characteristics in the design methodology. This contradicts the fact that most of the digital media is consumed only by the human end-users. We believe incorporating human feedback during the design and adaptation stage is key to the building process of multimedia systems. In this regard, we observe that affect is an important indicator of human perception and experience. This can be exploited in various ways for designing effective systems that will adapt more closely to the human response. We advocate an affect-based modeling approach for solving multimedia analysis problems by exploring new directions. In this dissertation, we select two representative multimedia analysis problems, e.g. Quality-of-Experience (QoE) evaluation and Image Enhancement in order to derive solutions based on affect-based modeling techniques. We formulate specific hypothesis for them by correlating system parameters to user\u27s affective response, and investigate their roles under varying conditions for each respective scenario. We conducted extensive user studies based on human-to-human interaction through an audio conferencing system.We also conducted user studies based on affective enhancement of images and evaluated the effectiveness of our proposed approaches. Moving forward, multimedia systems will become more media-rich, interactive, and sophisticated and therefore effective solutions for quality, retrieval, and enhancement will be more challenging. Our work thus represents an important step towards the application of affect-based modeling techniques for the future generation of multimedia systems

Remote Sensing for Land Administration 2.0

The reprint “Land Administration 2.0” is an extension of the previous reprint “Remote Sensing for Land Administration”, another Special Issue in Remote Sensing. This reprint unpacks the responsible use and integration of emerging remote sensing techniques into the domain of land administration, including land registration, cadastre, land use planning, land valuation, land taxation, and land development. The title was chosen as “Land Administration 2.0” in reference to both this Special Issue being the second volume on the topic “Land Administration” and the next-generation requirements of land administration including demands for 3D, indoor, underground, real-time, high-accuracy, lower-cost, and interoperable land data and information