Aesthetic-Driven Image Enhancement by Adversarial Learning

We introduce EnhanceGAN, an adversarial learning based model that performs automatic image enhancement. Traditional image enhancement frameworks typically involve training models in a fully-supervised manner, which require expensive annotations in the form of aligned image pairs. In contrast to these approaches, our proposed EnhanceGAN only requires weak supervision (binary labels on image aesthetic quality) and is able to learn enhancement operators for the task of aesthetic-based image enhancement. In particular, we show the effectiveness of a piecewise color enhancement module trained with weak supervision, and extend the proposed EnhanceGAN framework to learning a deep filtering-based aesthetic enhancer. The full differentiability of our image enhancement operators enables the training of EnhanceGAN in an end-to-end manner. We further demonstrate the capability of EnhanceGAN in learning aesthetic-based image cropping without any groundtruth cropping pairs. Our weakly-supervised EnhanceGAN reports competitive quantitative results on aesthetic-based color enhancement as well as automatic image cropping, and a user study confirms that our image enhancement results are on par with or even preferred over professional enhancement

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs

We present a new method for synthesizing high-resolution photo-realistic images from semantic label maps using conditional generative adversarial networks (conditional GANs). Conditional GANs have enabled a variety of applications, but the results are often limited to low-resolution and still far from realistic. In this work, we generate 2048x1024 visually appealing results with a novel adversarial loss, as well as new multi-scale generator and discriminator architectures. Furthermore, we extend our framework to interactive visual manipulation with two additional features. First, we incorporate object instance segmentation information, which enables object manipulations such as removing/adding objects and changing the object category. Second, we propose a method to generate diverse results given the same input, allowing users to edit the object appearance interactively. Human opinion studies demonstrate that our method significantly outperforms existing methods, advancing both the quality and the resolution of deep image synthesis and editing.Comment: v2: CVPR camera ready, adding more results for edge-to-photo example

Cross-Section Bead Image Prediction in Laser Keyhole Welding of AISI 1020 Steel Using Deep Learning Architectures

A deep learning model was applied for predicting a cross-sectional bead image from laser welding process parameters. The proposed model consists of two successive generators. The first generator produces a weld bead segmentation map from laser intensity and interaction time, which is subsequently translated into an optical microscopic (OM) image by the second generator. Both generators exhibit an encoder & x2013;decoder structure based on a convolutional neural network (CNN). In the second generator, a conditional generative adversarial network (cGAN) was additionally employed with multiscale discriminators and residual blocks, considering the size of the OM image. For a training dataset, laser welding experiments with AISI 1020 steel were conducted on a large process window using a 2 KW fiber laser, and a total of 39 process conditions were used for the training. High-resolution OM images were successfully generated, and the predicted bead shapes were reasonably accurate (R-Squared: 89.0 & x0025; for penetration depth, 93.6 & x0025; for weld bead area)

WESPE: Weakly Supervised Photo Enhancer for Digital Cameras

Low-end and compact mobile cameras demonstrate limited photo quality mainly due to space, hardware and budget constraints. In this work, we propose a deep learning solution that translates photos taken by cameras with limited capabilities into DSLR-quality photos automatically. We tackle this problem by introducing a weakly supervised photo enhancer (WESPE) - a novel image-to-image Generative Adversarial Network-based architecture. The proposed model is trained by under weak supervision: unlike previous works, there is no need for strong supervision in the form of a large annotated dataset of aligned original/enhanced photo pairs. The sole requirement is two distinct datasets: one from the source camera, and one composed of arbitrary high-quality images that can be generally crawled from the Internet - the visual content they exhibit may be unrelated. Hence, our solution is repeatable for any camera: collecting the data and training can be achieved in a couple of hours. In this work, we emphasize on extensive evaluation of obtained results. Besides standard objective metrics and subjective user study, we train a virtual rater in the form of a separate CNN that mimics human raters on Flickr data and use this network to get reference scores for both original and enhanced photos. Our experiments on the DPED, KITTI and Cityscapes datasets as well as pictures from several generations of smartphones demonstrate that WESPE produces comparable or improved qualitative results with state-of-the-art strongly supervised methods

Multiple conserved regulatory domains promote Fezf2 expression in the developing cerebral cortex.

BackgroundThe genetic programs required for development of the cerebral cortex are under intense investigation. However, non-coding DNA elements that control the expression of developmentally important genes remain poorly defined. Here we investigate the regulation of Fezf2, a transcription factor that is necessary for the generation of deep-layer cortical projection neurons.ResultsUsing a combination of chromatin immunoprecipitation followed by high throughput sequencing (ChIP-seq) we mapped the binding of four deep-layer-enriched transcription factors previously shown to be important for cortical development. Building upon this we characterized the activity of three regulatory regions around the Fezf2 locus at multiple stages throughout corticogenesis. We identified a promoter that was sufficient for expression in the cerebral cortex, and enhancers that drove reporter gene expression in distinct forebrain domains, including progenitor cells and cortical projection neurons.ConclusionsThese results provide insight into the regulatory logic controlling Fezf2 expression and further the understanding of how multiple non-coding regulatory domains can collaborate to control gene expression in vivo

Pluripotency factors functionally premark cell-type-restricted enhancers in ES cells.

Enhancers for embryonic stem (ES) cell-expressed genes and lineage-determining factors are characterized by conventional marks of enhancer activation in ES cells1-3, but it remains unclear whether enhancers destined to regulate cell-type-restricted transcription units might also have distinct signatures in ES cells. Here we show that cell-type-restricted enhancers are 'premarked' and activated as transcription units by the binding of one or two ES cell transcription factors, although they do not exhibit traditional enhancer epigenetic marks in ES cells, thus uncovering the initial temporal origins of cell-type-restricted enhancers. This premarking is required for future cell-type-restricted enhancer activity in the differentiated cells, with the strength of the ES cell signature being functionally important for the subsequent robustness of cell-type-restricted enhancer activation. We have experimentally validated this model in macrophage-restricted enhancers and neural precursor cell (NPC)-restricted enhancers using ES cell-derived macrophages or NPCs, edited to contain specific ES cell transcription factor motif deletions. DNA hydroxyl-methylation of enhancers in ES cells, determined by ES cell transcription factors, may serve as a potential molecular memory for subsequent enhancer activation in mature macrophages. These findings suggest that the massive repertoire of cell-type-restricted enhancers are essentially hierarchically and obligatorily premarked by binding of a defining ES cell transcription factor in ES cells, dictating the robustness of enhancer activation in mature cells

Residual magnifier: A dense information flow network for super resolution

© 2019 IEEE. Recently, deep learning methods have been successfully applied to single image super-resolution tasks. However, some networks with extreme depth failed to achieve better performance because of the insufficient utilization of the local residual information extracted at each stage. To solve the above question, we propose a Dense Information Flow Network (DIF-Net), which can fully extract and utilize the local residual information at each stage to accomplish a better reconstruction. Specifically, we present a Two-stage Residual Extraction Block (TREB) to extract the shallow and deep local residual information at each stage. The dense connection mechanism is introduced throughout the model and within TREBs to dramatically increase the information flow. Meanwhile this mechanism prevents the shallow features extracted earlier from being diluted. Finally, we propose a lightweight subnet (residual enhancer) to efficiently recycle the overflow residual information from the backbone net for detail enhancement of the residual image. Experimental results demonstrate that the proposed method performs favorably against the state-of-the-art methods with relatively-less parameters