Improving Sketch Colorization using Adversarial Segmentation Consistency

We propose a new method for producing color images from sketches. Current solutions in sketch colorization either necessitate additional user instruction or are restricted to the "paired" translation strategy. We leverage semantic image segmentation from a general-purpose panoptic segmentation network to generate an additional adversarial loss function. The proposed loss function is compatible with any GAN model. Our method is not restricted to datasets with segmentation labels and can be applied to unpaired translation tasks as well. Using qualitative, and quantitative analysis, and based on a user study, we demonstrate the efficacy of our method on four distinct image datasets. On the FID metric, our model improves the baseline by up to 35 points. Our code, pretrained models, scripts to produce newly introduced datasets and corresponding sketch images are available at https://github.com/giddyyupp/AdvSegLoss.Comment: Under review at Pattern Recognition Letters. arXiv admin note: substantial text overlap with arXiv:2102.0619

A survey of comics research in computer science

Graphical novels such as comics and mangas are well known all over the world. The digital transition started to change the way people are reading comics, more and more on smartphones and tablets and less and less on paper. In the recent years, a wide variety of research about comics has been proposed and might change the way comics are created, distributed and read in future years. Early work focuses on low level document image analysis: indeed comic books are complex, they contains text, drawings, balloon, panels, onomatopoeia, etc. Different fields of computer science covered research about user interaction and content generation such as multimedia, artificial intelligence, human-computer interaction, etc. with different sets of values. We propose in this paper to review the previous research about comics in computer science, to state what have been done and to give some insights about the main outlooks

TextureGAN: Controlling Deep Image Synthesis with Texture Patches

In this paper, we investigate deep image synthesis guided by sketch, color, and texture. Previous image synthesis methods can be controlled by sketch and color strokes but we are the first to examine texture control. We allow a user to place a texture patch on a sketch at arbitrary locations and scales to control the desired output texture. Our generative network learns to synthesize objects consistent with these texture suggestions. To achieve this, we develop a local texture loss in addition to adversarial and content loss to train the generative network. We conduct experiments using sketches generated from real images and textures sampled from a separate texture database and results show that our proposed algorithm is able to generate plausible images that are faithful to user controls. Ablation studies show that our proposed pipeline can generate more realistic images than adapting existing methods directly.Comment: CVPR 2018 spotligh

Image-to-Image Translation with Conditional Adversarial Networks

We investigate conditional adversarial networks as a general-purpose solution to image-to-image translation problems. These networks not only learn the mapping from input image to output image, but also learn a loss function to train this mapping. This makes it possible to apply the same generic approach to problems that traditionally would require very different loss formulations. We demonstrate that this approach is effective at synthesizing photos from label maps, reconstructing objects from edge maps, and colorizing images, among other tasks. Indeed, since the release of the pix2pix software associated with this paper, a large number of internet users (many of them artists) have posted their own experiments with our system, further demonstrating its wide applicability and ease of adoption without the need for parameter tweaking. As a community, we no longer hand-engineer our mapping functions, and this work suggests we can achieve reasonable results without hand-engineering our loss functions either.Comment: Website: https://phillipi.github.io/pix2pix/, CVPR 201

Semantic Photo Manipulation with a Generative Image Prior

Despite the recent success of GANs in synthesizing images conditioned on inputs such as a user sketch, text, or semantic labels, manipulating the high-level attributes of an existing natural photograph with GANs is challenging for two reasons. First, it is hard for GANs to precisely reproduce an input image. Second, after manipulation, the newly synthesized pixels often do not fit the original image. In this paper, we address these issues by adapting the image prior learned by GANs to image statistics of an individual image. Our method can accurately reconstruct the input image and synthesize new content, consistent with the appearance of the input image. We demonstrate our interactive system on several semantic image editing tasks, including synthesizing new objects consistent with background, removing unwanted objects, and changing the appearance of an object. Quantitative and qualitative comparisons against several existing methods demonstrate the effectiveness of our method.Comment: SIGGRAPH 201

변형된 FusionNet을 이용한 회색조 이미지의 자연스러운 채색

학위논문 (석사) -- 서울대학교 대학원 : 자연과학대학 협동과정 계산과학전공, 2021. 2. 강명주.In this paper, we propose a grayscale image colorizing technique. The colorization task can be divided into three main ways, the Scribble-based method, Exemplar-based method and Fully automatic method. Our proposed method is included in the third one. We use a deep learning model that is widely used in the colorization eld recently. We propose Encoder-Docoder model using Convolutional Neural Networks. In particular, we modify the FusionNet with good performance to suit this purpose. Also, in order to get better results, we do not use MSE loss function. Instead, we use the loss function suitable for the colorizing purpose. We use a subset of the ImageNet dataset as the training, validation and test dataset. We take some existing methods from Fully automatic Deep Learning method and compared them with our models. Our algorithm is evaluated using a quantitative metric called PSNR (Peak Signal-to-Noise Ratio). In addition, in order to evaluate the results qualitatively, our model was applied to the test dataset and compared with various other models. Our model has better performance both quantitatively and qualitatively than other models. Finally, we apply our model to old black and white photographs.본 논문에서는 회색조 이미지들에 대한 채색 기법을 제안한다. 채색 작업은 크게 Scribble 기반 방법, Exemplar 기반 방법, 완전 자동 방법의 세 가지로 나눌 수 있다. 본 논문에서는 세 번째 방법을 사용했다. 최근에 채색 분야에서 널리 사용되는 딥 러닝 모델을 사용한다. Convolutional Neural Networks를 이용한 Encoder-Docoder 모델을 제안한다. 특히 기존에 image segmetation 분야에서 좋은 성능을 보이는 FusionNet을 자동 채색 목적에 맞게 다양한 방법으로 수정했다. 또한 더 나은 결과를 얻기 위해 MSE 손실 함수를 사용하지 않았다. 대신, 우리는 자동 채색 목적에 적합한 손실 함수를 사용하였다. ImageNet 데이터셋의 부분 집합을 훈련, 검증 및 테스트 데이터셋으로 사용했다. 우리는 완전 자동 딥 러닝 방법에서 기존 방법을 가져와 우리의 모델과 비교했다. 우리의 알고리즘은 PSNR (Peak Signal-to-Noise Ratio)이라는 정량적 지표를 사용하여 평가되었다. 또한 결과를 정성적으로 평가하기 위해 테스트 데이터셋에 모델을 적용하여 다양한 모델과 비교했다. 그 결과 다른 모델에 비해 정성적으로도, 정량적으로도 좋은 성능을 보였다. 마지막으로 오래된 흑백 사진과 같은 다양한 유형의 이미지에 적용한 결과를 제시했다.Abstract i 1 Introduction 1 2 Related Works 4 2.1 Scribble-based method . . . . . . . . . . . . . . . . . . . . . . 4 2.2 Exemplar-based method . . . . . . . . . . . . . . . . . . . . . 5 2.3 Fully automatic method . . . . . . . . . . . . . . . . . . . . . 6 3 Proposed Method 8 3.1 Method Overview . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2 Loss Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.3 Architecture detail . . . . . . . . . . . . . . . . . . . . . . . . 10 3.3.1 Encoder . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.3.2 Decoder . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.3.3 Bridge . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 4 Experiments 14 4.1 CIE Lab Color Space . . . . . . . . . . . . . . . . . . . . . . . 15 4.2 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.3 Qualitative Evaluation . . . . . . . . . . . . . . . . . . . . . . 17 4.4 Quantitative Evaluation . . . . . . . . . . . . . . . . . . . . . 18 4.5 Legacy Old image Colorization . . . . . . . . . . . . . . . . . . 20 5 Conclusion 23 The bibliography 24 Abstract (in Korean) 28Maste

A review of image and video colorization: From analogies to deep learning

Image colorization is a classic and important topic in computer graphics, where the aim is to add color to a monochromatic input image to produce a colorful result. In this survey, we present the history of colorization research in chronological order and summarize popular algorithms in this field. Early works on colorization mostly focused on developing techniques to improve the colorization quality. In the last few years, researchers have considered more possibilities such as combining colorization with NLP (natural language processing) and focused more on industrial applications. To better control the color, various types of color control are designed, such as providing reference images or color-scribbles. We have created a taxonomy of the colorization methods according to the input type, divided into grayscale, sketch-based and hybrid. The pros and cons are discussed for each algorithm, and they are compared according to their main characteristics. Finally, we discuss how deep learning, and in particular Generative Adversarial Networks (GANs), has changed this field