Pixelated Semantic Colorization

While many image colorization algorithms have recently shown the capability of producing plausible color versions from gray-scale photographs, they still suffer from limited semantic understanding. To address this shortcoming, we propose to exploit pixelated object semantics to guide image colorization. The rationale is that human beings perceive and distinguish colors based on the semantic categories of objects. Starting from an autoregressive model, we generate image color distributions, from which diverse colored results are sampled. We propose two ways to incorporate object semantics into the colorization model: through a pixelated semantic embedding and a pixelated semantic generator. Specifically, the proposed convolutional neural network includes two branches. One branch learns what the object is, while the other branch learns the object colors. The network jointly optimizes a color embedding loss, a semantic segmentation loss and a color generation loss, in an end-to-end fashion. Experiments on PASCAL VOC2012 and COCO-stuff reveal that our network, when trained with semantic segmentation labels, produces more realistic and finer results compared to the colorization state-of-the-art

Multi-task Self-Supervised Visual Learning

We investigate methods for combining multiple self-supervised tasks--i.e., supervised tasks where data can be collected without manual labeling--in order to train a single visual representation. First, we provide an apples-to-apples comparison of four different self-supervised tasks using the very deep ResNet-101 architecture. We then combine tasks to jointly train a network. We also explore lasso regularization to encourage the network to factorize the information in its representation, and methods for "harmonizing" network inputs in order to learn a more unified representation. We evaluate all methods on ImageNet classification, PASCAL VOC detection, and NYU depth prediction. Our results show that deeper networks work better, and that combining tasks--even via a naive multi-head architecture--always improves performance. Our best joint network nearly matches the PASCAL performance of a model pre-trained on ImageNet classification, and matches the ImageNet network on NYU depth prediction.Comment: Published at ICCV 201

변형된 FusionNet을 이용한 회색조 이미지의 자연스러운 채색

학위논문 (석사) -- 서울대학교 대학원 : 자연과학대학 협동과정 계산과학전공, 2021. 2. 강명주.In this paper, we propose a grayscale image colorizing technique. The colorization task can be divided into three main ways, the Scribble-based method, Exemplar-based method and Fully automatic method. Our proposed method is included in the third one. We use a deep learning model that is widely used in the colorization eld recently. We propose Encoder-Docoder model using Convolutional Neural Networks. In particular, we modify the FusionNet with good performance to suit this purpose. Also, in order to get better results, we do not use MSE loss function. Instead, we use the loss function suitable for the colorizing purpose. We use a subset of the ImageNet dataset as the training, validation and test dataset. We take some existing methods from Fully automatic Deep Learning method and compared them with our models. Our algorithm is evaluated using a quantitative metric called PSNR (Peak Signal-to-Noise Ratio). In addition, in order to evaluate the results qualitatively, our model was applied to the test dataset and compared with various other models. Our model has better performance both quantitatively and qualitatively than other models. Finally, we apply our model to old black and white photographs.본 논문에서는 회색조 이미지들에 대한 채색 기법을 제안한다. 채색 작업은 크게 Scribble 기반 방법, Exemplar 기반 방법, 완전 자동 방법의 세 가지로 나눌 수 있다. 본 논문에서는 세 번째 방법을 사용했다. 최근에 채색 분야에서 널리 사용되는 딥 러닝 모델을 사용한다. Convolutional Neural Networks를 이용한 Encoder-Docoder 모델을 제안한다. 특히 기존에 image segmetation 분야에서 좋은 성능을 보이는 FusionNet을 자동 채색 목적에 맞게 다양한 방법으로 수정했다. 또한 더 나은 결과를 얻기 위해 MSE 손실 함수를 사용하지 않았다. 대신, 우리는 자동 채색 목적에 적합한 손실 함수를 사용하였다. ImageNet 데이터셋의 부분 집합을 훈련, 검증 및 테스트 데이터셋으로 사용했다. 우리는 완전 자동 딥 러닝 방법에서 기존 방법을 가져와 우리의 모델과 비교했다. 우리의 알고리즘은 PSNR (Peak Signal-to-Noise Ratio)이라는 정량적 지표를 사용하여 평가되었다. 또한 결과를 정성적으로 평가하기 위해 테스트 데이터셋에 모델을 적용하여 다양한 모델과 비교했다. 그 결과 다른 모델에 비해 정성적으로도, 정량적으로도 좋은 성능을 보였다. 마지막으로 오래된 흑백 사진과 같은 다양한 유형의 이미지에 적용한 결과를 제시했다.Abstract i 1 Introduction 1 2 Related Works 4 2.1 Scribble-based method . . . . . . . . . . . . . . . . . . . . . . 4 2.2 Exemplar-based method . . . . . . . . . . . . . . . . . . . . . 5 2.3 Fully automatic method . . . . . . . . . . . . . . . . . . . . . 6 3 Proposed Method 8 3.1 Method Overview . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2 Loss Function . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.3 Architecture detail . . . . . . . . . . . . . . . . . . . . . . . . 10 3.3.1 Encoder . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3.3.2 Decoder . . . . . . . . . . . . . . . . . . . . . . . . . . 12 3.3.3 Bridge . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 4 Experiments 14 4.1 CIE Lab Color Space . . . . . . . . . . . . . . . . . . . . . . . 15 4.2 Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 4.3 Qualitative Evaluation . . . . . . . . . . . . . . . . . . . . . . 17 4.4 Quantitative Evaluation . . . . . . . . . . . . . . . . . . . . . 18 4.5 Legacy Old image Colorization . . . . . . . . . . . . . . . . . . 20 5 Conclusion 23 The bibliography 24 Abstract (in Korean) 28Maste

ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution

The colorization of grayscale images is an ill-posed problem, with multiple correct solutions. In this paper, we propose an adversarial learning colorization approach coupled with semantic information. A generative network is used to infer the chromaticity of a given grayscale image conditioned to semantic clues. This network is framed in an adversarial model that learns to colorize by incorporating perceptual and semantic understanding of color and class distributions. The model is trained via a fully self-supervised strategy. Qualitative and quantitative results show the capacity of the proposed method to colorize images in a realistic way achieving state-of-the-art results.Comment: 8 pages + reference

Semantic-Sparse Colorization Network for Deep Exemplar-based Colorization

Exemplar-based colorization approaches rely on reference image to provide plausible colors for target gray-scale image. The key and difficulty of exemplar-based colorization is to establish an accurate correspondence between these two images. Previous approaches have attempted to construct such a correspondence but are faced with two obstacles. First, using luminance channels for the calculation of correspondence is inaccurate. Second, the dense correspondence they built introduces wrong matching results and increases the computation burden. To address these two problems, we propose Semantic-Sparse Colorization Network (SSCN) to transfer both the global image style and detailed semantic-related colors to the gray-scale image in a coarse-to-fine manner. Our network can perfectly balance the global and local colors while alleviating the ambiguous matching problem. Experiments show that our method outperforms existing methods in both quantitative and qualitative evaluation and achieves state-of-the-art performance.Comment: Accepted by ECCV2022; 14 pages, 10 figure