심층 신경망 검색 기법을 사용한 이미지 복원

학위논문(박사) -- 서울대학교대학원 : 공과대학 전기·정보공학부, 2021.8. 안준영.Image restoration is an important technology which can be used as a pre-processing step to increase the performances of various vision tasks. Image super-resolution is one of the important task in image restoration which restores a high-resolution (HR) image from low-resolution (LR) observation. The recent progress of deep convolutional neural networks has enabled great success in single image super-resolution (SISR). its performance is also being increased by deepening the networks and developing more sophisticated network structures. However, finding an optimal structure for the given problem is a difficult task, even for human experts. For this reason, neural architecture search (NAS) methods have been introduced, which automate the procedure of constructing the structures. In this dissertation, I propose a new single image super-resolution framework by using neural architecture search (NAS) method. As the performance improves, the network becomes more complex and deeper, so I apply NAS algorithm to find the optimal network while reducing the effort in network design. In detail, the proposed scheme is summarized to three topics: image super-resolution using efficient neural architecture search, multi-branch neural architecture search for lightweight image super-resolution, and neural architecture search for image super-resolution using meta-transfer learning. At first, I expand the NAS to the super-resolution domain and find a lightweight densely connected network named DeCoNASNet. I use a hierarchical search strategy to find the best connection with local and global features. In this process, I define a complexity-based-penalty and add it to the reward term of REINFORCE algorithm. Experiments show that my DeCoNASNet outperforms the state-of-the-art lightweight super-resolution networks designed by handcraft methods and existing NAS-based design. I propose a new search space design with multi-branch structure to enlarge the search space for capturing multi-scale features, resulting in better reconstruction on grainy areas. I also adopt parameter sharing scheme in multi-branch network to share their information and reduce the whole network parameter. Experiments show that the proposed method finds an optimal SISR network about twenty times faster than the existing methods, while showing comparable performance in terms of PSNR vs. parameters. Comparison of visual quality validates that the proposed SISR network reconstructs texture areas better than the previous methods because of the enlarged search space to find multi-scale features. Lastly, I apply meta-transfer learning to the NAS procedure for image super-resolution. I train the controller and child network with the meta-learning scheme, which enables the controllers to find promising network for several scale simultaneously. Furthermore, meta-trained child network is reused as the pre-trained parameters for final evaluation phase to improve the final image super-resolution results even better and search-evaluation gap problem is efficiently reduced.이미지 복원은 다양한 영상처리 문제의 성능을 높이기 위한 전 처리 단계로 사용할 수 있는 중요한 기술이다. 이미지 고해상도화는 이미지 복원방법 중 중요한 문제의 하나로써 저해상도의 이미지를 고해상도의 이미지로 복원하는 방법이다. 최근에는 컨벌루션 신경망 (CNN)을 사용하는 딥 러닝(deep learning) 기반의 방법들이 단일 이미지 고해상도화 (SISR) 문제를 푸는데 많이 사용되고 있다. 일반적으로 이미지 고해상도화 성능은 CNN을 깊게 쌓거나 복잡한 구조를 설계함으로써 향상시킬 수 있다. 그러나 주어진 문제에 대한 최적의 구조를 찾는 것은 해당 분야의 전문가라 할 지라도 어렵고 시간이 오래 걸리는 작업이다. 이러한 이유로 신경망 구축 절차를 자동화하는 신경망 구조 검색 (NAS) 방법이 도입되었다. 이 논문에서는 신경망 구조 검색 (NAS) 방법을 사용하여 새로운 단일 이미지 고해상도화 방법을 제안하였다. 이 논문에서 제안한 방법은 크게 세 가지로 요약 할 수 있다. 이는 효율적인 신경망 검색기법(ENAS)을 이용한 이미지 고해상도화, 병렬 신경망 검색 기법을 이용한 이미지 고해상도화, 메타 전송 학습을 이용하는 신경망 검색기법을 통한 이미지 고해상도화 이다. 우선, 우리는 주로 영상 분류에 쓰이던 신경망 검색 기법을 영상 고해상도화에 적용하였으며, DeCoNASNet이라 명명된 신경망 구조를 설계하였다. 또한 계층적 검색 전략을 사용하여 지역/전역 피쳐(feature) 합병을 위한 최상의 연결 방법을 검색하였다. 이 과정에서 필요 변수가 적으면서 좋은 성능을 낼 수 있도록 복잡성 기반 페널티 (complexity-based penalty) 를 정의하고 이를 REINFORCE 알고리즘의 보상 신호에 추가하였다. 실험 결과 DeCoNASNet은 기존의 사람이 직접 설계한 신경망과 신경망 검색 기법을 기반으로 설계된 최근의 고해상도화 구조의 성능을 능가하는 것을 확인 할 수 있었다. 우리는 또한 여러 크기의 피쳐(feature)를 학습하기 위해 신경망 검색 기법의 검색 공간을 확대하여 병렬 신경망을 설계하는 방법을 제안하였다. 이 때, 병렬신경망의 각 위치에서 매개 변수를 공유할 수 있도록 하여 병렬신경망의 각 구조끼리 정보를 공유하고 전체 구조를 설계하는데 필요한 매개 변수를 줄이도록 하였다. 실험 결과 제안된 방법을 통해 매개 변수 크기 대비 성능이 좋은 신경망 구조를 찾을 수 있었다. 실험 결과를 통해 확장된 검색 공간에서 여러 크기의 피쳐 (feature)를 학습하였기 때문에 이전 방법보다 복잡한 영역을 더 잘 복원하는 것을 확인하였다. 마지막으로 메타 전송 학습(meta-transfer learning)을 신경망 검색에 적용하여 다양한 크기의 이미지 고해상도화 문제를 해결하는 방법을 제안하였다. 이 논문에서는 메타 전송 학습 방법을 통해 제어기가 여러 크기의 좋은 신경망 구조를 동시에 찾을 수 있도록 설계하였다. 또한 메타 훈련된 신경망 구조는 최종 성능 평가 시 학습의 시작점으로 재사용 되어 최종 이미지 고해상도화 성능을 더욱 향상시킬 수 있었으며, 효과적으로 검색-평가 괴리 문제를 해결하였다.1 INTRODUCTION 1 1.1 contribution 3 1.2 contents 4 2 Neural Architecture Search for Image Super-Resolution Using Densely Constructed Search Space: DeCoNAS 5 2.1 Introduction 5 2.2 Proposed Method 9 2.2.1 Overall structure of DeCoNASNet 9 2.2.2 Constructing the DNB 11 2.2.3 Constructing controller for the DeCoNASNet 13 2.2.4 Training DeCoNAS and complexity-based penalty 13 2.3 Experimental results 15 2.3.1 Settings 15 2.3.2 Results 16 2.3.3 Ablation study 21 2.4 Summary 22 3 Multi-Branch Neural Architecture Search for Lightweight Image Super-resolution 23 3.1 Introduction 23 3.2 Related Work 26 3.2.1 Single image super-resolution 26 3.2.2 Neural architecture search 27 3.2.3 Image super-resolution with neural architecture search 29 3.3 Method 32 3.3.1 Overview of the Proposed MBNAS 32 3.3.2 Controller and complexity-based penalty 33 3.3.3 MBNASNet 35 3.3.4 Multi-scale block with partially shared Nodes 37 3.3.5 MBNAS 38 3.4 datasets and experiments 39 3.4.1 Settings 39 3.4.2 Experiments on single image super-resolution (SISR) 41 3.5 Discussion 48 3.5.1 Effect of the complexity-based penalty to the performance of controller 49 3.5.2 Effect of multi-branch structure and partial parameter sharing scheme 50 3.5.3 Effect of gradient flow control weights and complexity-based penalty coefficient 51 3.6 Summary 52 4 Meta-transfer learning for simultaneous search of various scale image super-resolution 54 4.1 Introduction 54 4.2 Related Work 56 4.2.1 Single image super-resolution 56 4.2.2 Neural architecture search 57 4.2.3 Image super-resolution with neural architecture search 58 4.2.4 Meta-learning 59 4.3 Method 59 4.3.1 Meta-learning 60 4.3.2 Meta-transfer learning 62 4.3.3 Transfer-learning 63 4.4 datasets and experiments 63 4.4.1 Settings 63 4.4.2 Experiments on single image super-resolution(SISR) 64 4.5 Summary 66 5 Conclusion 69 Abstract (In Korean) 80박

WESPE: Weakly Supervised Photo Enhancer for Digital Cameras

Low-end and compact mobile cameras demonstrate limited photo quality mainly due to space, hardware and budget constraints. In this work, we propose a deep learning solution that translates photos taken by cameras with limited capabilities into DSLR-quality photos automatically. We tackle this problem by introducing a weakly supervised photo enhancer (WESPE) - a novel image-to-image Generative Adversarial Network-based architecture. The proposed model is trained by under weak supervision: unlike previous works, there is no need for strong supervision in the form of a large annotated dataset of aligned original/enhanced photo pairs. The sole requirement is two distinct datasets: one from the source camera, and one composed of arbitrary high-quality images that can be generally crawled from the Internet - the visual content they exhibit may be unrelated. Hence, our solution is repeatable for any camera: collecting the data and training can be achieved in a couple of hours. In this work, we emphasize on extensive evaluation of obtained results. Besides standard objective metrics and subjective user study, we train a virtual rater in the form of a separate CNN that mimics human raters on Flickr data and use this network to get reference scores for both original and enhanced photos. Our experiments on the DPED, KITTI and Cityscapes datasets as well as pictures from several generations of smartphones demonstrate that WESPE produces comparable or improved qualitative results with state-of-the-art strongly supervised methods

Contextual-based Image Inpainting: Infer, Match, and Translate

We study the task of image inpainting, which is to fill in the missing region of an incomplete image with plausible contents. To this end, we propose a learning-based approach to generate visually coherent completion given a high-resolution image with missing components. In order to overcome the difficulty to directly learn the distribution of high-dimensional image data, we divide the task into inference and translation as two separate steps and model each step with a deep neural network. We also use simple heuristics to guide the propagation of local textures from the boundary to the hole. We show that, by using such techniques, inpainting reduces to the problem of learning two image-feature translation functions in much smaller space and hence easier to train. We evaluate our method on several public datasets and show that we generate results of better visual quality than previous state-of-the-art methods.Comment: ECCV 2018 camera read