DeepToF: Off-the-shelf real-time correction of multipath interference in time-of-flight imaging

Time-of-flight (ToF) imaging has become a widespread technique for depth estimation, allowing affordable off-the-shelf cameras to provide depth maps in real time. However, multipath interference (MPI) resulting from indirect illumination significantly degrades the captured depth. Most previous works have tried to solve this problem by means of complex hardware modifications or costly computations. In this work, we avoid these approaches and propose a new technique to correct errors in depth caused by MPI, which requires no camera modifications and takes just 10 milliseconds per frame. Our observations about the nature of MPI suggest that most of its information is available in image space; this allows us to formulate the depth imaging process as a spatially-varying convolution and use a convolutional neural network to correct MPI errors. Since the input and output data present similar structure, we base our network on an autoencoder, which we train in two stages. First, we use the encoder (convolution filters) to learn a suitable basis to represent MPI-corrupted depth images; then, we train the decoder (deconvolution filters) to correct depth from synthetic scenes, generated by using a physically-based, time-resolved renderer. This approach allows us to tackle a key problem in ToF, the lack of ground-truth data, by using a large-scale captured training set with MPI-corrupted depth to train the encoder, and a smaller synthetic training set with ground truth depth to train the decoder stage of the network. We demonstrate and validate our method on both synthetic and real complex scenarios, using an off-the-shelf ToF camera, and with only the captured, incorrect depth as input

적대적 생성 신경망을 활용한 실영상 잡음 제거 기법

학위논문 (석사) -- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2021. 2. 이경무.Learning-based image denoising models have been bounded to situations where well-aligned noisy and clean images are given, or training samples can be synthesized from predetermined noise models. While recent generative methods introduce a methodology to accurately simulate the unknown distribution of real-world noise, several limitations still exist. The existing methods are restrained to the case that unrealistic assumptions are made, or the data of actual noise distribution is available. In a real situation, a noise generator should learn to simulate the general and complex noise distribution without using paired noisy and clean images. As a noise generator learned for the real situation tends to fail to express complex noise maps and fits to generate specific texture patterns, we propose an architecture designed to resolve this problem. Therefore, we introduce the C2N, a Clean-to-Noisy image generation framework, to imitate complex real-world noise without using any paired examples. Our C2N combined with a conventional denoising model outperforms existing unsupervised methods on a challenging real-world denoising benchmark by a large margin, validating the effectiveness of the proposed formulation. We also extend our method to a practical situation when there are several data constraints, an area not previously explored by the previous generative noise modeling methods.학습 기반 영상 잡음 제거 모델의 사용은, 잡음이 있는 이미지들과 깨끗한 이미지들이 잘 정렬된 쌍을 이룬 상태로 제공되거나, 주어진 잡음의 분포로부터 학습용 샘플들을 합성할 수 있는 상황에 한정되어 있다. 최근의 생성모델 기반의 방법들은 실제 잡음의 분포가 알려지지 않은 경우에도 그것을 정확하게 시뮬레이션하는 방법론을 도입하고 있지만, 몇 가지 제한점들이 여전히 존재한다. 기존의 그러한 방법들은 실제 잡음의 분포를 얻을 수 있는 데이터가 주어지거나 잡음에 대해 비현실적인 가정이 내려진 경우로 적용 범위가 제한되었다. 실제 상황에서의 잡음 생성모델은 잡음이 있는 이미지와 깨끗한 이미지의 쌍을 사용하지 않고도 복잡하며 일반적인 잡음의 시뮬레이션을 학습할 수 있어야 한다. 이러한 실제적 상황에서 학습한 잡음 생성모델은 복잡한 잡음의 분포가 아닌 특정 질감의 패턴을 만들어내는 동작을 하게 되어버리기 쉽기에, 이 문제를 해결하기 위해 설계한 모델 구조를 제안한다. 이렇게 설계한, C2N 즉 Clean-to-Noisy 영상 생성 프레임워크를 개발하여 복잡한 실영상의 잡음을 어떠한 쌍을 이룬 학습 데이터 없이 모방할 수 있다. 이 C2N을 기존의 잡음 제거 모델과 결합하는 것으로 실영상 잡음 제거 벤치마크에서 기존의 비감독 학습 방법들을 큰 폭으로 능가할 수 있으며, 이를 통해 제안 방법의 효과를 검증한다. 또한 이전의 잡음 생성모델 방법들에 의해선 탐구되지 않았던 영역인, 데이터에 대한 여러 제약이 있는 실용적 상황에 대해 본 방법을 확장한다.Abstract - i Contents - ii List of Tables - iv List of Figures - v 1 INTRODUCTION 1 2 RELATED WORK 5 2.1 Deep Image Denoising 5 2.2 Deep Denoising of Real-World Noise 5 3 C2N: Clean-to-Noisy Image Generation Framework - 8 3.1 Complexity of Real-World Noise 8 3.2 Learning to Generate Pseudo-Noisy Images 9 3.3 C2N Architecture 12 3.3.1 Signal-Independent Pixel-Wise Transforms 12 3.3.2 Signal-Dependent Sampling and Transforms 12 3.3.3 Spatially Correlated Transforms 13 3.3.4 Discriminator 14 3.4 Learning to Denoise with the Generated Pairs 14 4 Experiment 16 4.1 Experimental Setup 16 4.1.1 Dataset 16 4.1.2 Implementation Details and Optimization 17 4.2 Model Analysis 17 4.3 Results on Real-World Noise 23 4.4 Performance Under Practical Data Constraints 26 4.5 Generating noise by interpolation in latent space 30 4.6 Verifying C2N in Denoiser Training 31 5 Conclusion 33 Abstract (In Korean) 40 Acknowlegement 41Maste