context-driven hybrid image inpainting

학위논문 (석사)-- 서울대학교 대학원 : 전기·컴퓨터공학부, 2015. 8. 김태환.Image inpainting, which is the filling-in of missing regions in an image, is one of the most important topics in the area of computer vision and image processing. The existing non-hybrid image inpainting techniques can be broadly classified into two types. One is the texture-based inpainting and the other is the structure-based inpainting. One critical drawback of those techniques is that their inpainting results are not effective for the images with a mixture of texture and structure features in terms of visual quality or processing time. However, the conventional hybrid inpainting algorithms, which aim at inpainting images with texture and structure features, do not effectively deal with the two items: (1) what is the most effective application order of the constituents? and (2) how can we extract a minimal sub-image that may contain best candidates of inpaint- ing source? In this work, we propose a new hybrid inpainting algorithm to address the two tasks fully and effectively. Precisely, our algorithm attempts to solve two key ingredients: (1) (right time) determining the best application order for inpainting textural and structural missing regions and (2) (right place) extracting the sub-image containing best candidates of source patches to be used to fill in a target region. Through experiments with diverse image test cases, it is shown that our algorithm integrating the enhancements has greatly improved the inpainting quality compared to that of the previous non-hybrid inpainting methods while even spending much shorter processing time compared to the conventional hybrid inpainting methods.Abstract i Contents ii List of Tables iv List of Figures v 1 INTRODUCTION 1 2 Exemplar-based Inpainting: Review and Enhancement 7 2.1 Preliminary: A State-of-the-Art Exemplar-based Inpainting . . . . . . 7 2.2 Context-Driven Determination of Window Sizes . . . . . . . . . . . . 10 3 The Proposed Context-Driven Hybrid Inpainting 12 3.1 OverallFlow .............................. 12 3.2 Step1:Pre-processing ......................... 14 3.3 Step2:Exemplar-basedInpainting................... 15 3.4 Step3:Diffusion-basedInpainting ................... 18 4 Experimental Results 5 Conclusion Abstract (In Korean) ................... 29 Acknowlegement ................... 30Maste

Deep speech inpainting of time-frequency masks

Transient loud intrusions, often occurring in noisy environments, can completely overpower speech signal and lead to an inevitable loss of information. While existing algorithms for noise suppression can yield impressive results, their efficacy remains limited for very low signal-to-noise ratios or when parts of the signal are missing. To address these limitations, here we propose an end-to-end framework for speech inpainting, the context-based retrieval of missing or severely distorted parts of time-frequency representation of speech. The framework is based on a convolutional U-Net trained via deep feature losses, obtained using speechVGG, a deep speech feature extractor pre-trained on an auxiliary word classification task. Our evaluation results demonstrate that the proposed framework can recover large portions of missing or distorted time-frequency representation of speech, up to 400 ms and 3.2 kHz in bandwidth. In particular, our approach provided a substantial increase in STOI & PESQ objective metrics of the initially corrupted speech samples. Notably, using deep feature losses to train the framework led to the best results, as compared to conventional approaches.Comment: Accepted to InterSpeech202

Where and Who? Automatic Semantic-Aware Person Composition

Image compositing is a method used to generate realistic yet fake imagery by inserting contents from one image to another. Previous work in compositing has focused on improving appearance compatibility of a user selected foreground segment and a background image (i.e. color and illumination consistency). In this work, we instead develop a fully automated compositing model that additionally learns to select and transform compatible foreground segments from a large collection given only an input image background. To simplify the task, we restrict our problem by focusing on human instance composition, because human segments exhibit strong correlations with their background and because of the availability of large annotated data. We develop a novel branching Convolutional Neural Network (CNN) that jointly predicts candidate person locations given a background image. We then use pre-trained deep feature representations to retrieve person instances from a large segment database. Experimental results show that our model can generate composite images that look visually convincing. We also develop a user interface to demonstrate the potential application of our method.Comment: 10 pages, 9 figure

Can graph-cutting improve microarray gene expression reconstructions?

Microarrays produce high-resolution image data that are, unfortunately, permeated with a great deal of “noise” that must be removed for precision purposes. This paper presents a technique for such a removal process. On completion of this non-trivial task, a new surface (devoid of gene spots) is subtracted from the original to render more precise gene expressions. The graph-cutting technique as implemented has the benefits that only the most appropriate pixels are replaced and these replacements are replicates rather than estimates. This means the influence of outliers and other artifacts are handled more appropriately (than in previous methods) as well as the variability of the final gene expressions being considerably reduced. Experiments are carried out to test the technique against commercial and previously researched reconstruction methods. Final results show that the graph-cutting inspired identification mechanism has a positive significant impact on reconstruction accuracy