Comparative Analysis and Evaluation of Image inpainting Algorithms

Image inpainting refers to the task of filling in the missing or damaged regions of an image in an undetectable manner. There are a large variety of image inpainting algorithms existing in the literature. They can broadly be grouped into two categories such as Partial Differential Equation (PDE) based algorithms and Exemplar based Texture synthesis algorithms. However no recent study has been undertaken for a comparative evaluation of these algorithms. In this paper, we are comparing two different types of image inpainting algorithms. The algorithms analyzed are Marcelo Bertalmio’s PDE based inpainting algorithm and Zhaolin Lu et al’s exemplar based Image inpainting algorithm.Both theoretical analysis and experiments have made to analyze the results of these image inpainting algorithms on the basis of both qualitative and quantitative way. Keywords:Image inpainting, Exemplar based, Texture synthesis, Partial Differential Equation (PDE)

Combined Structure and Texture Image Inpainting Algorithm for Natural Scene Image Completion

Image inpainting or image completion refers to the task of filling in the missing or damaged regions of an image in a visually plausible way. Many works on this subject have been proposed these recent years. We present a hybrid method for completion of images of natural scenery, where the removal of a foreground object creates a hole in the image. The basic idea is to decompose the original image into a structure and a texture image. Reconstruction of each image is performed separately. The missing information in the structure component is reconstructed using a structure inpainting algorithm, while the texture component is repaired by an improved exemplar based texture synthesis technique. Taking advantage of both the structure inpainting methods and texture synthesis techniques, we designed an effective image reconstruction method. A comparison with some existing methods on different natural images shows the merits of our proposed approach in providing high quality inpainted images. Keywords: Image inpainting, Decomposition method, Structure inpainting, Exemplar based, Texture synthesi

Inpainting large missing regions based on Seam Carving

Inpainting techniques are developed to recover missing image information. Existing inpainting approaches are: Partial Differential Equations Based Inpainting (PDE-BI) and Exemplar-Based Inpainting (EBI). PDE-BI methods used to fill in the missing information via information propagation from neighbouring areas. However, it can only reconstruct successfully small missing regions that are surrounded by limited texture. However, EBI methods are used to recover large regions with richly-textured/structured areas around them, moreover, artefacts are likely to occur. This paper proposes a technique to reduce the missing region size based on seam carving approach, which enables EBI and PDE-BI to recover the missing part. In our proposal, seam carving is used to reduce only the size of the missing region, to be subsequently recovered using EBI method. The added extra paths resulting from the added seams is repaired using PDE-BI. This method outperformed the state-of-art EBI methods

Automatic Detection and Correction for Glossy Reflections in Digital Photograph

[[abstract]]The popularization of digital technology has made shooting digital photos and using related applications a part of daily life. However, the use of flash, to compensate for low atmospheric lighting, often leads to overexposure or glossy reflections. This study proposes an auto-detection and inpainting technique to correct overexposed faces in digital photography. This algorithm segments the skin color in the photo as well as uses face detection and capturing to determine candidate bright spots on the face. Based on the statistical analysis of color brightness and filtering, the bright spots are identified. Finally, bright spots are corrected through inpainting technology. From the experimental results, this study demonstrates the high accuracy and efficiency of the method

Video inpainting for non-repetitive motion

Master'sMASTER OF SCIENC

Mathematical Approaches for Image Enhancement Problems

This thesis develops novel techniques that can solve some image enhancement problems using theoretically and technically proven and very useful mathematical tools to image processing such as wavelet transforms, partial differential equations, and variational models. Three subtopics are mainly covered. First, color image denoising framework is introduced to achieve high quality denoising results by considering correlations between color components while existing denoising approaches can be plugged in flexibly. Second, a new and efficient framework for image contrast and color enhancement in the compressed wavelet domain is proposed. The proposed approach is capable of enhancing both global and local contrast and brightness as well as preserving color consistency. The framework does not require inverse transform for image enhancement since linear scale factors are directly applied to both scaling and wavelet coefficients in the compressed domain, which results in high computational efficiency. Also contaminated noise in the image can be efficiently reduced by introducing wavelet shrinkage terms adaptively in different scales. The proposed method is able to enhance a wavelet-coded image computationally efficiently with high image quality and less noise or other artifact. The experimental results show that the proposed method produces encouraging results both visually and numerically compared to some existing approaches. Finally, image inpainting problem is discussed. Literature review, psychological analysis, and challenges on image inpainting problem and related topics are described. An inpainting algorithm using energy minimization and texture mapping is proposed. Mumford-Shah energy minimization model detects and preserves edges in the inpainting domain by detecting both the main structure and the detailed edges. This approach utilizes faster hierarchical level set method and guarantees convergence independent of initial conditions. The estimated segmentation results in the inpainting domain are stored in segmentation map, which is referred by a texture mapping algorithm for filling textured regions. We also propose an inpainting algorithm using wavelet transform that can expect better global structure estimation of the unknown region in addition to shape and texture properties since wavelet transforms have been used for various image analysis problems due to its nice multi-resolution properties and decoupling characteristics

Modelado de sistemas de transmisión y reconstrucción de imágenes basados en mazos de fibra óptica no coherentes

La presente tesis doctoral se enmarca en el ámbito científico de los sistemas de inspección visual . El trabajo se ha centrado en la temática de la supervisión remota de entornos. Este tema resulta de un gran interés en múltiples sectores industriales y de investigación.La finalidad de esta tesis es proponer un nuevo modelo global de calibración y transmisión de imágenes utilizando mazos incoherentes de fibras ópticas. El modelo que se presenta incluye toda la problemática a resolver para calcular la función de transferencia necesaria en la transmisión de imágenes, y también, aspectos generales para diseñar dispositivos de este tipo. Se ha desarrollado toda una metodología de calibración y de formación de imágenes basada en dicho modelo. La misma ha sido validada sobre una instalación experimental que es capaz de caracterizar diferentes mazos incoherentes y, además, evaluar diferentes metodologías de calibración espacial y de formación de imágenes. Para llevar acabo los diferentes experimentos mostrados, se ha desarrollado una aplicación que facilita la evaluación empírica del modelo referenciado ante un amplio abanico de condiciones. Los resultados que se exponen demuestran las ventajas que un sistema de estas características puede aportar a la inspección de entornos remotos con difícil acceso, y/o donde resulta arriesgado el uso de cámaras electrónicas convencionales

Modelado de sistemas de transmisión y reconstrucción de imágenes basados en mazos de fibra óptica no coherentes

La presente tesis doctoral se enmarca en el ámbito científico de los sistemas de inspección visual . El trabajo se ha centrado en la temática de la supervisión remota de entornos. Este tema resulta de un gran interés en múltiples sectores industriales y de investigación.La finalidad de esta tesis es proponer un nuevo modelo global de calibración y transmisión de imágenes utilizando mazos incoherentes de fibras ópticas. El modelo que se presenta incluye toda la problemática a resolver para calcular la función de transferencia necesaria en la transmisión de imágenes, y también, aspectos generales para diseñar dispositivos de este tipo. Se ha desarrollado toda una metodología de calibración y de formación de imágenes basada en dicho modelo. La misma ha sido validada sobre una instalación experimental que es capaz de caracterizar diferentes mazos incoherentes y, además, evaluar diferentes metodologías de calibración espacial y de formación de imágenes. Para llevar acabo los diferentes experimentos mostrados, se ha desarrollado una aplicación que facilita la evaluación empírica del modelo referenciado ante un amplio abanico de condiciones. Los resultados que se exponen demuestran las ventajas que un sistema de estas características puede aportar a la inspección de entornos remotos con difícil acceso, y/o donde resulta arriesgado el uso de cámaras electrónicas convencionales

Novel Video Completion Approaches and Their Applications

Video completion refers to automatically restoring damaged or removed objects in a video sequence, with applications ranging from sophisticated video removal of undesired static or dynamic objects to correction of missing or corrupted video frames in old movies and synthesis of new video frames to add, modify, or generate a new visual story. The video completion problem can be solved using texture synthesis and/or data interpolation to fill-in the holes of the sequence inward. This thesis makes a distinction between still image completion and video completion. The latter requires visually pleasing consistency by taking into account the temporal information. Based on their applied concepts, video completion techniques are categorized as inpainting and texture synthesis. We present a bandlet transform-based technique for each of these categories of video completion techniques. The proposed inpainting-based technique is a 3D volume regularization scheme that takes advantage of bandlet bases for exploiting the anisotropic regularities to reconstruct a damaged video. The proposed exemplar-based approach, on the other hand, performs video completion using a precise patch fusion in the bandlet domain instead of patch replacement. The video completion task is extended to two important applications in video restoration. First, we develop an automatic video text detection and removal that benefits from the proposed inpainting scheme and a novel video text detector. Second, we propose a novel video super-resolution technique that employs the inpainting algorithm spatially in conjunction with an effective structure tensor, generated using bandlet geometry. The experimental results show a good performance of the proposed video inpainting method and demonstrate the effectiveness of bandlets in video completion tasks. The proposed video text detector and the video super resolution scheme also show a high performance in comparison with existing methods

An evaluation of partial differential equations based digital inpainting algorithms

Partial Differential equations (PDEs) have been used to model various phenomena/tasks in different scientific and engineering endeavours. This thesis is devoted to modelling image inpainting by numerical implementations of certain PDEs. The main objectives of image inpainting include reconstructing damaged parts and filling-in regions in which data/colour information are missing. Different automatic and semi-automatic approaches to image inpainting have been developed including PDE-based, texture synthesis-based, exemplar-based, and hybrid approaches. Various challenges remain unresolved in reconstructing large size missing regions and/or missing areas with highly textured surroundings. Our main aim is to address such challenges by developing new advanced schemes with particular focus on using PDEs of different orders to preserve continuity of textural and geometric information in the surrounding of missing regions. We first investigated the problem of partial colour restoration in an image region whose greyscale channel is intact. A PDE-based solution is known that is modelled as minimising total variation of gradients in the different colour channels. We extend the applicability of this model to partial inpainting in other 3-channels colour spaces (such as RGB where information is missing in any of the two colours), simply by exploiting the known linear/affine relationships between different colouring models in the derivation of a modified PDE solution obtained by using the Euler-Lagrange minimisation of the corresponding gradient Total Variation (TV). We also developed two TV models on the relations between greyscale and colour channels using the Laplacian operator and the directional derivatives of gradients. The corresponding Euler-Lagrange minimisation yields two new PDEs of different orders for partial colourisation. We implemented these solutions in both spatial and frequency domains. We measure the success of these models by evaluating known image quality measures in inpainted regions for sufficiently large datasets and scenarios. The results reveal that our schemes compare well with existing algorithms, but inpainting large regions remains a challenge. Secondly, we investigate the Total Inpainting (TI) problem where all colour channels are missing in an image region. Reviewing and implementing existing PDE-based total inpainting methods reveal that high order PDEs, applied to each colour channel separately, perform well but are influenced by the size of the region and the quantity of texture surrounding it. Here we developed a TI scheme that benefits from our partial inpainting approach and apply two PDE methods to recover the missing regions in the image. First, we extract the (Y, Cb, Cr) of the image outside the missing region, apply the above PDE methods for reconstructing the missing regions in the luminance channel (Y), and then use the colourisation method to recover the missing (Cb, Cr) colours in the region. We shall demonstrate that compared to existing TI algorithms, our proposed method (using 2 PDE methods) performs well when tested on large datasets of natural and face images. Furthermore, this helps understanding of the impact of the texture in the surrounding areas on inpainting and opens new research directions. Thirdly, we investigate existing Exemplar-Based Inpainting (EBI) methods that do not use PDEs but simultaneously propagate the texture and structure into the missing region by finding similar patches within the rest of image and copying them into the boundary of the missing region. The order of patch propagation is determined by a priority function, and the similarity is determined by matching criteria. We shall exploit recently emerging Topological Data Analysis (TDA) tools to create innovative EBI schemes, referred to as TEBI. TDA studies shapes of data/objects to quantify image texture in terms of connectivity and closeness properties of certain data landmarks. Such quantifications help determine the appropriate size of patch propagation and will be used to modify the patch propagation priority function using the geometrical properties of curvature of isophotes, and to improve the matching criteria of patches by calculating the correlation coefficients from the spatial, gradient and Laplacian domains. The performance of this TEBI method will be tested by applying it to natural dataset images, resulting in improved inpainting when compared with other EBI methods. Fourthly, the recent hybrid-based inpainting techniques are reviewed and a number of highly performing innovative hybrid techniques that combine the use of high order PDE methods with the TEBI method for the simultaneous rebuilding of the missing texture and structure regions in an image are proposed. Such a hybrid scheme first decomposes the image into texture and structure components, and then the missing regions in these components are recovered by TEBI and PDE based methods respectively. The performance of our hybrid schemes will be compared with two existing hybrid algorithms. Fifthly, we turn our attention to inpainting large missing regions, and develop an innovative inpainting scheme that uses the concept of seam carving to reduce this problem to that of inpainting a smaller size missing region that can be dealt with efficiently using the inpainting schemes developed above. Seam carving resizes images based on content-awareness of the image for both reduction and expansion without affecting those image regions that have rich information. The missing region of the seam-carved version will be recovered by the TEBI method, original image size is restored by adding the removed seams and the missing parts of the added seams are then repaired using a high order PDE inpainting scheme. The benefits of this approach in dealing with large missing regions are demonstrated. The extensive performance testing of the developed inpainting methods shows that these methods significantly outperform existing inpainting methods for such a challenging task. However, the performance is still not acceptable in recovering large missing regions in high texture and structure images, and hence we shall identify remaining challenges to be investigated in the future. We shall also extend our work by investigating recently developed deep learning based image/video colourisation, with the aim of overcoming its limitations and shortcoming. Finally, we should also describe our on-going research into using TDA to detect recently growing serious “malicious” use of inpainting to create Fake images/videos