STEFANN: Scene Text Editor using Font Adaptive Neural Network

Textual information in a captured scene plays an important role in scene interpretation and decision making. Though there exist methods that can successfully detect and interpret complex text regions present in a scene, to the best of our knowledge, there is no significant prior work that aims to modify the textual information in an image. The ability to edit text directly on images has several advantages including error correction, text restoration and image reusability. In this paper, we propose a method to modify text in an image at character-level. We approach the problem in two stages. At first, the unobserved character (target) is generated from an observed character (source) being modified. We propose two different neural network architectures - (a) FANnet to achieve structural consistency with source font and (b) Colornet to preserve source color. Next, we replace the source character with the generated character maintaining both geometric and visual consistency with neighboring characters. Our method works as a unified platform for modifying text in images. We present the effectiveness of our method on COCO-Text and ICDAR datasets both qualitatively and quantitatively.Comment: Accepted in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 202

Forensic research on detecting seam carving in digital images

Digital images have been playing an important role in our daily life for the last several decades. Naturally, image editing technologies have been tremendously developed due to the increasing demands. As a result, digital images can be easily manipulated on a personal computer or even a cellphone for many purposes nowadays, so that the authenticity of digital images becomes an important issue. In this dissertation research, four machine learning based forensic methods are presented to detect one of the popular image editing techniques, called ‘seam carving’. To reveal seam carving applied to uncompressed images from the perspective of energy distribution change, an energy based statistical model is proposed as the first work in this dissertation. Features measured global energy of images, remaining optimal seams, and noise level are extracted from four local derivative pattern (LDP) domains instead of from the original pixel domain to heighten the energy change caused by seam carving. A support vector machine (SVM) based classifier is employed to determine whether an image has been seam carved or not. In the second work, an advanced feature model is presented for seam carving detection by investigating the statistical variation among neighboring pixels. Comprised with three types of statistical features, i.e., LDP features, Markov features, and SPAM features, the powerful feature model significantly improved the state-of-the-art accuracy in detecting low carving rate seam carving. After the feature selection by utilizing SVM based recursive feature elimination (SVM-RFE), with a small amount of features selected from the proposed model the overall performance is further improved. Combining above mentioned two works, a hybrid feature model is then proposed as the third work to further boost the accuracy in detecting seam carving at low carving rate. The proposed model consists of two sets of features, which capture energy change and neighboring relationship variation respectively, achieves remarkable performance on revealing seam carving, especially low carving rate seam carving, in digital images. Besides these three hand crafted feature models, a deep convolutional neural network is designed for seam carving detection. It is the first work that successfully utilizes deep learning technology to solve this forensic problem. The experimental works demonstrate their much more improved performance in the cases where the amount of seam carving is not serious. Although these four pieces of work move the seam carving detection ahead substantially, future research works with more advanced statistical model or deep neural network along this line are expected

Symmetry Shape Prior for Object Segmentation

Symmetry is a useful segmentation cue. We develop an algorithm for segmenting a single symmetric object from the background. Our algorithm is formulated in the principled global optimization framework. Thus we can incorporate all the useful segmentation cues in the global energy function, in addition to the symmetry shape prior. We use the standard cues of regular boundary and coherent object (background) appearance. Our algorithm consists of two stages. The first stage, based on seam carving, detects a set of symmetry axis candidates. Symmetry axis is detected by first finding image “seams” that are aligned with intensity gradients and then matching them based on pairwise symmetry. The second stage formulates symmetric object segmentation in discrete optimization framework. We choose the longest symmetry axis as the object axis. Object symmetry is encouraged through submodular long-range pairwise terms. These pairwise terms are submodular, so optimization with a graph cut is applicable. We demonstrate the effectiveness of symmetry cue on a new symmetric object dataset

Texture and Colour in Image Analysis

Research in colour and texture has experienced major changes in the last few years. This book presents some recent advances in the field, specifically in the theory and applications of colour texture analysis. This volume also features benchmarks, comparative evaluations and reviews

A study of holistic strategies for the recognition of characters in natural scene images

Recognition and understanding of text in scene images is an important and challenging task. The importance can be seen in the context of tasks such as assisted navigation for the blind, providing directions to driverless cars, e.g. Google car, etc. Other applications include automated document archival services, mining text from images, and so on. The challenge comes from a variety of factors, like variable typefaces, uncontrolled imaging conditions, and various sources of noise corrupting the captured images. In this work, we study and address the fundamental problem of recognition of characters extracted from natural scene images, and contribute three holistic strategies to deal with this challenging task. Scene text recognition (STR) has been a known problem in computer vision and pattern recognition community for over two decades, and is still an active area of research owing to the fact that the recognition performance has still got a lot of room for improvement. Recognition of characters lies at the heart of STR and is a crucial component for a reliable STR system. Most of the current methods heavily rely on discriminative power of local features, such as histograms of oriented gradient (HoG), scale invariant feature transform (SIFT), shape contexts (SC), geometric blur (GB), etc. One of the problems with such methods is that the local features are rasterized in an ad hoc manner to get a single vector for subsequent use in recognition. This rearrangement of features clearly perturbs the spatial correlations that may carry crucial information vis-á-vis recognition. Moreover, such approaches, in general, do not take into account the rotational invariance property that often leads to failed recognition in cases where characters in scene images do not occur in upright position. To eliminate this local feature dependency and the associated problems, we propose the following three holistic solutions: The first one is based on modelling character images of a class as a 3-mode tensor and then factoring it into a set of rank-1 matrices and the associated mixing coefficients. Each set of rank-1 matrices spans the solution subspace of a specific image class and enables us to capture the required holistic signature for each character class along with the mixing coefficients associated with each character image. During recognition, we project each test image onto the candidate subspaces to derive its mixing coefficients, which are eventually used for final classification. The second approach we study in this work lets us form a novel holistic feature for character recognition based on active contour model, also known as snakes. Our feature vector is based on two variables, direction and distance, cumulatively traversed by each point as the initial circular contour evolves under the force field induced by the character image. The initial contour design in conjunction with cross-correlation based similarity metric enables us to account for rotational variance in the character image. Our third approach is based on modelling a 3-mode tensor via rotation of a single image. This is different from our tensor based approach described above in that we form the tensor using a single image instead of collecting a specific number of samples of a particular class. In this case, to generate a 3D image cube, we rotate an image through a predefined range of angles. This enables us to explicitly capture rotational variance and leads to better performance than various local approaches. Finally, as an application, we use our holistic model to recognize word images extracted from natural scenes. Here we first use our novel word segmentation method based on image seam analysis to split a scene word into individual character images. We then apply our holistic model to recognize individual letters and use a spell-checker module to get the final word prediction. Throughout our work, we employ popular scene text datasets, like Chars74K-Font, Chars74K-Image, SVT, and ICDAR03, which include synthetic and natural image sets, to test the performance of our strategies. We compare results of our recognition models with several baseline methods and show comparable or better performance than several local feature-based methods justifying thus the importance of holistic strategies

Calipso: Physics-based Image and Video Editing through CAD Model Proxies

We present Calipso, an interactive method for editing images and videos in a physically-coherent manner. Our main idea is to realize physics-based manipulations by running a full physics simulation on proxy geometries given by non-rigidly aligned CAD models. Running these simulations allows us to apply new, unseen forces to move or deform selected objects, change physical parameters such as mass or elasticity, or even add entire new objects that interact with the rest of the underlying scene. In Calipso, the user makes edits directly in 3D; these edits are processed by the simulation and then transfered to the target 2D content using shape-to-image correspondences in a photo-realistic rendering process. To align the CAD models, we introduce an efficient CAD-to-image alignment procedure that jointly minimizes for rigid and non-rigid alignment while preserving the high-level structure of the input shape. Moreover, the user can choose to exploit image flow to estimate scene motion, producing coherent physical behavior with ambient dynamics. We demonstrate Calipso's physics-based editing on a wide range of examples producing myriad physical behavior while preserving geometric and visual consistency.Comment: 11 page

Implementasi Deteksi Seam Carving Berdasarkan Perubahan Ukuran Citra Menggunakan Local Binary Patterns dan Support Vector Machine

Seam carving adalah metode yang digunakan untuk content-aware image resizing. Seam carving bertujuan untuk mengubah ukuran citra atau image resizing dengan tidak menghilangkan konten penting yang ada pada citra. Dalam bidang forensik digital, seam carving banyak dibahas khususnya tentang deteksi seam carving pada citra. Hal tersebut bertujuan untuk mengetahui apakah suatu citra sudah pernah melalui proses pengubahan ukuran menggunakan seam carving atau belum.Tugas akhir ini mengusulkan sebuah metode deteksi seam carving berdasarkan perubahan ukuran citra menggunakan Local Binary Patterns dan Support Vector Machine. Citra yang akan dideteksi dihitung variasi teksturnya menggunakan Local Binary Patterns. Proses selanjutnya adalah ekstraksi fitur dari distribusi energy yang menghasilkan 24 fitur. Data fitur citra selanjutnya dilakukan proses normalisasi. Uji coba fitur menggunakan k-fold cross validation dengan membagi data menjadi training dan testing. Selanjutnya data tersebut akan memasuki proses klasifikasi menggunakan Support Vector Machine dengan kernel Radial Basis Function.Uji coba dilakukan terhadap citra asli dan citra seam carving. Citra seam carving yang digunakan dibedakanviiiberdasarkan skala rasionya yaitu 10%, 20%, 30%, 40%, dan 50%. Jumlah data yang digunakan adalah sebanyak 400 citra untuk setiap uji coba pada tiap skala rasio dengan menggunakan 10-fold cross validation. Rata-rata akurasi terbaik yang dihasilkan sebesar 73,95%