Learning Visual Importance for Graphic Designs and Data Visualizations

Knowing where people look and click on visual designs can provide clues about how the designs are perceived, and where the most important or relevant content lies. The most important content of a visual design can be used for effective summarization or to facilitate retrieval from a database. We present automated models that predict the relative importance of different elements in data visualizations and graphic designs. Our models are neural networks trained on human clicks and importance annotations on hundreds of designs. We collected a new dataset of crowdsourced importance, and analyzed the predictions of our models with respect to ground truth importance and human eye movements. We demonstrate how such predictions of importance can be used for automatic design retargeting and thumbnailing. User studies with hundreds of MTurk participants validate that, with limited post-processing, our importance-driven applications are on par with, or outperform, current state-of-the-art methods, including natural image saliency. We also provide a demonstration of how our importance predictions can be built into interactive design tools to offer immediate feedback during the design process

Adaptation of Images and Videos for Different Screen Sizes

With the increasing popularity of smartphones and similar mobile devices, the demand for media to consume on the go rises. As most images and videos today are captured with HD or even higher resolutions, there is a need to adapt them in a content-aware fashion before they can be watched comfortably on screens with small sizes and varying aspect ratios. This process is called retargeting. Most distortions during this process are caused by a change of the aspect ratio. Thus, retargeting mainly focuses on adapting the aspect ratio of a video while the rest can be scaled uniformly. The main objective of this dissertation is to contribute to the modern image and video retargeting, especially regarding the potential of the seam carving operator. There are still unsolved problems in this research field that should be addressed in order to improve the quality of the results or speed up the performance of the retargeting process. This dissertation presents novel algorithms that are able to retarget images, videos and stereoscopic videos while dealing with problems like the preservation of straight lines or the reduction of the required memory space and computation time. Additionally, a GPU implementation is used to achieve the retargeting of videos in real-time. Furthermore, an enhancement of face detection is presented which is able to distinguish between faces that are important for the retargeting and faces that are not. Results show that the developed techniques are suitable for the desired scenarios

FUZZY KERNEL REGRESSION FOR REGISTRATION AND OTHER IMAGE WARPING APPLICATIONS

In this dissertation a new approach for non-rigid medical im- age registration is presented. It relies onto a probabilistic framework based on the novel concept of Fuzzy Kernel Regression. The theoric framework, after a formal introduction is applied to develop several complete registration systems, two of them are interactive and one is fully automatic. They all use the composition of local deforma- tions to achieve the final alignment. Automatic one is based onto the maximization of mutual information to produce local affine aligments which are merged into the global transformation. Mutual Information maximization procedure uses gradient descent method. Due to the huge amount of data associated to medical images, a multi-resolution topology is embodied, reducing processing time. The distance based interpolation scheme injected facilitates the similairity measure op- timization by attenuating the presence of local maxima in the func- tional. System blocks are implemented on GPGPUs allowing efficient parallel computation of large 3d datasets using SIMT execution. Due to the flexibility of Mutual Information, it can be applied to multi- modality image scans (MRI, CT, PET, etc.). Both quantitative and qualitative experiments show promising results and great potential for future extension. Finally the framework flexibility is shown by means of its succesful application to the image retargeting issue, methods and results are presented

Sayısal görüntülerde piksel yolu çıkarma esaslı boyut değişikliği tespiti

06.03.2018 tarihli ve 30352 sayılı Resmi Gazetede yayımlanan “Yükseköğretim Kanunu İle Bazı Kanun Ve Kanun Hükmünde Kararnamelerde Değişiklik Yapılması Hakkında Kanun” ile 18.06.2018 tarihli “Lisansüstü Tezlerin Elektronik Ortamda Toplanması, Düzenlenmesi ve Erişime Açılmasına İlişkin Yönerge” gereğince tam metin erişime açılmıştır.Piksel yolu çıkarma (seam carving), günümüzde en çok uygulanan içeriğe duyarlı görüntü boyutlandırma yöntemlerinden biridir. Piksel yolu çıkarmanın sebep olduğu bozukluklar çok yüksek oranlarda ölçekleme yapılmadıkça insan gözü tarafından algılanamaz. Bu görsel başarının sebebi görüntüdeki piksellerin önem değerlerine göre değerlendiriliyor olmasıdır. Görüntünün optimal seam'i, görüntü genelinde toplamda en az enerji (önem) değerine sahip piksel yoludur. Tek piksel genişliğindeki önemsiz bu piksel yolları birer azaltılarak her iterasyonda görüntünün genişliği ya da yüksekliği bir azaltılır. Anlamsal olarak önemli olan ön plan nesnelerine mümkün olduğunca dokunulmaz. Görüntünün içeriğinin bu denli korunduğu bir ölçekleme yaklaşımı kötü niyetli olarak da kullanılabileceğinden, bu şekilde ölçeklenmiş görüntülerin tespiti büyük önem arz etmektedir. Piksel yolu çıkarma tabanlı ölçeklemenin tespiti diğer ölçekleme yöntemlerine göre oldukça zordur. çünkü görüntülerin geometrik açıdan ele alınması yetmez, anlamsal bir değerlendirme içeren detaylı bir analiz yapılması gerekmektedir. Bu çalışmada, piksel yolu çıkarılarak boyutları değiştirilmiş görüntülerin tespiti, görüntülerden özellik çıkarılması ve çıkarılan özelliklerle Destek Vektör Makinesi'nin eğitilmesi şeklinde gerçekleştirilmektedir. Çıkarılan özellikler piksel yolu çıkarma algoritmasının uygulanışı ile alakalı özelliklerdir. Ayrıca, yöntemin başarımını artırmak amacıyla, özellik çıkarımı öncesinde görüntülere Yerel İkili Örüntüler dönüşümü uygulanmış ve piksel yolu çıkarmanın sebep olabileceği yerel bozukluklar belirginleştirilmiştir. Tüm bunlara ek olarak, piksel yolu çıkarmanın görüntülerin farklı parçalarındaki etkileri de incelenmiştir. Bu amaçla görüntüler şeritlere ayrılarak her bir şerit seam özellikleri bakımından değerlendirilmiş ve tespit doğrulukları bu şekilde oldukça artırılmıştır. Geliştirilen yöntem ile piksel yolu çıkarma tabanlı ölçekleme %30 ölçeklenmiş görüntülerde %99,9'lara kadar tespit edilebilmiştir. Performans literatürdeki diğer yöntemlere göre ortalamada %20'den fazla artırılmıştır. Tespit performansı özellikle tespit edilmesi daha zor olan %3, %6 gibi küçük ölçekleme oranlarında %26 geliştirilmiştir.Seam carving is one of the mostly applied content-aware image resizing methods today. The deteriorations caused by seam carving are mostly unnoticeable for human eyes unless the scaling ratio is very high. The reason of this visual success comes from evaluating the pixels according to their importance values. Optimal seam of an image is a pixel path which contains the least energy (importance) throughout the image. Image width or height is decreased by one in each iteration by removing those unimportant, one-pixel width pixel paths. The semantically important foreground objects remain untouched as far as possible. Since such a scaling approach which perfectly preserves the image content can be used malevolently, the detection of the images that are scaled in this manner becomes more of an issue. The detection of seam carving is more difficult than the other scaling methods since evaluating the images geometrically is not sufficient, but a detailed analysis investigating the semantical concept is required. In this study, the detection of the images scaled by seam carving is realized by feature extraction and training a Support Vector Machine with those features. The extracted features are related to the seam carving process. In addition, Local Binary Patterns transform is applied to the images before feature extraction to reveal the local artifacts caused by seam carving. Besides, the effect of seam carving in sub parts of the images is investigated. For this purpose, the images are divided into several stripes and each and every stripe is evaluated in terms of seam features. This evaluation has been improved the detection accuracies. Seam carving based resizing has been detected up to 99,9% in 30%scaled images by the developed method. The detection performance has been improved 20% on the average when compared with other methods in the literature. The detection performance is improved 26% in low scaling ratios like 3% and 6% which are harder to detect

Hybrid LSTM and Encoder-Decoder Architecture for Detection of Image Forgeries

With advanced image journaling tools, one can easily alter the semantic meaning of an image by exploiting certain manipulation techniques such as copy-clone, object splicing, and removal, which mislead the viewers. In contrast, the identification of these manipulations becomes a very challenging task as manipulated regions are not visually apparent. This paper proposes a high-confidence manipulation localization architecture which utilizes resampling features, Long-Short Term Memory (LSTM) cells, and encoder-decoder network to segment out manipulated regions from non-manipulated ones. Resampling features are used to capture artifacts like JPEG quality loss, upsampling, downsampling, rotation, and shearing. The proposed network exploits larger receptive fields (spatial maps) and frequency domain correlation to analyze the discriminative characteristics between manipulated and non-manipulated regions by incorporating encoder and LSTM network. Finally, decoder network learns the mapping from low-resolution feature maps to pixel-wise predictions for image tamper localization. With predicted mask provided by final layer (softmax) of the proposed architecture, end-to-end training is performed to learn the network parameters through back-propagation using ground-truth masks. Furthermore, a large image splicing dataset is introduced to guide the training process. The proposed method is capable of localizing image manipulations at pixel level with high precision, which is demonstrated through rigorous experimentation on three diverse datasets

Deep Rectangling for Image Stitching: A Learning Baseline

Stitched images provide a wide field-of-view (FoV) but suffer from unpleasant irregular boundaries. To deal with this problem, existing image rectangling methods devote to searching an initial mesh and optimizing a target mesh to form the mesh deformation in two stages. Then rectangular images can be generated by warping stitched images. However, these solutions only work for images with rich linear structures, leading to noticeable distortions for portraits and landscapes with non-linear objects. In this paper, we address these issues by proposing the first deep learning solution to image rectangling. Concretely, we predefine a rigid target mesh and only estimate an initial mesh to form the mesh deformation, contributing to a compact one-stage solution. The initial mesh is predicted using a fully convolutional network with a residual progressive regression strategy. To obtain results with high content fidelity, a comprehensive objective function is proposed to simultaneously encourage the boundary rectangular, mesh shape-preserving, and content perceptually natural. Besides, we build the first image stitching rectangling dataset with a large diversity in irregular boundaries and scenes. Experiments demonstrate our superiority over traditional methods both quantitatively and qualitatively.Comment: Accepted by CVPR2022 (oral); Codes and dataset: https://github.com/nie-lang/DeepRectanglin

Texture and Colour in Image Analysis

Research in colour and texture has experienced major changes in the last few years. This book presents some recent advances in the field, specifically in the theory and applications of colour texture analysis. This volume also features benchmarks, comparative evaluations and reviews