Stereoscopic Seam Carving With Temporal Consistency

In this paper, we present a novel technique for seam carving of stereoscopic video. It removes seams of pixels in areas that are most likely not noticed by the viewer. When applying seam carving to stereoscopic video rather than monoscopic still images, new challenges arise. The detected seams must be consistent between the left and the right view, so that no depth information is destroyed. When removing seams in two consecutive frames, temporal consistency between the removed seams must be established to avoid flicker in the resulting video. By making certain assumptions, the available depth information can be harnessed to improve the quality achieved by seam carving. Assuming that closer pixels are more important, the algorithm can focus on removing distant pixels first. Furthermore, we assume that coherent pixels belonging to the same object have similar depth. By avoiding to cut through edges in the depth map, we can thus avoid cutting through object boundaries

Supervised Deep Learning for Content-Aware Image Retargeting with Fourier Convolutions

Image retargeting aims to alter the size of the image with attention to the contents. One of the main obstacles to training deep learning models for image retargeting is the need for a vast labeled dataset. Labeled datasets are unavailable for training deep learning models in the image retargeting tasks. As a result, we present a new supervised approach for training deep learning models. We use the original images as ground truth and create inputs for the model by resizing and cropping the original images. A second challenge is generating different image sizes in inference time. However, regular convolutional neural networks cannot generate images of different sizes than the input image. To address this issue, we introduced a new method for supervised learning. In our approach, a mask is generated to show the desired size and location of the object. Then the mask and the input image are fed to the network. Comparing image retargeting methods and our proposed method demonstrates the model's ability to produce high-quality retargeted images. Afterward, we compute the image quality assessment score for each output image based on different techniques and illustrate the effectiveness of our approach.Comment: 18 pages, 5 figure

Adaptation of Images and Videos for Different Screen Sizes

With the increasing popularity of smartphones and similar mobile devices, the demand for media to consume on the go rises. As most images and videos today are captured with HD or even higher resolutions, there is a need to adapt them in a content-aware fashion before they can be watched comfortably on screens with small sizes and varying aspect ratios. This process is called retargeting. Most distortions during this process are caused by a change of the aspect ratio. Thus, retargeting mainly focuses on adapting the aspect ratio of a video while the rest can be scaled uniformly. The main objective of this dissertation is to contribute to the modern image and video retargeting, especially regarding the potential of the seam carving operator. There are still unsolved problems in this research field that should be addressed in order to improve the quality of the results or speed up the performance of the retargeting process. This dissertation presents novel algorithms that are able to retarget images, videos and stereoscopic videos while dealing with problems like the preservation of straight lines or the reduction of the required memory space and computation time. Additionally, a GPU implementation is used to achieve the retargeting of videos in real-time. Furthermore, an enhancement of face detection is presented which is able to distinguish between faces that are important for the retargeting and faces that are not. Results show that the developed techniques are suitable for the desired scenarios

Intelligent visual media processing: when graphics meets vision

The computer graphics and computer vision communities have been working closely together in recent years, and a variety of algorithms and applications have been developed to analyze and manipulate the visual media around us. There are three major driving forces behind this phenomenon: i) the availability of big data from the Internet has created a demand for dealing with the ever increasing, vast amount of resources; ii) powerful processing tools, such as deep neural networks, provide e�ective ways for learning how to deal with heterogeneous visual data; iii) new data capture devices, such as the Kinect, bridge between algorithms for 2D image understanding and 3D model analysis. These driving forces have emerged only recently, and we believe that the computer graphics and computer vision communities are still in the beginning of their honeymoon phase. In this work we survey recent research on how computer vision techniques bene�t computer graphics techniques and vice versa, and cover research on analysis, manipulation, synthesis, and interaction. We also discuss existing problems and suggest possible further research directions

3次元画像の高画質化・高機能化に向けた解像度変換処理の研究

学位の種別:課程博士University of Tokyo(東京大学

Pseudo-Dolly-In Video Generation Combining 3D Modeling and Image Reconstruction

This paper proposes a pseudo-dolly-in video generation method that reproduces motion parallax by applying image reconstruction processing to multi-view videos. Since dolly-in video is taken by moving a camera forward to reproduce motion parallax, we can present a sense of immersion. However, at a sporting event in a large-scale space, moving a camera is difficult. Our research generates dolly-in video from multi-view images captured by fixed cameras. By applying the Image-Based Modeling technique, dolly-in video can be generated. Unfortunately, the video quality is often damaged by the 3D estimation error. On the other hand, Bullet-Time realizes high-quality video observation. However, moving the virtual-viewpoint from the capturing positions is difficult. To solve these problems, we propose a method to generate a pseudo-dolly-in image by installing 3D estimation and image reconstruction techniques into Bullet-Time and show its effectiveness by applying it to multi-view videos captured at an actual soccer stadium. In the experiment, we compared the proposed method with digital zoom images and with the dolly-in video generated from the Image-Based Modeling and Rendering method.Published in: 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) Date of Conference: 9-13 Oct. 2017 Conference Location: Nantes, Franc

Sayısal görüntülerde piksel yolu çıkarma esaslı boyut değişikliği tespiti

06.03.2018 tarihli ve 30352 sayılı Resmi Gazetede yayımlanan “Yükseköğretim Kanunu İle Bazı Kanun Ve Kanun Hükmünde Kararnamelerde Değişiklik Yapılması Hakkında Kanun” ile 18.06.2018 tarihli “Lisansüstü Tezlerin Elektronik Ortamda Toplanması, Düzenlenmesi ve Erişime Açılmasına İlişkin Yönerge” gereğince tam metin erişime açılmıştır.Piksel yolu çıkarma (seam carving), günümüzde en çok uygulanan içeriğe duyarlı görüntü boyutlandırma yöntemlerinden biridir. Piksel yolu çıkarmanın sebep olduğu bozukluklar çok yüksek oranlarda ölçekleme yapılmadıkça insan gözü tarafından algılanamaz. Bu görsel başarının sebebi görüntüdeki piksellerin önem değerlerine göre değerlendiriliyor olmasıdır. Görüntünün optimal seam'i, görüntü genelinde toplamda en az enerji (önem) değerine sahip piksel yoludur. Tek piksel genişliğindeki önemsiz bu piksel yolları birer azaltılarak her iterasyonda görüntünün genişliği ya da yüksekliği bir azaltılır. Anlamsal olarak önemli olan ön plan nesnelerine mümkün olduğunca dokunulmaz. Görüntünün içeriğinin bu denli korunduğu bir ölçekleme yaklaşımı kötü niyetli olarak da kullanılabileceğinden, bu şekilde ölçeklenmiş görüntülerin tespiti büyük önem arz etmektedir. Piksel yolu çıkarma tabanlı ölçeklemenin tespiti diğer ölçekleme yöntemlerine göre oldukça zordur. çünkü görüntülerin geometrik açıdan ele alınması yetmez, anlamsal bir değerlendirme içeren detaylı bir analiz yapılması gerekmektedir. Bu çalışmada, piksel yolu çıkarılarak boyutları değiştirilmiş görüntülerin tespiti, görüntülerden özellik çıkarılması ve çıkarılan özelliklerle Destek Vektör Makinesi'nin eğitilmesi şeklinde gerçekleştirilmektedir. Çıkarılan özellikler piksel yolu çıkarma algoritmasının uygulanışı ile alakalı özelliklerdir. Ayrıca, yöntemin başarımını artırmak amacıyla, özellik çıkarımı öncesinde görüntülere Yerel İkili Örüntüler dönüşümü uygulanmış ve piksel yolu çıkarmanın sebep olabileceği yerel bozukluklar belirginleştirilmiştir. Tüm bunlara ek olarak, piksel yolu çıkarmanın görüntülerin farklı parçalarındaki etkileri de incelenmiştir. Bu amaçla görüntüler şeritlere ayrılarak her bir şerit seam özellikleri bakımından değerlendirilmiş ve tespit doğrulukları bu şekilde oldukça artırılmıştır. Geliştirilen yöntem ile piksel yolu çıkarma tabanlı ölçekleme %30 ölçeklenmiş görüntülerde %99,9'lara kadar tespit edilebilmiştir. Performans literatürdeki diğer yöntemlere göre ortalamada %20'den fazla artırılmıştır. Tespit performansı özellikle tespit edilmesi daha zor olan %3, %6 gibi küçük ölçekleme oranlarında %26 geliştirilmiştir.Seam carving is one of the mostly applied content-aware image resizing methods today. The deteriorations caused by seam carving are mostly unnoticeable for human eyes unless the scaling ratio is very high. The reason of this visual success comes from evaluating the pixels according to their importance values. Optimal seam of an image is a pixel path which contains the least energy (importance) throughout the image. Image width or height is decreased by one in each iteration by removing those unimportant, one-pixel width pixel paths. The semantically important foreground objects remain untouched as far as possible. Since such a scaling approach which perfectly preserves the image content can be used malevolently, the detection of the images that are scaled in this manner becomes more of an issue. The detection of seam carving is more difficult than the other scaling methods since evaluating the images geometrically is not sufficient, but a detailed analysis investigating the semantical concept is required. In this study, the detection of the images scaled by seam carving is realized by feature extraction and training a Support Vector Machine with those features. The extracted features are related to the seam carving process. In addition, Local Binary Patterns transform is applied to the images before feature extraction to reveal the local artifacts caused by seam carving. Besides, the effect of seam carving in sub parts of the images is investigated. For this purpose, the images are divided into several stripes and each and every stripe is evaluated in terms of seam features. This evaluation has been improved the detection accuracies. Seam carving based resizing has been detected up to 99,9% in 30%scaled images by the developed method. The detection performance has been improved 20% on the average when compared with other methods in the literature. The detection performance is improved 26% in low scaling ratios like 3% and 6% which are harder to detect