Neural Nearest Neighbors Networks

Non-local methods exploiting the self-similarity of natural signals have been well studied, for example in image analysis and restoration. Existing approaches, however, rely on k-nearest neighbors (KNN) matching in a fixed feature space. The main hurdle in optimizing this feature space w.r.t. application performance is the non-differentiability of the KNN selection rule. To overcome this, we propose a continuous deterministic relaxation of KNN selection that maintains differentiability w.r.t. pairwise distances, but retains the original KNN as the limit of a temperature parameter approaching zero. To exploit our relaxation, we propose the neural nearest neighbors block (N3 block), a novel non-local processing layer that leverages the principle of self-similarity and can be used as building block in modern neural network architectures. We show its effectiveness for the set reasoning task of correspondence classification as well as for image restoration, including image denoising and single image super-resolution, where we outperform strong convolutional neural network (CNN) baselines and recent non-local models that rely on KNN selection in hand-chosen features spaces.Comment: to appear at NIPS*2018, code available at https://github.com/visinf/n3net

Single Image Super-Resolution based on Wiener Filter in Similarity Domain

publishedVersionPeer reviewe

Single Image Super-Resolution Using Convolutional Neural Networks

Enlargement of images is a common need in many applications. Although increasing the pixel count of an image is easy with simple interpolation methods, those fail to increase the amount of details in the image. Single image super-resolution (SISR) aims to solve this ill-posed problem of producing a high resolution (HR) image from a given low resolution (LR) image. A single LR image has always an infinite number of corresponding LR images, but some of those are more probable than others. This probability density can be estimated with machine learning techniques, and the most probable HR image can be constructed based on that estimate. In recent years artificial neural networks have become the most popular machine learning methods. Convolutional neural networks (CNN) are a subtype of them, inspired by the human visual system. They are used extensively in all fields of image processing, including single image super-resolution. In this thesis different CNN based methods for SISR are compared, and their performance is analyzed using both quantitative and qualitative methods. In total four CNN methods were chosen, and they were compared to three other methods. One of the reference methods was based on more traditional machine learning, and the two others were based on self-similarity of the input images. In contrast to machine learning approach, self-similarity based methods utilize only information in the input image and do not require any training on external images. The results show that CNN based methods outperform the alternative approaches in both quantitative metrics and qualitative analysis. The methods perform especially well with images that have clear structures and sharp edges, but highly textured images tend to be problematic. Six of the methods aim to minimize pixel-wise reconstruction error, which leads to overly smooth output on textured areas. One method was instead designed to maximize the perceptual quality of the images, at the cost of increased reconstruction error. It was able to generate very realistic textures in some cases, but had a tendency to hallucinate very implausible textures into flat areas. Also other CNN based methods tended to create erroneous but plausible details, which might be misleading in critical applications like medical imaging. CNN based SISR is more suitable for entertainment and other consumer applications, especially when the perceptually optimized methods are developed further

New model of partial filtering in implementation of algorithms for edge detection and digital image segmetation

Ova disertacija je doprinos digitalnoj analizi i obradi slike. Problematika koja je obrađena u disertaciji pokriva oblasti ocene kvaliteta, detekcije ivica, restauracije, klaster filtriranja, klasifikacije, superrezolucije, dizajna filtera i filtriranja digitalne slike. Za primenu u svim pomenutim oblastima razvijen je, a u disertaciji detaljno opisan novi metod parcijalnog filtriranja digitalne slike ‒ metod mozaika. Takođe, predstavljen je i model detekcije ivica ‒ hibridni metod ‒ koji čini sastavni deo metoda mozaika. Detaljno su analizirani parametri ocene kvaliteta. Na taj način rezultati disertacije predstavljeni su na adekvatan i sa drugim radovima merljiv način. Zbog preciznosti ocene filtriranja razvijen je model za ocenu sličnosti slike po kanalima – CSI. Dobijeni rezultati u disertaciji vrednovani su numerički na osnovu relevantnih parametara za ocenu kvaliteta multimedijalnih signala kao što su: PSNR, MSE, SNR, entropije, LoD, SSIM, MSSIM, DSSIM i CSI. Zasnovan na detaljnoj analizi algoritama detekcije ivica, kao još jedan doprinos disertacije, predložen je hibridni metod detekcije ivica. Upotrebom metoda mozaika izvršena je restauracija digitalne slike različitim klaster filtriranjem. Rezultati su prikazani nad slikama snimljenim niskim stepenom osvetljenja, kao i nad defokusiranim i zamućenim slikama. Adekvatnom analizom i obradom izvršena je klasifikacija segmenata u odnosu na parametar nivoa detalja. Praktična primena urađena je na BI-RADS medicinskim slikama. Superrezolucija digitalne slike izvršena je segmentacijom i klasifikacijom segmenata u okviru metoda mozaika. Analizom statističke vrednosti okoline piksela predložen je model za procenu koncentracije Snow & Rain šuma i dizajnirani su filteri za Snow & Rain i Salt & Papper šum. Modeli opisani u disertaciji testirani su korišćenjem poslednjih verzija softverskih rešenja kao što su Matlab, VCDemo, CVIPTools, Gimp, ImageQualityMeasurement, NeatImagePro i SofAS