Spatially-Adaptive Filter Units for Compact and Efficient Deep Neural Networks

Convolutional neural networks excel in a number of computer vision tasks. One of their most crucial architectural elements is the effective receptive field size, that has to be manually set to accommodate a specific task. Standard solutions involve large kernels, down/up-sampling and dilated convolutions. These require testing a variety of dilation and down/up-sampling factors and result in non-compact representations and excessive number of parameters. We address this issue by proposing a new convolution filter composed of displaced aggregation units (DAU). DAUs learn spatial displacements and adapt the receptive field sizes of individual convolution filters to a given problem, thus eliminating the need for hand-crafted modifications. DAUs provide a seamless substitution of convolutional filters in existing state-of-the-art architectures, which we demonstrate on AlexNet, ResNet50, ResNet101, DeepLab and SRN-DeblurNet. The benefits of this design are demonstrated on a variety of computer vision tasks and datasets, such as image classification (ILSVRC 2012), semantic segmentation (PASCAL VOC 2011, Cityscape) and blind image de-blurring (GOPRO). Results show that DAUs efficiently allocate parameters resulting in up to four times more compact networks at similar or better performance.Comment: Accepted for publication in International Journal of Computer Vision, Jan 02 202

Spatially-adaptive filter units for deep neural networks

Classical deep convolutional networks increase receptive field size by either gradual resolution reduction or application of hand-crafted dilated convolutions to prevent increase in the number of parameters. In this paper we propose a novel displaced aggregation unit (DAU) that does not require hand-crafting. In contrast to classical filters with units (pixels) placed on a fixed regular grid, the displacement of the DAUs are learned, which enables filters to spatially-adapt their receptive field to a given problem. We extensively demonstrate the strength of DAUs on a classification and semantic segmentation tasks. Compared to ConvNets with regular filter, ConvNets with DAUs achieve comparable performance at faster convergence and up to 3-times reduction in parameters. Furthermore, DAUs allow us to study deep networks from novel perspectives. We study spatial distributions of DAU filters and analyze the number of parameters allocated for spatial coverage in a filter.Comment: Accepted to Computer Vision and Pattern Recognition 201

Towards deep compositional networks

Hierarchical feature learning based on convolutional neural networks (CNN) has recently shown significant potential in various computer vision tasks. While allowing high-quality discriminative feature learning, the downside of CNNs is the lack of explicit structure in features, which often leads to overfitting, absence of reconstruction from partial observations and limited generative abilities. Explicit structure is inherent in hierarchical compositional models, however, these lack the ability to optimize a well-defined cost function. We propose a novel analytic model of a basic unit in a layered hierarchical model with both explicit compositional structure and a well-defined discriminative cost function. Our experiments on two datasets show that the proposed compositional model performs on a par with standard CNNs on discriminative tasks, while, due to explicit modeling of the structure in the feature units, affording a straight-forward visualization of parts and faster inference due to separability of the units. ActionsComment: Published in proceedings of 23th International Conference on Pattern Recognition (ICPR 2016

Reprezentacija vizualnih entitet z globokimi hierarhičnimi in kompozicionalnimi modeli

The doctoral thesis explores two prominent hierarchical approaches for the modeling of visual entities: (a) compositional hierarchies and (b) deep neural networks. Both approaches are explored in detail together with their advantages and disadvantages. In compositional hierarchies, poor discriminative power is identified as a major limiting factor, which is address with a novel discriminative feature, termed Histogram of Compositions, proposed in the first part of this thesis. HoC is shown to successfully capture important discriminative information to improve classification accuracy. The second part of the thesis highlights the lack of a spatial relationship between features as an important limitation of deep convolutional networks (ConvNets). This limitation leads to rigid and non-learnable receptive field sizes, poor utilization of parameters and low flexibility of deep architectures. All of those problems are addressed by introducing the explicit compositional structure into deep neural networks, which is implemented with the proposed novel filter unit for ConvNets, termed Displaced Aggregation Unit. DAUs enable novel properties for deep models: (a) the decoupling of the parameters from the receptive field, (b) the learning of the receptive field sizes and (c) the automatic adjustment of the spatial focus of features. The benefits of DAUs are demonstrated on three practical problems: image classification, semantic segmentation and blind image de-blurring. In all cases, the inclusion of DAUs into modern architectures enables simpler networks with fewer number of operations and parameters, significantly reduces the manual modification of architectures for specific tasks and domains while it also retains or even improves the overall prediction accuracy.Doktorska disertacija obravnava dva pomembna hierarhična pristopa za modeliranje vizualnih entitet: (a) kompozicijsko hierarhijo in (b) globoke nevronske mreže. Oba pristopa sta podrobno ovrednotena skupaj z njunimi prednosti in slabosti. V kompozicijski hierarhiji je kot glavna pomanjkljivost naslovljena slaba diskriminativna moč, kar je obravnavano v prvem delu disertacije. Predlagana je nova diskriminativna značilka, imenovana Histogram Kompozicij (ang. Histogram of Compositons - HoC), ki uspešno zajame pomembne diskriminativne informacije za izboljšanje natančnosti klasifikacije. V drugem delu disertacije je v globokih konvolucijskih mrežah (ConvNet) kot pomembna pomanjkljivost izpostavljena slaba prostorska relacija med značilkami. Slednje pripelje do rigidnih in ne-učljivih velikosti dovzetnih polij, do slabe izkoriščenosti parametrov ter do nizke fleksibilnosti globokih arhitektur. Omenjeni problemi so naslovljeni z integracijo eksplicitne kompozicijske strukture v globoke nevronske mreže. V ta namen je predstavljena nova enota filtra za konvolucijske mreže, imenovana premikajoča agregacijska enota (ang. Displaced Aggregation Unit - DAU), ki omogoči vpeljavo novih lastnosti v globoke mreže: (a) neodvisnost števila parametrov od dovzetnega polja, (b) učenje velikosti dovzetnega polja in (c) samodejno prilagajanje prostorskega fokusa značilk. Prednosti filtra DAU so prikazane na treh praktičnih problemih: klasifikacija slik, semantična segmentacija slik ter razmeglejevanje slik. V vseh primerih vključitev filtra DAU v sodobne arhitekture omogoči enostavnejše globoke mreže z manjšim številom operacij in parametrov ter z manjšo potrebo po ročni modifikaciji arhitekture za specifične naloge in domene, hkrati pa ohranja ali celo izboljša klasifikacijsko točnost

Describing Visual Categories by Attributes

This thesis focuses on one of the most challenging problems in the field of computer vision, i.e. general object recognition. In the introduction we first delineate the problem and possible solutions that have been formulated over the past thirty years. We proceed by concentrating on mostly one of the latest approaches introduced by Ali Farhadi et al, who have suggested a system which ascribes certain semantic and discriminative attributes to each object, which then act as a basis for performing quite satisfactory object recognition. We observe that this system has many additional advantages such as faster learning of greater number of categories, reporting unusual object traits as well as even learning and recognizing objects on the basis of word description alone. Given this we believe that the suggested approach indicates a step into the right direction, therefore this thesis offers a more detailed presentation of the attribute learning concept. In addition, we repeated certain experiments with our version of implementation, demonstrating that the produced results are similar to those of Ali Farhadi et al. Doing this we observed that it would be logical to introduce additional improvements as for example transformed implementation of discriminative attributes and the use of separate localization, which on one hand enables better learning of semantic attributes, and on the other hand better category leaning. In practically every case, our improvements turned out to be useful, which we have supported with appropriate experiments. The attribute system also showed to have certain similarities with the LHOP method; therefore we decided to combine both the attribute system and the LHOP method by employing LHOP parts up to the third level in the base feature instead of the edge descriptor and HOG descriptor. This approach illustrated that it works similarly well as the original attribute learning method; however, in view of the fact that there is always room for improvements, we suggest some possible directions for future research