Holistic interpretation of visual data based on topology:semantic segmentation of architectural facades

The work presented in this dissertation is a step towards effectively incorporating contextual knowledge in the task of semantic segmentation. To date, the use of context has been confined to the genre of the scene with a few exceptions in the field. Research has been directed towards enhancing appearance descriptors. While this is unarguably important, recent studies show that computer vision has reached a near-human level of performance in relying on these descriptors when objects have stable distinctive surface properties and in proper imaging conditions. When these conditions are not met, humans exploit their knowledge about the intrinsic geometric layout of the scene to make local decisions. Computer vision lags behind when it comes to this asset. For this reason, we aim to bridge the gap by presenting algorithms for semantic segmentation of building facades making use of scene topological aspects. We provide a classification scheme to carry out segmentation and recognition simultaneously.The algorithm is able to solve a single optimization function and yield a semantic interpretation of facades, relying on the modeling power of probabilistic graphs and efficient discrete combinatorial optimization tools. We tackle the same problem of semantic facade segmentation with the neural network approach.We attain accuracy figures that are on-par with the state-of-the-art in a fully automated pipeline.Starting from pixelwise classifications obtained via Convolutional Neural Networks (CNN). These are then structurally validated through a cascade of Restricted Boltzmann Machines (RBM) and Multi-Layer Perceptron (MLP) that regenerates the most likely layout. In the domain of architectural modeling, there is geometric multi-model fitting. We introduce a novel guided sampling algorithm based on Minimum Spanning Trees (MST), which surpasses other propagation techniques in terms of robustness to noise. We make a number of additional contributions such as measure of model deviation which captures variations among fitted models

Quasi-regular facade structure extraction

In this paper we present a novel two-stage framework for extracting what we define as a quasi-regular structure in facade images. A quasi-regular structure is an irregular rectangular grid representing the placements of repetitive structural architecture objects, e.g., windows, in a facade. Such a structure generalizes a perfect lattice structure generated by the 2D symmetry groups, studied by the previous work. First, we propose to formulate the quasi-regular structure detection in an object-oriented Marked Point Process framework by treating the architectural elements as objects. This leads to an initial quasi-regular structure map which serves as an indicator map of potential object locations. Then, we propose a regularization scheme to recover the complete quasi-regular structures from the initial incomplete structure. This stage takes advantage of the intrinsic low rank constraint of the quasi-regular structure representing a regularized facade. By applying such a regularization, the complete quasi-regular facade structure is obtained. We have extensively tested our method on a large variety of facade images, and demonstrated both the effectiveness and the robustness of our two-stage framework. © 2013 Springer-Verlag