Inverse Procedural Modeling of Buildings

3D city modeling is a thriving area of research, as high quality models of real-world cities are in ever-rising demand. These models are used not only among architects and urban planners, but also find their application in navigation, virtual tourism, and entertainment industry. Manual modeling of individual buildings usually provides good results, but the process is very time consuming and expensive. Current automatically-built models using Structure from Motion, followed by simple plane fitting and texturing are a good starting point, but provide inadequate 3D visual perception. No matter the capturing technology, the resulting models are deficient, particularly when dealing with thin objects, fragmented volumes and reflective surfaces. Furthermore, conventional bottom-up models lack any semantic knowledge about the scene. Yet, adding a good understanding of what it is that needs to be modeled is a strong cue, not only to improve the visual and 3D quality of the model, but also to substantially widen its usage. Conversely, procedural modeling provides an effective way to create detailed and realistic 3D building models that do come with all the semantic labels required. This elegant yet powerful framework representsmodels such as buildings through instantiations of a series of parameterized rules, forming a grammar. The resulting procedural models are compact, rich in terms of semantics, and allow for more aesthetic rendering than would be possible from pure 3D capturing. Thus far, procedural modeling has largely been used for synthesizing virtual buildings. In this thesis, we investigate how procedural models can be used in the context of urban reconstruction. Our ultimate goal isto automatically create procedural models of structures as-built, a process referred to as inverse procedural modeling. The main challenge in this process is to determine the appropriate rules of the procedural grammar and their parameters, which typically results in a large search space. In the first part of the thesis we assume the grammar rules are already known, while parameters are allowed to vary. We develop a system for 3D building reconstruction where the grammar leads the modeling process andreceives structural information from object detectors. A drawback of this approach is that the grammar needs to be selected beforehand. Therefore, we develop an algorithm for automatic selection of the appropriate style grammar based on the visual recognition of the architectural style of the observed building. Presently, the main drawback of procedural grammars is that expert architectural knowledge is needed for their creation, which is a non-trivial manual process. Moreover, the abundance of different architectural styles in the world would require many such grammars to bedesigned. To tackle this problem, in the second part of the thesis we simplify the prior knowledge necessary for building reconstruction to a set of general and style-independent architectural principles. We use these weaker priors in a bottom-up approach, by producing high-quality semantic labeling of perspective images and Structure-from-Motion point clouds. This labeling is afterwards transformed into building-specific procedural models, allowing realistic rendering. In the third part of the thesis, we address the problem of procedural grammar scarcity by proposing to learn the grammars from data. First, weshow that probabilistic grammars can be learned from annotated facade imagery. The inferred grammars are shown to be comparable to expert-written grammars on the task of facade reconstruction. Second, weeliminate the need for manual image annotation by replacing it with the previously proposed automatic facade labeling approach. Finally, the learned representations are shown to be useful for virtual facade synthesis, facade comparison and retrieval. The proposed models have been evaluated on several datasets of urban scenes, advancing the state of the art in terms of accuracy and speed. More importantly, it is the conclusion of this thesis that the problem of inverse procedural modeling of buildings could be solved with grammarlearning from labeled and noisy data, obviating the need for a human in the loop, and opening up novel directions for future research.Martinović A., ''Inverse procedural modeling of buildings'', Proefschrift voorgedragen tot het behalen van het doctoraat in de ingenieurswetenschappen, KU Leuven, September 2015, Leuven, Belgium.status: publishe

Bayesian Grammar Learning for Inverse Procedural Modeling

In the fields of urban reconstruction and city modeling, shape grammars have emerged as a powerful tool for both synthesizing novel designs and reconstructing existing buildings. So far, a human expert was required to write grammars for specific building styles, which limited the large-scale applicability of these methods. We present a way to automatically learn two-dimensional stochastic context-free grammars (2D-SCFGs) from a set of labeled building facades. To this end, we use Bayesian Model Merging, a technique originally developed in the field of natural language processing, which we extend to the domain of two-dimensional languages. Given a set of labeled positive examples, we induce a grammar which can be sampled to create novel instances of the same building style. Additionally, we demonstrate that our learned grammar can be used for parsing existing facade imagery. Experiments conducted on the dataset of Haussmannian buildings in Paris show that our parsing with learned grammars outperforms bottom-up classifiers and is on par with approaches that use a manually designed style grammar.Martinovic A., Van Gool L., ''Bayesian grammar learning for inverse procedural modeling'', 26th IEEE computer society conference on computer vision and pattern recognition - CVPR 2013, pp. 201-208, June 23-28, 2013, Portland, Oregon, USA.status: publishe

Hierarchical Co-Segmentation of Building Facades

In this work, we introduce a new system for automatic discovery of high-level structural representations of building facades. Under the assumption that each facade can be represented as a hierarchy of rectilinear subdivisions, our goal is to find the optimal direction of splitting, along with the number and positions of the split lines at each level of the tree. Unlike previous approaches, where each facade is analysed in isolation, we propose a joint analysis of a set of facade images. Initially, a co-segmentation approach is used to produce consistent decompositions across all facade images. Afterwards, a clustering step identifies semantically similar segments. Each cluster of similar segments is then used as the input for the joint segmentation in the next level of the hierarchy. We show that our approach produces consistent hierarchical segmentations on two different facade datasets. Furthermore, we argue that the discovered hierarchies capture essential structural information, which is demonstrated on the tasks of facade retrieval and virtual facade synthesis.Martinovic A., Van Gool L., ''Hierarchical co-segmentation of building facades'', Proceedings 2nd international conference on 3D Vision - 3DV 2014, pp. 409-416, December 8-11, 2014, Tokyo, Japan.status: publishe

ATLAS: A Three-Layered Approach to Facade Parsing

© 2015, Springer Science+Business Media New York. We propose a novel approach for semantic segmentation of building facades. Our system consists of three distinct layers, representing different levels of abstraction in facade images: segments, objects and architectural elements. In the first layer, the facade is segmented into regions, each of which is assigned a probability distribution over semantic classes. We evaluate different state-of-the-art segmentation and classification strategies to obtain the initial probabilistic semantic labeling. In the second layer, we investigate the performance of different object detectors and show the benefit of using such detectors to improve our initial labeling. The generic approaches of the first two layers are then specialized for the task of facade labeling in the third layer. There, we incorporate additional meta-knowledge in the form of weak architectural principles, which enforces architectural plausibility and consistency on the final reconstruction. Rigorous tests performed on two existing datasets of building facades demonstrate that we outperform the current state of the art, even when using outputs from lower layers of the pipeline. Finally, we demonstrate how the output of the highest layer can be used to create a procedural building reconstruction.Mathias M., Martinovic A., Van Gool L., ''ATLAS: A three-layered approach to facade parsing'', International journal of computer vision, vol. 118, no. 1, pp. 22-48, May 2016.status: publishe

Towards semantic city models

Van Gool L., Martinovic A., Mathias M., ''Towards semantic city models'', Photogrammetric week '13, proceedings of 54th photogrammetric week, pp. 217-232, September 9-13, 2013, Stuttgart, Germany.status: publishe

A Three-Layered Approach to Facade Parsing

We propose a novel three-layered approach for semantic segmentation of building facades. In the first layer, starting from an oversegmentation of a facade, we employ the recently introduced machine learning technique Recursive Neural Networks (RNN) to obtain a probabilistic interpretation of each segment. In the middle layer, initial labeling is augmented with the information coming from specialized facade component detectors. The information is merged using a Markov Random Field defined over the image. In the highest layer, we introduce weak architectural knowledge, which enforces the final reconstruction to be architecturally plausible and consistent. Rigorous tests performed on two existing datasets of building facades demonstrate that we significantly outperform the current-state of the art, even when using outputs from lower layers of the pipeline. In the end, we show how the output of the highest layer can be used to create a procedural reconstruction.Martinovic A., Mathias M., Weissenberg J., Van Gool L., ''A three-layered approach to facade parsing'', Lecture notes in computer science, vol. 7578, pp. 416-429, 2012 (12th European conference on computer vision - ECCV 2012, October 7-13, 2012, Firenze, Italy).status: publishe

Procedural 3D Building Reconstruction Using Shape Grammars and Detectors

We propose a novel grammar-driven approach for reconstruction of buildings and landmarks. Our approach complements Structure-from-Motion and image-based analysis with a 'inverse' procedural modeling strategy. So far, procedural modeling has mostly been used for creation of virtual buildings, while the inverse approaches typically focus on reconstruction of single facades. In our work, we reconstruct complete buildings as procedural models using template shape grammars. In the reconstruction process, we let the grammar interpreter automatically decide on which step to take next. The process can be seen as instantiating the template by determining the correct grammar parameters. As an example, we have chosen the reconstruction of Greek Doric temples. This process significantly differs from single facade segmentation due to the immediate need for 3D reconstruction.Mathias M., Martinovic A., Weissenberg J., Van Gool L., ''Procedural 3D building reconstruction using shape grammars and detectors'', International conference on 3D imaging, modeling, processing, visualization and transmission - 3DIMPVT 2011, pp. 304-311, May 16-19, 2011, Hangzhou, China.status: publishe

3D All TheWay: Semantic Segmentation of Urban Scenes From Start to End in 3D

© 2015 IEEE. We propose a new approach for semantic segmentation of 3D city models. Starting from an SfM reconstruction of a street-side scene, we perform classification and facade splitting purely in 3D, obviating the need for slow image-based semantic segmentation methods. We show that a properly trained pure-3D approach produces high quality labelings, with significant speed benefits (20x faster) allowing us to analyze entire streets in a matter of minutes. Additionally, if speed is not of the essence, the 3D labeling can be combined with the results of a state-of-the-art 2D classifier, further boosting the performance. Further, we propose a novel facade separation based on semantic nuances between facades. Finally, inspired by the use of architectural principles for 2D facade labeling, we propose new 3D-specific principles and an efficient optimization scheme based on an integer quadratic programming formulation.Martinovic A., Knopp J., Riemenschneider H., Van Gool L., ''3D All TheWay: Semantic Segmentation of Urban Scenes From Start to End in 3D'', 28th IEEE conference on computer vision and pattern recognition - CVPR 2015, pp. 4456-4465, June 7-12, 2015, Boston, Massachusetts, USA.status: publishe

Automatic architectural style recognition

Procedural modeling has proven to be a very valuable tool in the field of architecture. In the last few years, research has soared to automatically create procedural models from images. However, current algorithms for this process of inverse procedural modeling rely on the assumption that the building style is known. So far, the determination of the building style has remained a manual task. In this paper, we propose an algorithm which automates this process through classification of architectural styles from facade images. Our classifier first identifies the images containing buildings, then separates individual facades within an image and determines the building style. This information could then be used to initialize the building reconstruction process. We have trained our classifier to distinguish between several distinct architectural styles, namely Flemish Renaissance, Haussmannian and Neoclassical. Finally, we demonstrate our approach on various street-side images.Mathias M., Martinovic A., Weissenberg J., Haegler S., Van Gool L., ''Automatic architectural style recognition'', Proceedings 4th ISPRS international workshop 3D-ARCH 2011 : 3D virtual reconstruction and visualization of complex architectures, March 2-4, 2011, Trento, Italy.status: publishe