FaceShop: Deep Sketch-based Face Image Editing

We present a novel system for sketch-based face image editing, enabling users to edit images intuitively by sketching a few strokes on a region of interest. Our interface features tools to express a desired image manipulation by providing both geometry and color constraints as user-drawn strokes. As an alternative to the direct user input, our proposed system naturally supports a copy-paste mode, which allows users to edit a given image region by using parts of another exemplar image without the need of hand-drawn sketching at all. The proposed interface runs in real-time and facilitates an interactive and iterative workflow to quickly express the intended edits. Our system is based on a novel sketch domain and a convolutional neural network trained end-to-end to automatically learn to render image regions corresponding to the input strokes. To achieve high quality and semantically consistent results we train our neural network on two simultaneous tasks, namely image completion and image translation. To the best of our knowledge, we are the first to combine these two tasks in a unified framework for interactive image editing. Our results show that the proposed sketch domain, network architecture, and training procedure generalize well to real user input and enable high quality synthesis results without additional post-processing.Comment: 13 pages, 20 figure

Visualization and analysis of diffusion tensor fields

technical reportThe power of medical imaging modalities to measure and characterize biological tissue is amplified by visualization and analysis methods that help researchers to see and understand the structures within their data. Diffusion tensor magnetic resonance imaging can measure microstructural properties of biological tissue, such as the coherent linear organization of white matter of the central nervous system, or the fibrous texture of muscle tissue. This dissertation describes new methods for visualizing and analyzing the salient structure of diffusion tensor datasets. Glyphs from superquadric surfaces and textures from reactiondiffusion systems facilitate inspection of data properties and trends. Fiber tractography based on vector-tensor multiplication allows major white matter pathways to be visualized. The generalization of direct volume rendering to tensor data allows large-scale structures to be shaded and rendered. Finally, a mathematical framework for analyzing the derivatives of tensor values, in terms of shape and orientation change, enables analytical shading in volume renderings, and a method of feature detection important for feature-preserving filtering of tensor fields. Together, the combination of methods enhances the ability of diffusion tensor imaging to provide insight into the local and global structure of biological tissue

Example based texture synthesis and quantification of texture quality

Textures have been used effectively to create realistic environments for virtual worlds by reproducing the surface appearances. One of the widely-used methods for creating textures is the example based texture synthesis method. In this method of generating a texture of arbitrary size, an input image from the real world is provided. This input image is used for the basis of generating large textures. Various methods based on the underlying pattern of the image have been used to create these textures; however, the problem of finding an algorithm which provides a good output is still an open research issue. Moreover, the process of determining the best of the outputs produced by the existing methods is a subjective one and requires human intervention. No quantification measure exists to do a relative comparison between the outputs. This dissertation addresses both problems using a novel approach. The dissertation also proposes an improved algorithm for image inpainting which yields better results than existing methods. Firstly, this dissertation presents a methodology which uses a HSI (hue, saturation, intensity) color model in conjunction with the hybrid approach to improve the quality of the synthesized texture. Unlike the RGB (red, green, blue) color model, the HSI color model is more intuitive and closer to human perception. The hue, saturation and intensity are better indicators than the three color channels used in the RGB model. They represent the exact way, in which the eye sees color in the real world. Secondly, this dissertation addresses the issue of quantifying the quality of the output textures generated using the various texture synthesis methods. Quantifying the quality of the output generated is an important issue and a novel method using statistical measures and a color autocorrelogram has been proposed. It is a two step method; in the first step a measure of the energy, entropy and similar statistical measures helps determine the consistency of the output texture. In the second step an autocorelogram is used to analyze color images as well and quantify them effectively. Finally, this disseratation prsesents a method for improving image inpainting. In the case of inpainting, small sections of the image missing due to noise or other similar reasons can be reproduced using example based texture synthesis. The region of the image immediately surrounding the missing section is treated as sample input. Inpainting can also be used to alter images by removing large sections of the image and filling the removed section with the image data from the rest of the image. For this, a maximum edge detector method is proposed to determine the correct order of section filling and produces significantly better results

Quad Meshing

Triangle meshes have been nearly ubiquitous in computer graphics, and a large body of data structures and geometry processing algorithms based on them has been developed in the literature. At the same time, quadrilateral meshes, especially semi-regular ones, have advantages for many applications, and significant progress was made in quadrilateral mesh generation and processing during the last several years. In this State of the Art Report, we discuss the advantages and problems of techniques operating on quadrilateral meshes, including surface analysis and mesh quality, simplification, adaptive refinement, alignment with features, parametrization, and remeshing

Patient-specific anatomical illustration via model-guided texture synthesis

Medical illustrations can make powerful use of textures to attractively, effectively, and efficiently visualize the appearance of the surface or cut surface of anatomic structures. It can do this by implying the anatomic structure's physical composition and clarifying its identity and 3-D shape. Current visualization methods are only capable of conveying detailed information about the orientation, internal structure, and other local properties of the anatomical objects for a typical individual, not for a particular patient. Although one can derive the shape of the individual patient's object from CT or MRI, it is important to apply these illustrative techniques to those particular shapes. In this research patient-specific anatomical illustrations are created by model-guided texture synthesis (MGTS). Given 2D exemplar textures and model-based guidance information as input, MGTS uses exemplar-based texture synthesis techniques to create patient-specific surface and solid textures. It consists of three main components. The first component includes a novel texture metamorphosis approach for creating interpolated exemplar textures given two exemplar textures. This component uses an energy optimization scheme derived from optimal control principles that utilizes intensity and structure information in obtaining the transformation. The second component consists of creating the model-based guidance information, such as directions and layers, for that specific model. This component uses coordinates implied by discrete medial 3D anatomical models (m-reps). The last component accomplishes exemplar-based texture synthesis by textures whose characteristics are spatially variant on and inside the 3D models. It considers the exemplar textures from the first component and guidance information from the second component in synthesizing high-quality, high-resolution solid and surface textures. Patient-specific illustrations with a variety of textures for different anatomical models, such as muscles and bones, are shown to be useful for our clinician to comprehend the shape of the models under radiation dose and to distinguish the models from one another

Example-Based Urban Modeling

The manual modeling of virtual cities or suburban regions is an extremely time-consuming task, which expects expert knowledge of different fields. Existing modeling tool-sets have a steep learning curve and may need special education skills to work with them productively. Existing automatic methods rely on rule sets and grammars to generate urban structures; however, their expressiveness is limited by the rule-sets. Expert skills are necessary to typeset rule sets successfully and, in many cases, new rule-sets need to be defined for every new building style or street network style. To enable non-expert users, the possibility to construct urban structures for individual experiments, this work proposes a portfolio of novel example-based synthesis algorithms and applications for the controlled generation of virtual urban environments. The notion example-based denotes here that new virtual urban environments are created by computer programs that re-use existing digitized real-world data serving as templates. The data, i.e., street networks, topography, layouts of building footprints, or even 3D building models, necessary to realize the envisioned task is already publicly available via online services. To enable the reuse of existing urban datasets, novel algorithms need to be developed by encapsulating expert knowledge and thus allow the controlled generation of virtual urban structures from sparse user input. The focus of this work is the automatic generation of three fundamental structures that are common in urban environments: road networks, city block, and individual buildings. In order to achieve this goal, the thesis proposes a portfolio of algorithms that are briefly summarized next. In a theoretical chapter, we propose a general optimization technique that allows formulating example-based synthesis as a general resource-constrained k-shortest path (RCKSP) problem. From an abstract problem specification and a database of exemplars carrying resource attributes, we construct an intermediate graph and employ a path-search optimization technique. This allows determining either the best or the k-best solutions. The resulting algorithm has a reduced complexity for the single constraint case when compared to other graph search-based techniques. For the generation of road networks, two different techniques are proposed. The first algorithm synthesizes a novel road network from user input, i.e., a desired arterial street skeleton, topography map, and a collection of hierarchical fragments extracted from real-world road networks. The algorithm recursively constructs a novel road network reusing these fragments. Candidate fragments are inserted into the current state of the road network, while shape differences will be compensated by warping. The second algorithm synthesizes road networks using generative adversarial networks (GANs), a recently introduced deep learning technique. A pre- and postprocessing pipeline allows using GANs for the generation of road networks. An in-depth evaluation shows that GANs faithfully learn the road structure present in the example network and that graph measures such as area, aspect ratio, and compactness, are maintained within the virtual road networks. To fill empty city blocks in road networks we propose two novel techniques. The first algorithm re-uses real-world city blocks and synthesizes building footprint layouts into empty city blocks by retrieving viable candidate blocks from a database. We evaluate the algorithm and synthesize a multitude of city block layouts reusing real-world building footprint arrangements from European and US-cities. In addition, we increase the realism of the synthesized layouts by performing example-based placement of 3D building models. This technique is evaluated by placing buildings onto challenging footprint layouts using different example building databases. The second algorithm computes a city block layout, resembling the style of a real-world city block. The original footprint layout is deformed to construct a textit{guidance map}, i.e., the original layout is transferred to a target city block using warping. This guidance map and the original footprints are used by an optimization technique that computes a novel footprint layout along the city block edges. We perform a detailed evaluation and show that using the guidance map allows transferring of the original layout, locally as well as globally, even when the source and target shapes drastically differ. To synthesize individual buildings, we use the general optimization technique described first and formulate the building generation process as a resource-constrained optimization problem. From an input database of annotated building parts, an abstract description of the building shape, and the specification of resource constraints such as length, area, or a number of architectural elements, a novel building is synthesized. We evaluate the technique by synthesizing a multitude of challenging buildings fulfilling several global and local resource constraints. Finally, we show how this technique can even be used to synthesize buildings having the shape of city blocks and might also be used to fill empty city blocks in virtual street networks. All algorithms presented in this work were developed to work with a small amount of user input. In most cases, simple sketches and the definition of constraints are enough to produce plausible results. Manual work is necessary to set up the building part databases and to download example data from mapping services available on the Internet

VOLMAP: a Large Scale Benchmark for Volume Mappings to Simple Base Domains

Correspondences between geometric domains (mappings) are ubiquitous in computer graphics and engineering, both for a variety of downstream applications and as core building blocks for higher level algorithms. In particular, mapping a shape to a convex or star-shaped domain with simple geometry is a fundamental module in existing pipelines for mesh generation, solid texturing, generation of shape correspondences, advanced manufacturing etc. For the case of surfaces, computing such a mapping with guarantees of injectivity is a solved problem. Conversely, robust algorithms for the generation of injective volume mappings to simple polytopes are yet to be found, making this a fundamental open problem in volume mesh processing. VOLMAP is a large scale benchmark aimed to support ongoing research in volume mapping algorithms. The dataset contains 4.7K tetrahedral meshes, whose boundary vertices are mapped to a variety of simple domains, either convex or star-shaped. This data constitutes the input for candidate algorithms, which are then required to position interior vertices in the domain to obtain a volume map. Overall, this yields more than 22K alternative test cases. VOLMAP also comprises tools to process this data, analyze the resulting maps, and extend the dataset with new meshes, boundary maps and base domains. This article provides a brief overview of the field, discussing its importance and the lack of effective techniques. We then introduce both the dataset and its major features. An example of comparative analysis between two existing methods is also present