State of the Art in Example-based Texture Synthesis

International audienceRecent years have witnessed significant progress in example-based texture synthesis algorithms. Given an example texture, these methods produce a larger texture that is tailored to the user's needs. In this state-of-the-art report, we aim to achieve three goals: (1) provide a tutorial that is easy to follow for readers who are not already familiar with the subject, (2) make a comprehensive survey and comparisons of different methods, and (3) sketch a vision for future work that can help motivate and guide readers that are interested in texture synthesis research. We cover fundamental algorithms as well as extensions and applications of texture synthesis

Synthesis and evaluation of geometric textures

Two-dimensional geometric textures are the geometric analogues of raster (pixel-based) textures and consist of planar distributions of discrete shapes with an inherent structure. These textures have many potential applications in art, computer graphics, and cartography. Synthesizing large textures by hand is generally a tedious task. In raster-based synthesis, many algorithms have been developed to limit the amount of manual effort required. These algorithms take in a small example as a reference and produce larger similar textures using a wide range of approaches. Recently, an increasing number of example-based geometric synthesis algorithms have been proposed. I refer to them in this dissertation as Geometric Texture Synthesis (GTS) algorithms. Analogous to their raster-based counterparts, GTS algorithms synthesize arrangements that ought to be judged by human viewers as “similar” to the example inputs. However, an absence of conventional evaluation procedures in current attempts demands an inquiry into the visual significance of synthesized results. In this dissertation, I present an investigation into GTS and report on my findings from three projects. I start by offering initial steps towards grounding texture synthesis techniques more firmly with our understanding of visual perception through two psychophysical studies. My observations throughout these studies result in important visual cues used by people when generating and/or comparing similarity of geometric arrangements as well a set of strategies adopted by participants when generating arrangements. Based on one of the generation strategies devised in these studies I develop a new geometric synthesis algorithm that uses a tile-based approach to generate arrangements. Textures synthesized by this algorithm are comparable to the state of the art in GTS and provide an additional reference in subsequent evaluations. To conduct effective evaluations of GTS, I start by collecting a set of representative examples, use them to acquire arrangements from multiple sources, and then gather them into a dataset that acts as a standard for the GTS research community. I then utilize this dataset in a second set of psychophysical studies that define an effective methodology for comparing current and future geometric synthesis algorithms

Texture Synthesis Through Convolutional Neural Networks and Spectrum Constraints

This paper presents a significant improvement for the synthesis of texture images using convolutional neural networks (CNNs), making use of constraints on the Fourier spectrum of the results. More precisely, the texture synthesis is regarded as a constrained optimization problem, with constraints conditioning both the Fourier spectrum and statistical features learned by CNNs. In contrast with existing methods, the presented method inherits from previous CNN approaches the ability to depict local structures and fine scale details, and at the same time yields coherent large scale structures, even in the case of quasi-periodic images. This is done at no extra computational cost. Synthesis experiments on various images show a clear improvement compared to a recent state-of-the art method relying on CNN constraints only

A survey of exemplar-based texture synthesis

Exemplar-based texture synthesis is the process of generating, from an input sample, new texture images of arbitrary size and which are perceptually equivalent to the sample. The two main approaches are statistics-based methods and patch re-arrangement methods. In the first class, a texture is characterized by a statistical signature; then, a random sampling conditioned to this signature produces genuinely different texture images. The second class boils down to a clever "copy-paste" procedure, which stitches together large regions of the sample. Hybrid methods try to combine ideas from both approaches to avoid their hurdles. The recent approaches using convolutional neural networks fit to this classification, some being statistical and others performing patch re-arrangement in the feature space. They produce impressive synthesis on various kinds of textures. Nevertheless, we found that most real textures are organized at multiple scales, with global structures revealed at coarse scales and highly varying details at finer ones. Thus, when confronted with large natural images of textures the results of state-of-the-art methods degrade rapidly, and the problem of modeling them remains wide open.Comment: v2: Added comments and typos fixes. New section added to describe FRAME. New method presented: CNNMR