A survey of exemplar-based texture synthesis

Exemplar-based texture synthesis is the process of generating, from an input sample, new texture images of arbitrary size and which are perceptually equivalent to the sample. The two main approaches are statistics-based methods and patch re-arrangement methods. In the first class, a texture is characterized by a statistical signature; then, a random sampling conditioned to this signature produces genuinely different texture images. The second class boils down to a clever "copy-paste" procedure, which stitches together large regions of the sample. Hybrid methods try to combine ideas from both approaches to avoid their hurdles. The recent approaches using convolutional neural networks fit to this classification, some being statistical and others performing patch re-arrangement in the feature space. They produce impressive synthesis on various kinds of textures. Nevertheless, we found that most real textures are organized at multiple scales, with global structures revealed at coarse scales and highly varying details at finer ones. Thus, when confronted with large natural images of textures the results of state-of-the-art methods degrade rapidly, and the problem of modeling them remains wide open.Comment: v2: Added comments and typos fixes. New section added to describe FRAME. New method presented: CNNMR

ChromaGAN: Adversarial Picture Colorization with Semantic Class Distribution

The colorization of grayscale images is an ill-posed problem, with multiple correct solutions. In this paper, we propose an adversarial learning colorization approach coupled with semantic information. A generative network is used to infer the chromaticity of a given grayscale image conditioned to semantic clues. This network is framed in an adversarial model that learns to colorize by incorporating perceptual and semantic understanding of color and class distributions. The model is trained via a fully self-supervised strategy. Qualitative and quantitative results show the capacity of the proposed method to colorize images in a realistic way achieving state-of-the-art results.Comment: 8 pages + reference

Scaling Painting Style Transfer

Neural style transfer is a deep learning technique that produces an unprecedentedly rich style transfer from a style image to a content image and is particularly impressive when it comes to transferring style from a painting to an image. It was originally achieved by solving an optimization problem to match the global style statistics of the style image while preserving the local geometric features of the content image. The two main drawbacks of this original approach is that it is computationally expensive and that the resolution of the output images is limited by high GPU memory requirements. Many solutions have been proposed to both accelerate neural style transfer and increase its resolution, but they all compromise the quality of the produced images. Indeed, transferring the style of a painting is a complex task involving features at different scales, from the color palette and compositional style to the fine brushstrokes and texture of the canvas. This paper provides a solution to solve the original global optimization for ultra-high resolution images, enabling multiscale style transfer at unprecedented image sizes. This is achieved by spatially localizing the computation of each forward and backward passes through the VGG network. Extensive qualitative and quantitative comparisons show that our method produces a style transfer of unmatched quality for such high resolution painting styles.Comment: 10 pages, 5 figure

Advanced security testing using a cyber-attack forecasting model: A case study of financial institutions

As the number of cyber-attacks on financial institutions has increased over the past few years, an advanced system that is capable of predicting the target of an attack is essential. Such a system needs to be integrated into the existing detection systems of financial institutions as it provides them with proactive controls with which to halt an attack by predicting patterns. Advanced prediction systems also enhance the software design and security testing of new advanced cyber-security measures by providing new testing scenarios supported by attack forecasting. This present study developed a model that forecasts future network-based cyber-attacks on financial institutions using a deep neural network. The dataset that was used to train and test the model consisted of some of the biggest cyber-attacks on banking institutions over the past three years. This provided insight into new patterns that may end with a cyber-crime. These new attacks were also evaluated to determine behavioral similarities with the nearest known attack or a combination of several existing attacks. The performance of the forecasting model was then evaluated in a real banking environment and provided a forecasting accuracy of 90.36%. As such, financial institutions can use the proposed forecasting model to improve their security testing measures.Dado que el número de ciberataques a instituciones financieras ha aumentado en los últimos años, es esencial contar con un sistema avanzado que sea capaz de predecir el objetivo de un ataque. Un sistema de este tipo debe integrarse en los sistemas de detección existentes de las instituciones financieras, ya que les proporciona controles proactivos con los que detener un ataque mediante la predicción de patrones. Los sistemas de predicción avanzados también mejoran el diseño de software y las pruebas de seguridad de nuevas medidas avanzadas de ciberseguridad al proporcionar nuevos escenarios de prueba respaldados por la previsión de ataques. Este presente estudio desarrolló un modelo que pronostica futuros ciberataques basados ​​en redes contra instituciones financieras utilizando una red neuronal profunda. El conjunto de datos que se utilizó para entrenar y probar el modelo consistió en algunos de los mayores ataques cibernéticos a instituciones bancarias en los últimos tres años. Esto proporcionó información sobre nuevos patrones que pueden terminar en un delito cibernético. Estos nuevos ataques también fueron evaluados para determinar similitudes de comportamiento con el ataque conocido más cercano o una combinación de varios ataques existentes. Luego se evaluó el desempeño del modelo de pronóstico en un entorno bancario real y proporcionó una precisión de pronóstico del 90,36%. Como tal, las instituciones financieras pueden utilizar el modelo de pronóstico propuesto para mejorar sus medidas de prueba de seguridad.2022-2

Sperm DNA fragmentation: A new guideline for clinicians

Sperm DNA integrity is crucial for fertilization and development of healthy offspring. The spermatozoon undergoes extensive molecular remodeling of its nucleus during later phases of spermatogenesis, which imparts compaction and protects the genetic content. Testicular (defective maturation and abortive apoptosis) and post-testicular (oxidative stress) mechanisms are implicated in the etiology of sperm DNA fragmentation (SDF), which affects both natural and assisted reproduction. Several clinical and environmental factors are known to negatively impact sperm DNA integrity. An increasing number of reports emphasizes the direct relationship between sperm DNA damage and male infertility. Currently, several assays are available to assess sperm DNA damage, however, routine assessment of SDF in clinical practice is not recommended by professional organizations

Synthèse de texture à partir d’exemples : modèles et applications

This dissertation contributes to the problem of exemplar based texture synthesis by introducing the use of local Gaussian patch models to generate new texture images. Exemplar based texture synthesis is the process of generating, from an input texture sample, new texture images that are perceptually equivalent to the input. There are roughly two main categories of algorithms: the statistics based methods and the non parametric patch based methods. The first one aims to characterize a given texture sample by estimating a set of statistics which will define an underlying stochastic process. The results of this kind of methods are satisfying but only on a small group of textures, failing when important structures are visible in the input provided. The second category methods reorganize local neighborhoods from the input sample in a consistent way creating new texture images. These methods return impressive visual results. Nevertheless, they often yield verbatim copies of large parts of the input sample. Furthermore, they can diverge, starting to reproduce iteratively one part of the input sample and neglecting the rest of it, thus growing ``garbage''. In this thesis we propose a technique combining ideas from the statistic based methods and from the non parametric patch based methods. We call it the locally Gaussian method. The method keeps the positive aspects of both categories: the innovation capacity of the parametric methods and the ability to synthesize highly structured textures of the non parametric methods. To this aim, the self-similarities of a given input texture are modeled with conditional multivariate Gaussian distributions in the patch space. In general, the results that we obtain are visually superior to those obtained with statistic based methods while using local parametric models. On the other hand, our results are comparable to the visual results obtained with the non parametric patch based methods. This dissertation addresses another weakness of all patch based methods. They are strongly dependent on the patch size used, which is decided manually. It is therefore crucial to fix a correct patch size for each synthesis. Since texture images have, in general, details at different scales, we decided to extend the method to a multiscale approach which reduces the strong dependency of the method on the patch size. Patch based methods involve a stitching step. Indeed, the patches used for the synthesis process overlap each other. This overlap must be taken into account to avoid any transition artifact from patch to patch. Our first attempt to deal with it was to consider directly the overlap constraints in the local parametric model. The experiments show that for periodic and pseudo-periodic textures, considering these constraints in the parametrization is enough to avoid the stitching step. Nevertheless, for more complex textures it is not enough, and this led us to suggest a new stitching technique inspired by optimal transport and midway histogram equalization.This thesis ends with an extensive analysis of the generation of several natural textures. This study shows that, in spite of remarkable progress for local textures, the methods proposed in the extensive literature of exemplar based texture synthesis still are incapable of dealing with complex and non-stationary textures.Cette thèse s’attaque au problème de la synthèse de texture par l’exemple en utilisant des modèles stochastiques locaux de patchs pour générer de nouvelles images. La synthèse de texture par l’exemple a pour but de générer à partir d’un échantillon de texture de nouvelles images qui sont perceptuellement équivalentes à celle de départ. Les méthodes peuvent se regrouper en deux catégories: les méthodes paramétriques et les non paramétriques à base de patchs. Le premier groupe a pour but de caractériser une image de texture à partir d’un ensemble de statistiques qui définissent un processus stochastique sous-jacent. Les résultats visuels de ces méthodes sont satisfaisants, mais seulement pour un groupe réduit de types de texture. La synthèse pour des images de textures ayant des structures très contrastées peut échouer. La deuxième catégorie d’algorithme découpe, puis recolle de manière consistante des voisinages locaux de l’image de départ pour générer de nouvelles configurations plausibles de ces voisinages (ou patchs). Les résultats visuels de ces méthodes sont impressionnants. Néanmoins, on observe souvent des répétitions verbatim de grandes parties de l’image d’entrée qui du coup peuvent être reproduites plusieurs fois. De plus, ces algorithmes peuvent diverger, reproduisant de façon itérative une partie de l’image de l’entrée en négligeant le reste. La première partie de cette thèse présente une approche combinant des idées des deux catégories de méthodes, sous le nom de synthèse localement Gaussienne. On préserve dans cette nouvelle méthode les aspects positifs de chaque approche: la capacité d’innover des méthodes paramétriques, et la capacité de générer des textures fortement structurées des méthodes non paramétriques à base de patchs. Pour ce faire, on construit un modèle Gaussien multidimensionnel des auto-similarités d’une image de texture. Ainsi, on obtient des résultats qui sont visuellement supérieurs à ceux obtenus avec les méthodes paramétriques et qui sont comparables à ceux obtenus avec les méthodes non-paramétriques à base de patchs tout en utilisant une paramétrization locale de l’image. La thèse s’attache aussi à résoudre une autre difficulté des méthodes à base de patchs: le choix de la taille du patch. Afin de réduire significativement cette dépendance, on propose une extension multi échelle de la méthode. Les méthodes à bases de patchs supposent une étape de recollement. En effet, les patchs de l’image synthétisée se superposent entre eux, il faut donc gérer le recollement dans ces zones. La première approche qu’on a considérée consiste à prendre en compte cette contrainte de superposition dans la modélisation des patchs. Les expériences montrent que cela est satisfaisant pour des images de textures périodiques ou pseudo-périodiques et qu’en conséquence l’étape de recollement peut être supprimée pour ces textures. Cependant, pour des images de textures plus complexes ce n’est pas le cas, ce qui nous a menée à suggérer une nouvelle méthode de recollement inspirée du transport optimal. Cette thèse conclut avec une étude complète de l’état de l’art en génération d’images de textures naturelles. L’étude que nous présentons montre que, malgré les progrès considérables des méthodes de synthèse à base d’exemples proposées dans la vaste littérature, et même en les combinant astucieusement, celles-ci sont encore incapables d’émuler des textures complexes et non stationnaires

Efros and Freeman Image Quilting Algorithm for Texture Synthesis

International audienceExemplar-based texture synthesis is defined as the process of generating, from an input texture sample, new texture images that are perceptually equivalent to the input. Efros and Freeman's method is a non-parametric patch-based method which computes an output texture image by quilting together patches taken from the input sample. The main innovation of their work relies in the stitching technique which significantly reduces the transition effect between patches. In this paper, we propose a detailed analysis and implementation of their work. We provide a complete mathematical description of the linear programing problem used for the quilting step as well as implementation details. Additionally we propose a partially parallel version of the quilting technique.https://doi.org/10.5201/ipol.2017.17

Scaling Painting Style Transfer

Neural style transfer is a deep learning technique that produces an unprecedentedly rich style transfer from a style image to a content image and is particularly impressive when it comes to transferring style from a painting to an image. It was originally achieved by solving an optimization problem to match the global style statistics of the style image while preserving the local geometric features of the content image. The two main drawbacks of this original approach is that it is computationally expensive and that the resolution of the output images is limited by high GPU memory requirements.Many solutions have been proposed to both accelerate neural style transfer and increase its resolution, but they all compromise the quality of the produced images. Indeed, transferring the style of a painting is a complex task involving features at different scales, from the color palette and compositional style to the fine brushstrokes and texture of the canvas. This paper provides a solution to solve the original global optimization for ultra-high resolution images, enabling multiscale style transfer at unprecedented image sizes. This is achieved by spatially localizing the computation of each forward and backward passes through the VGG network. Extensive qualitative and quantitative comparisons show that our method produces a style transfer of unmatched quality for such high resolution painting styles