StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learners

We investigate the potential of learning visual representations using synthetic images generated by text-to-image models. This is a natural question in the light of the excellent performance of such models in generating high-quality images. We consider specifically the Stable Diffusion, one of the leading open source text-to-image models. We show that (1) when the generative model is configured with proper classifier-free guidance scale, training self-supervised methods on synthetic images can match or beat the real image counterpart; (2) by treating the multiple images generated from the same text prompt as positives for each other, we develop a multi-positive contrastive learning method, which we call StableRep. With solely synthetic images, the representations learned by StableRep surpass the performance of representations learned by SimCLR and CLIP using the same set of text prompts and corresponding real images, on large scale datasets. When we further add language supervision, StableRep trained with 20M synthetic images achieves better accuracy than CLIP trained with 50M real images.Comment: code is available at: https://github.com/google-research/syn-rep-lear

BLT: Bidirectional Layout Transformer for Controllable Layout Generation

Creating visual layouts is a critical step in graphic design. Automatic generation of such layouts is essential for scalable and diverse visual designs. To advance conditional layout generation, we introduce BLT, a bidirectional layout transformer. BLT differs from previous work on transformers in adopting non-autoregressive transformers. In training, BLT learns to predict the masked attributes by attending to surrounding attributes in two directions. During inference, BLT first generates a draft layout from the input and then iteratively refines it into a high-quality layout by masking out low-confident attributes. The masks generated in both training and inference are controlled by a new hierarchical sampling policy. We verify the proposed model on six benchmarks of diverse design tasks. Experimental results demonstrate two benefits compared to the state-of-the-art layout transformer models. First, our model empowers layout transformers to fulfill controllable layout generation. Second, it achieves up to 10x speedup in generating a layout at inference time than the layout transformer baseline. Code is released at https://shawnkx.github.io/blt.Comment: ECCV 202

Learning Disentangled Prompts for Compositional Image Synthesis

We study domain-adaptive image synthesis, the problem of teaching pretrained image generative models a new style or concept from as few as one image to synthesize novel images, to better understand the compositional image synthesis. We present a framework that leverages a pretrained class-conditional generation model and visual prompt tuning. Specifically, we propose a novel source class distilled visual prompt that learns disentangled prompts of semantic (e.g., class) and domain (e.g., style) from a few images. Learned domain prompt is then used to synthesize images of any classes in the style of target domain. We conduct studies on various target domains with the number of images ranging from one to a few to many, and show qualitative results which show the compositional generalization of our method. Moreover, we show that our method can help improve zero-shot domain adaptation classification accuracy.Comment: tech repor

VQ3D: Learning a 3D-Aware Generative Model on ImageNet

Recent work has shown the possibility of training generative models of 3D content from 2D image collections on small datasets corresponding to a single object class, such as human faces, animal faces, or cars. However, these models struggle on larger, more complex datasets. To model diverse and unconstrained image collections such as ImageNet, we present VQ3D, which introduces a NeRF-based decoder into a two-stage vector-quantized autoencoder. Our Stage 1 allows for the reconstruction of an input image and the ability to change the camera position around the image, and our Stage 2 allows for the generation of new 3D scenes. VQ3D is capable of generating and reconstructing 3D-aware images from the 1000-class ImageNet dataset of 1.2 million training images. We achieve an ImageNet generation FID score of 16.8, compared to 69.8 for the next best baseline method.Comment: 15 pages. For visual results, please visit the project webpage at http://kylesargent.github.io/vq3

Palette: Image-to-Image Diffusion Models

This paper develops a unified framework for image-to-image translation based on conditional diffusion models and evaluates this framework on four challenging image-to-image translation tasks, namely colorization, inpainting, uncropping, and JPEG restoration. Our simple implementation of image-to-image diffusion models outperforms strong GAN and regression baselines on all tasks, without task-specific hyper-parameter tuning, architecture customization, or any auxiliary loss or sophisticated new techniques needed. We uncover the impact of an L2 vs. L1 loss in the denoising diffusion objective on sample diversity, and demonstrate the importance of self-attention in the neural architecture through empirical studies. Importantly, we advocate a unified evaluation protocol based on ImageNet, with human evaluation and sample quality scores (FID, Inception Score, Classification Accuracy of a pre-trained ResNet-50, and Perceptual Distance against original images). We expect this standardized evaluation protocol to play a role in advancing image-to-image translation research. Finally, we show that a generalist, multi-task diffusion model performs as well or better than task-specific specialist counterparts. Check out https://diffusion-palette.github.io for an overview of the results

Zoom-to-Inpaint: Image Inpainting with High-Frequency Details

Although deep learning has enabled a huge leap forward in image inpainting, current methods are often unable to synthesize realistic high-frequency details. In this paper, we propose applying super-resolution to coarsely reconstructed outputs, refining them at high resolution, and then downscaling the output to the original resolution. By introducing high-resolution images to the refinement network, our framework is able to reconstruct finer details that are usually smoothed out due to spectral bias - the tendency of neural networks to reconstruct low frequencies better than high frequencies. To assist training the refinement network on large upscaled holes, we propose a progressive learning technique in which the size of the missing regions increases as training progresses. Our zoom-in, refine and zoom-out strategy, combined with high-resolution supervision and progressive learning, constitutes a framework-agnostic approach for enhancing high-frequency details that can be applied to any CNN-based inpainting method. We provide qualitative and quantitative evaluations along with an ablation analysis to show the effectiveness of our approach. This seemingly simple, yet powerful approach, outperforms state-of-the-art inpainting methods

RNA-binding protein CUGBP1 regulates insulin secretion via activation of phosphodiesterase 3B in mice

International audienceAims/hypothesis: CUG-binding protein 1 (CUGBP1) is a multifunctional RNA-binding protein that regulates RNA processing at several stages including translation, deadenylation and alternative splicing, as well as RNA stability. Recent studies indicate that CUGBP1 may play a role in metabolic disorders. Our objective was to examine its role in endocrine pancreas function through gain- and loss-of-function experiments and to further decipher the underlying molecular mechanisms.Methods: A mouse model in which type 2 diabetes was induced by a high-fat diet (HFD; 60% energy from fat) and mice on a standard chow diet (10% energy from fat) were compared. Pancreas-specific CUGBP1 overexpression and knockdown mice were generated. Different lengths of the phosphodiesterase subtype 3B (PDE3B) 3′ untranslated region (UTR) were cloned for luciferase reporter analysis. Purified CUGBP1 protein was used for gel shift experiments.Results: CUGBP1 is present in rodent islets and in beta cell lines; it is overexpressed in the islets of diabetic mice. Compared with control mice, the plasma insulin level after a glucose load was significantly lower and glucose clearance was greatly delayed in mice with pancreas-specific CUGBP1 overexpression; the opposite results were obtained upon pancreas-specific CUGBP1 knockdown. Glucose- and glucagon-like peptide1 (GLP-1)-stimulated insulin secretion was significantly attenuated in mouse islets upon CUGBP1 overexpression. This was associated with a strong decrease in intracellular cAMP levels, pointing to a potential role for cAMP PDEs. CUGBP1 overexpression had no effect on the mRNA levels of PDE1A, 1C, 2A, 3A, 4A, 4B, 4D, 7A and 8B subtypes, but resulted in increased PDE3B expression. CUGBP1 was found to directly bind to a specific ATTTGTT sequence residing in the 3′ UTR of PDE3B and stabilised PDE3B mRNA. In the presence of the PDE3 inhibitor cilostamide, glucose- and GLP-1-stimulated insulin secretion was no longer reduced by CUGBP1 overexpression. Similar to CUGBP1, PDE3B was overexpressed in the islets of diabetic mice.Conclusions/interpretation: We conclude that CUGBP1 is a critical regulator of insulin secretion via activating PDE3B. Repressing this protein might provide a potential strategy for treating type 2 diabetes