34 research outputs found
Cramming: Training a Language Model on a Single GPU in One Day
Recent trends in language modeling have focused on increasing performance
through scaling, and have resulted in an environment where training language
models is out of reach for most researchers and practitioners. While most in
the community are asking how to push the limits of extreme computation, we ask
the opposite question: How far can we get with a single GPU in just one day?
We investigate the downstream performance achievable with a transformer-based
language model trained completely from scratch with masked language modeling
for a single day on a single consumer GPU. Aside from re-analyzing nearly all
components of the pretraining pipeline for this scenario and providing a
modified pipeline with performance close to BERT, we investigate why scaling
down is hard, and which modifications actually improve performance in this
scenario. We provide evidence that even in this constrained setting,
performance closely follows scaling laws observed in large-compute settings.
Through the lens of scaling laws, we categorize a range of recent improvements
to training and architecture and discuss their merit and practical
applicability (or lack thereof) for the limited compute setting.Comment: 22 pages, we provide code at https://github.com/JonasGeiping/crammin
JPEG Compressed Images Can Bypass Protections Against AI Editing
Recently developed text-to-image diffusion models make it easy to edit or
create high-quality images. Their ease of use has raised concerns about the
potential for malicious editing or deepfake creation. Imperceptible
perturbations have been proposed as a means of protecting images from malicious
editing by preventing diffusion models from generating realistic images.
However, we find that the aforementioned perturbations are not robust to JPEG
compression, which poses a major weakness because of the common usage and
availability of JPEG. We discuss the importance of robustness for additive
imperceptible perturbations and encourage alternative approaches to protect
images against editing.Comment: 8 pages, 8 figure
DARTS for Inverse Problems: a Study on Hyperparameter Sensitivity
Differentiable architecture search (DARTS) is a widely researched tool for
the discovery of novel architectures, due to its promising results for image
classification. The main benefit of DARTS is the effectiveness achieved through
the weight-sharing one-shot paradigm, which allows efficient architecture
search. In this work, we investigate DARTS in a systematic case study of
inverse problems, which allows us to analyze these potential benefits in a
controlled manner. We demonstrate that the success of DARTS can be extended
from image classification to signal reconstruction, in principle. However, our
experiments also expose three fundamental difficulties in the evaluation of
DARTS-based methods in inverse problems: First, the results show a large
variance in all test cases. Second, the final performance is highly dependent
on the hyperparameters of the optimizer. And third, the performance of the
weight-sharing architecture used during training does not reflect the final
performance of the found architecture well. Thus, we conclude the necessity to
1) report the results of any DARTS-based methods from several runs along with
its underlying performance statistics, 2) show the correlation of the training
and final architecture performance, and 3) carefully consider if the
computational efficiency of DARTS outweighs the costs of hyperparameter
optimization and multiple runs.Comment: 11 pages, 5 figures. First two and last two authors contributed each
equall
Tree-Ring Watermarks: Fingerprints for Diffusion Images that are Invisible and Robust
Watermarking the outputs of generative models is a crucial technique for
tracing copyright and preventing potential harm from AI-generated content. In
this paper, we introduce a novel technique called Tree-Ring Watermarking that
robustly fingerprints diffusion model outputs. Unlike existing methods that
perform post-hoc modifications to images after sampling, Tree-Ring Watermarking
subtly influences the entire sampling process, resulting in a model fingerprint
that is invisible to humans. The watermark embeds a pattern into the initial
noise vector used for sampling. These patterns are structured in Fourier space
so that they are invariant to convolutions, crops, dilations, flips, and
rotations. After image generation, the watermark signal is detected by
inverting the diffusion process to retrieve the noise vector, which is then
checked for the embedded signal. We demonstrate that this technique can be
easily applied to arbitrary diffusion models, including text-conditioned Stable
Diffusion, as a plug-in with negligible loss in FID. Our watermark is
semantically hidden in the image space and is far more robust than watermarking
alternatives that are currently deployed. Code is available at
https://github.com/YuxinWenRick/tree-ring-watermark.Comment: 16 pages, 8 figures, code is available at
https://github.com/YuxinWenRick/tree-ring-watermark, fixed the repo lin
Understanding and Mitigating Copying in Diffusion Models
Images generated by diffusion models like Stable Diffusion are increasingly
widespread. Recent works and even lawsuits have shown that these models are
prone to replicating their training data, unbeknownst to the user. In this
paper, we first analyze this memorization problem in text-to-image diffusion
models. While it is widely believed that duplicated images in the training set
are responsible for content replication at inference time, we observe that the
text conditioning of the model plays a similarly important role. In fact, we
see in our experiments that data replication often does not happen for
unconditional models, while it is common in the text-conditional case.
Motivated by our findings, we then propose several techniques for reducing data
replication at both training and inference time by randomizing and augmenting
image captions in the training set.Comment: 17 pages, preprint. Code is available at
https://github.com/somepago/DC
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models
Cutting-edge diffusion models produce images with high quality and
customizability, enabling them to be used for commercial art and graphic design
purposes. But do diffusion models create unique works of art, or are they
replicating content directly from their training sets? In this work, we study
image retrieval frameworks that enable us to compare generated images with
training samples and detect when content has been replicated. Applying our
frameworks to diffusion models trained on multiple datasets including Oxford
flowers, Celeb-A, ImageNet, and LAION, we discuss how factors such as training
set size impact rates of content replication. We also identify cases where
diffusion models, including the popular Stable Diffusion model, blatantly copy
from their training data.Comment: Updated draft with the following changes (1) Clarified the LAION
Aesthetics versions everywhere (2) Correction on which LAION Aesthetics
version SD - 1.4 is finetuned on and updated figure 12 based on this (3) A
section on possible causes of replicatio