6 research outputs found
Noise-Free Score Distillation
Score Distillation Sampling (SDS) has emerged as the de facto approach for
text-to-content generation in non-image domains. In this paper, we reexamine
the SDS process and introduce a straightforward interpretation that demystifies
the necessity for large Classifier-Free Guidance (CFG) scales, rooted in the
distillation of an undesired noise term. Building upon our interpretation, we
propose a novel Noise-Free Score Distillation (NFSD) process, which requires
minimal modifications to the original SDS framework. Through this streamlined
design, we achieve more effective distillation of pre-trained text-to-image
diffusion models while using a nominal CFG scale. This strategic choice allows
us to prevent the over-smoothing of results, ensuring that the generated data
is both realistic and complies with the desired prompt. To demonstrate the
efficacy of NFSD, we provide qualitative examples that compare NFSD and SDS, as
well as several other methods.Comment: Project page at https://orenkatzir.github.io/nfsd
Localizing Object-level Shape Variations with Text-to-Image Diffusion Models
Text-to-image models give rise to workflows which often begin with an
exploration step, where users sift through a large collection of generated
images. The global nature of the text-to-image generation process prevents
users from narrowing their exploration to a particular object in the image. In
this paper, we present a technique to generate a collection of images that
depicts variations in the shape of a specific object, enabling an object-level
shape exploration process. Creating plausible variations is challenging as it
requires control over the shape of the generated object while respecting its
semantics. A particular challenge when generating object variations is
accurately localizing the manipulation applied over the object's shape. We
introduce a prompt-mixing technique that switches between prompts along the
denoising process to attain a variety of shape choices. To localize the
image-space operation, we present two techniques that use the self-attention
layers in conjunction with the cross-attention layers. Moreover, we show that
these localization techniques are general and effective beyond the scope of
generating object variations. Extensive results and comparisons demonstrate the
effectiveness of our method in generating object variations, and the competence
of our localization techniques.Comment: ICCV 2023. Project page at
https://orpatashnik.github.io/local-prompt-mixing
CLiC: Concept Learning in Context
This paper addresses the challenge of learning a local visual pattern of an
object from one image, and generating images depicting objects with that
pattern. Learning a localized concept and placing it on an object in a target
image is a nontrivial task, as the objects may have different orientations and
shapes. Our approach builds upon recent advancements in visual concept
learning. It involves acquiring a visual concept (e.g., an ornament) from a
source image and subsequently applying it to an object (e.g., a chair) in a
target image. Our key idea is to perform in-context concept learning, acquiring
the local visual concept within the broader context of the objects they belong
to. To localize the concept learning, we employ soft masks that contain both
the concept within the mask and the surrounding image area. We demonstrate our
approach through object generation within an image, showcasing plausible
embedding of in-context learned concepts. We also introduce methods for
directing acquired concepts to specific locations within target images,
employing cross-attention mechanisms, and establishing correspondences between
source and target objects. The effectiveness of our method is demonstrated
through quantitative and qualitative experiments, along with comparisons
against baseline techniques
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion
Text-to-image models offer unprecedented freedom to guide creation through
natural language. Yet, it is unclear how such freedom can be exercised to
generate images of specific unique concepts, modify their appearance, or
compose them in new roles and novel scenes. In other words, we ask: how can we
use language-guided models to turn our cat into a painting, or imagine a new
product based on our favorite toy? Here we present a simple approach that
allows such creative freedom. Using only 3-5 images of a user-provided concept,
like an object or a style, we learn to represent it through new "words" in the
embedding space of a frozen text-to-image model. These "words" can be composed
into natural language sentences, guiding personalized creation in an intuitive
way. Notably, we find evidence that a single word embedding is sufficient for
capturing unique and varied concepts. We compare our approach to a wide range
of baselines, and demonstrate that it can more faithfully portray the concepts
across a range of applications and tasks.
Our code, data and new words will be available at:
https://textual-inversion.github.ioComment: Project page: https://textual-inversion.github.i