35 research outputs found
A Cheaper and Better Diffusion Language Model with Soft-Masked Noise
Diffusion models that are based on iterative denoising have been recently
proposed and leveraged in various generation tasks like image generation.
Whereas, as a way inherently built for continuous data, existing diffusion
models still have some limitations in modeling discrete data, e.g., languages.
For example, the generally used Gaussian noise can not handle the discrete
corruption well, and the objectives in continuous spaces fail to be stable for
textual data in the diffusion process especially when the dimension is high. To
alleviate these issues, we introduce a novel diffusion model for language
modeling, Masked-Diffuse LM, with lower training cost and better performances,
inspired by linguistic features in languages. Specifically, we design a
linguistic-informed forward process which adds corruptions to the text through
strategically soft-masking to better noise the textual data. Also, we directly
predict the categorical distribution with cross-entropy loss function in every
diffusion step to connect the continuous space and discrete space in a more
efficient and straightforward way. Through experiments on 5 controlled
generation tasks, we demonstrate that our Masked-Diffuse LM can achieve better
generation quality than the state-of-the-art diffusion models with better
efficiency.Comment: Code is available at
https://github.com/amazon-science/masked-diffusion-l