1 research outputs found
Adversarial Mutual Information for Text Generation
Recent advances in maximizing mutual information (MI) between the source and
target have demonstrated its effectiveness in text generation. However,
previous works paid little attention to modeling the backward network of MI
(i.e., dependency from the target to the source), which is crucial to the
tightness of the variational information maximization lower bound. In this
paper, we propose Adversarial Mutual Information (AMI): a text generation
framework which is formed as a novel saddle point (min-max) optimization aiming
to identify joint interactions between the source and target. Within this
framework, the forward and backward networks are able to iteratively promote or
demote each other's generated instances by comparing the real and synthetic
data distributions. We also develop a latent noise sampling strategy that
leverages random variations at the high-level semantic space to enhance the
long term dependency in the generation process. Extensive experiments based on
different text generation tasks demonstrate that the proposed AMI framework can
significantly outperform several strong baselines, and we also show that AMI
has potential to lead to a tighter lower bound of maximum mutual information
for the variational information maximization problem.Comment: Published at ICML 202