103 research outputs found
Partially Randomizing Transformer Weights for Dialogue Response Diversity
Despite recent progress in generative open-domain dialogue, the issue of low
response diversity persists. Prior works have addressed this issue via either
novel objective functions, alternative learning approaches such as variational
frameworks, or architectural extensions such as the Randomized Link (RL)
Transformer. However, these approaches typically entail either additional
difficulties during training/inference, or a significant increase in model size
and complexity. Hence, we propose the \underline{Pa}rtially
\underline{Ra}ndomized trans\underline{Former} (PaRaFormer), a simple extension
of the transformer which involves freezing the weights of selected layers after
random initialization. Experimental results reveal that the performance of the
PaRaformer is comparable to that of the aforementioned approaches, despite not
entailing any additional training difficulty or increase in model complexity
An Empirical Bayes Framework for Open-Domain Dialogue Generation
To engage human users in meaningful conversation, open-domain dialogue agents
are required to generate diverse and contextually coherent dialogue. Despite
recent advancements, which can be attributed to the usage of pretrained
language models, the generation of diverse and coherent dialogue remains an
open research problem. A popular approach to address this issue involves the
adaptation of variational frameworks. However, while these approaches
successfully improve diversity, they tend to compromise on contextual
coherence. Hence, we propose the Bayesian Open-domain Dialogue with Empirical
Bayes (BODEB) framework, an empirical bayes framework for constructing an
Bayesian open-domain dialogue agent by leveraging pretrained parameters to
inform the prior and posterior parameter distributions. Empirical results show
that BODEB achieves better results in terms of both diversity and coherence
compared to variational frameworks
- …