This  work  was  funded  by  a  Google  Faculty  Re-
search Award (RG91111), an EPSRC studentship
(RG80792),   an  EPSRC  grant  (EP/M018946/1)
and   by   Toshiba   Research   Europe   Ltd,   Cam-
bridge  Research  Laboratory  (RG85875

Budzianowski, PF

Casanueva, Inigo

Gasic, M

Tseng, B-H

English

Apollo (Cambridge)

Towards end-to-end multi-domain dialogue modellingTechnical ReportCUED/F-INFENG/TR.70629th September 2018Paweł Budzianowski, Iñigo Casanueva,Bo-Hsiang Tseng, Milica Gašić{pfb30,mg436}@cam.ac.ukDepartment of Engineering, Cambridge UniversityISSN: 0951-921111 IntroductionStatistical approaches to dialogue modelling allow automatic optimisation of theSpoken Dialogue Systems (SDS) [Young, 2002]. SDS typically comprise variousstatistical components. This includes a spoken language understanding module,which takes a sentence as input and gives a dialogue act as output, a dialoguebelief state tracker that predicts user intent and track the dialogue history, adialogue policy to determine the dialogue flow, and a natural language generatorto convert conceptual representations into system responses.At the other end of the spectrum, sequence to sequence learning [Sutskever et al., 2014]has inspired several efforts to build end-to-end trainable conversational systems[Serban et al., 2015, Wen et al., 2017b]. This family of approaches treats dia-logue as a source to target sequence transduction problem, applying an encodernetwork [Cho et al., 2014] to encode a user query into a distributed vector rep-resenting its semantics, which then conditions a decoder network to generateeach system response. These models are limited to one domain and involve thepre-training of some of their components. Moreover, they typically require alarge amount of data to train.In this report we introduce a multi-domain dialogue architecture with cross-domain database pointer and the attention over the input sequence. The modelis inspired by the previous works of [Wen et al., 2017b, Eric et al., 2017]. Themodel is evaluated on a recently collected multi-domain dataset spanning acrossmany domains. Initial benchmarks on dialogue-context-to-text generation areprovided illustrating a room for improvement. We discuss inefficiencies of thecurrent architecture and possible future work.2 End-to-end dialogue modellingAdvances in sequence to sequence learning [Sutskever et al., 2014] has inspiredseveral efforts to build end-to-end trainable, task-oriented conversational sys-tems where modular approach [Young et al., 2013b] can be replaced with oneneural architecture. This approach should in theory reduce the cost of buildingand maintaining task-oriented systems and ease-up transfer learning betweendifferent domains.One family of such approaches treats dialogue as a source to target sequencetransduction problem, applying an encoder network [Bahdanau et al., 2014] toencode a user query into a distributed vector representing its semantics, whichthen conditions a decoder network to generate each system response. [Eric et al., 2017]proposed to combine attentions over the input sequence and the database.The output probabilities from both sources are combined to form the gener-ated answer. Such architectures do not break differentiability, however, thesemodels typically require a large amount of data to train [Bordes et al., 2017,Eric et al., 2017].[Wen et al., 2017b] proposed a neural network-based model for task-orienteddialogue systems by balancing the strengths and the weaknesses of the two re-search communities. The model is end-to-end trainable but modularly connec-ted; it does not directly model the user goal, but nevertheless, it still learns toaccomplish the required task by providing relevant and appropriate responsesat each turn. The model consists of a separate belief tracker, a database pointer2Figure 1: An architecture of the multi-domain dialogue system.which is based on results from the understanding, and policy and generationmodules.All of these models were created and evaluated on simple and single-domainoriented datasets. Most of dialogues have around 3 system turns making thismore similar to a question-answering set-up. Moreover, reinforcement learningwas used only as a fine-tuning phase to improve the performance per-turn level.3 Dialogue-Context-to-Text GenerationIn this section we focus on modelling the dialogue management and responsegeneration components. These problems can either be addressed separately [Young et al., 2013a],or jointly in an end-to-end fashion [Bordes et al., 2017, Wen et al., 2017b, Li et al., 2017].We experimented with a baseline neural response generation model with an or-acle belief-state obtained from the wizard annotations. This allows us to obtaina clear benchmark where the performance of the composite of dialogue manage-ment and response generation is completely independent of the belief tracking.Following [Wen et al., 2017b] which frames the dialogue as a context to responsemapping problem, a sequence-to-sequence model [Sutskever et al., 2014] is aug-mented with an oracle belief tracker and a discrete database accessing compon-ent as additional features to inform the word decisions in the decoder. Figure 1presents the architecture of the system.3.1 Model3.1.1 Intent modellingWe consider a dialogue that consists of T turns. Where it does not createconfusion, we denote by L a length of both user and system sentences per turn.The full ontology consists of D domains, each with Sd slots and Vd values. Wedifferentiate between a user output (u) and a machine (system) output (m). Ateach turn, the encoder takes a sequence of input tokens ut = (wt0, wt1, ..., wtL)and uses a (bi-directional) recurrent neural network to output a distributed user3utterance representation ut which is the final hidden state:ut = huL = RNNθ(ut).[Wen et al., 2017b] defines this as a distributed intent representation which re-places the hand-coded dialogue act representation.The summarised belief state is formed by going through the user belief state.For each slot, we encode whether the slot was not mentioned until this turn,some value has been given or the user does not care about this constraint. Thisgives 3-bin one-hot encoding for each slot. The global belief state vector is thenformed by concatenating all slot-dependent vectors over all domains:bt =⊕d⊕sbsd .Given an oracle belief state bt, we can use this information to query thedatabase for all the domains. Based on the numbers of entities in the databasethat satisfy the query, we form a 6-bin one-hot encodings for each domain kbd.The vector indicates if 0, 1, 2, 3, 4 or more entities were matched by the query.All domain-dependent vectors are then concatenated:kbt =⊕dkbdto form a global domain vector.3.1.2 Decision makingIn the next step, a policy vector is created to mimic the dialogue manager inthe traditional modular approach. The intent vector ut, the belief state vectorbt and the knowledge database vector kbt are combined together and processedthrough a nonlinear layer [Wen et al., 2017b]:at = tanh(Wuut +Wkkbt +Wbbt).This vector can be seen as a continuous version of a system act in the tradi-tional modular approach summarizing the current state and action in a high-dimensional space.3.2 GenerationThe generation module uses a language model in the form of the recurrent neuralnetwork that outputs probabilities over the vocabulary set at each time step:P (wj+1|wj ,hmj−1) = softmax(RNN(wj ,hmj−1)),where wj is the last output token and hmt−1 is the hidden vector from the previousstep. We condition a language generator through the action vector at by usingit as the first hidden vector, i.e:hm0 = at.To begin the generation process we use a special token (signifying the beginningof a sentence, SOS) as w0. The generation process stops when the networkoutputs a special token informing about the end of the sentence (EOS).4The standard cross entropy is adopted as our objective function to train alanguage model:L(θ) =∑t∑jytj log ptj ,where ytj and ptj are output token targets and predictions respectively, at turnt of output step j. In our case every token is treated equally to make the modelas simple and as general as possible.3.2.1 Attention mechanismThe signal from the user sentence can be amplified through the attention mech-anism that enable focusing on the most relevant part of the utterance at eachstep. After each pass of the RNN over the user sentence the hidden vectors foreach word are stored as:hu = (hu1 ,hu2 , ...,huL).These are combined through a score function with the current hidden vector inthe generation module hmj−1:aij = score(hu,hmj−1) = vT tanh(WT (hui ⊕ hmj−1) + b)and passed through a softmax operator to produce the attention weights:αij =eaij∑Li=1 eaij.The current attentive hidden vector (the context vector) from the user sentenceis created by weighting each hidden output with the attention weights:huc =L∑i=1αijhui .The context vector is concatenated with the previous generated embedded wordwnewj−1 = huc ⊕wj−1to output the prediction for the next word:P (wj+1|wj ,hmj−1,hu) = softmax(RNN(wnewj−1 ,hmj−1)).3.3 Training set-upAll models were trained for 20 full passes over the training dataset. The best setof hyper-parameters was found by a grid search, evaluating on the held-out val-idation subset of the date. We varied between size of embedding vectors, hiddenrepresentation, l2-norm penalty and the size of the dictionary. The gradients areclipped by 5.0 throughout the whole training process. Since often times the eval-uation of a dialogue system without a direct interaction with the real users canbe misleading [Liu et al., 2016], three different automatic metrics are includedto ensure the result is better interpreted. Among them, the first two metricsrelate to the dialogue task completion - whether the system has provided anappropriate entity (Inform rate) and then answered all the requested attributes(Success rate); while fluency is measured via BLEU score [Papineni et al., 2002].5Cam676 MultiWOZw/o attention w/ attention w/o attention w/ attentionInform (%) 99.17 99.58 71.29 71.33Success (%) 75.08 73.75 60.29 60.96BLEU 0.219 0.204 0.188 0.189Table 1: Performance comparison of two different model architectures using acorpus-based evaluation.3.4 EvaluationNeural approaches to statistical dialogue development, especially in a task-oriented paradigm, are greatly hindered by the lack of large scale datasets. Thatis why, following theWizard-of-Oz (WOZ) approach [Kelley, 1984, Wen et al., 2017b],we ran text-based multi-domain corpus data collection scheme through AmazonMTurk. The WOZ paradigm allowed us to obtain natural and semantically richmulti-topic dialogues spanning over multiple domains such as hotels, attrac-tions, restaurants, booking trains or taxis. The dialogues cover from 1 up to 5domains per dialogue greatly varying in length and complexity. We gatheredaround 10400 multi-domain dialogues of an average length and complexity sub-stantially higher than in previous corpora. As expected, the model achievesalmost perfect score on the Inform metric on the Cam676 dataset taking theadvantage of an oracle belief state signal. However, even with the perfect dia-logue state tracking of the user intent, the baseline models obtain almost 30%lower score on the Inform metric on the new corpus. The addition of the atten-tion improves the score on the Success metric on the new dataset by less than1%. Nevertheless, as expected, the best model on MultiWOZ is still fallingbehind by a large margin in comparison to the results on the Cam676 corpustaking into account both Inform and Success metrics. As most of dialogues spanover at least two domains, the model has to be much more effective in order toexecute a successful dialogue. Moreover, the BLEU score on the MultiWOZ islower than the one reported on the Cam676 dataset. This is mainly caused bythe much more diverse linguistic expressions observed in the MultiWOZ dataset.4 Future workRecently, [Wen et al., 2017a] employed a discrete latent variable to learn under-lying dialogue intentions in the framework of neural variational inference. In agoal-oriented dialogue scenario, these latent intentions can be interpreted as ac-tions guiding the generation of machine responses, which can be further refinedautonomously by reinforcement learning. We plan to employ models presentedin the previous section in order to create a model that is easier to learn anddoes not suffer from the disconnection phenomenon [Wen et al., 2017a]. Therecent advances allows for creating only one loss that the model is optimizedover contrary to previous approaches.Moreover, the recently acquired dataset is annotated with system dialogueacts which enables pre-training in a supervised way [Su et al., 2017]. Initialanalysis shows that over 55% of the annotated turns have multiple actions.These pose theoretical challenges requiring adapting standard RL framework toparallel action-choice [Harmer et al., 2018].6Finally, we are planning to incorporate the fine-tuning phase with dialogue-level RL loss. Previous methods used turn-level RL loss which results in anover-informative dialogue agent that does not learn multi-turn policy.5 ConclusionIn this report, we have examined end-to-end dialogue modelling with neuralarchitectures. Recently proposed methods were analyzed in the context ofmulti-domain dialogue systems. We created a multi-domain dialogue systemwith multi-domain belief state, database pointer and attention component overdecoded space. The initial model was evaluated on the recently collected multi-domain dataset MultiWOZ. The initial framework is falling behind the per-formance of the model in the single-domain set-up. Multi-domain dialogues arelonger often including two domains in one turn requiring better architecturesthat will allow for producing richer system outputs.7References[Bahdanau et al., 2014] Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neuralmachine translation by jointly learning to align and translate. ICLR.[Bordes et al., 2017] Bordes, A., Boureau, Y.-L., and Weston, J. (2017). Learn-ing end-to-end goal-oriented dialog. Proceedings of ICLR.[Cho et al., 2014] Cho, K., Gulcehre, B. v. M. C., Bahdanau, D., Schwenk,F. B. H., and Bengio, Y. (2014). Learning phrase representations usingrnn encoder–decoder for statistical machine translation. arXiv preprintarXiv:1406.1078.[Eric et al., 2017] Eric, M., Krishnan, L., Charette, F., and Manning, C. D.(2017). Key-value retrieval networks for task-oriented dialogue. In Proceedingsof the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 37–49.[Harmer et al., 2018] Harmer, J., Gisslén, L., Holst, H., Bergdahl, J., Olsson,T., Sjöö, K., and Nordin, M. (2018). Imitation learning with concurrentactions in 3d games. CoRR, abs/1803.05402.[Kelley, 1984] Kelley, J. F. (1984). An iterative design methodology for user-friendly natural language office information applications. ACM Transactionson Information Systems (TOIS), 2(1):26–41.[Li et al., 2017] Li, X., Chen, Y.-N., Li, L., Gao, J., and Celikyilmaz, A.(2017). End-to-end task-completion neural dialogue systems. In Proceedingsof the Eighth International Joint Conference on Natural Language Processing(Volume 1: Long Papers), volume 1, pages 733–743.[Liu et al., 2016] Liu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin,L., and Pineau, J. (2016). How not to evaluate your dialogue system: Anempirical study of unsupervised evaluation metrics for dialogue response gen-eration. In Proceedings of the 2016 Conference on Empirical Methods inNatural Language Processing, pages 2122–2132.[Papineni et al., 2002] Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J.(2002). Bleu: a method for automatic evaluation of machine translation.In Proceedings of the 40th annual meeting on association for computationallinguistics, pages 311–318. Association for Computational Linguistics.[Serban et al., 2015] Serban, I. V., Sordoni, A., Bengio, Y., Courville, A., andPineau, J. (2015). Hierarchical neural network generative models for moviedialogues. arXiv preprint arXiv:1507.04808.[Su et al., 2017] Su, P.-H., Budzianowski, P., Ultes, S., Gašić, M., and Young,S. J. (2017). Sample-efficient actor-critic reinforcement learning with super-vised data for dialogue management. In Proceedings of the SIGDIAL 2017Conference.[Sutskever et al., 2014] Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Se-quence to sequence learning with neural networks. In Advances in neuralinformation processing systems, pages 3104–3112.8[Wen et al., 2017a] Wen, T.-H., Miao, Y., Blunsom, P., and Young, S. (2017a).Latent intention dialogue models. In International Conference on MachineLearning, pages 3732–3741.[Wen et al., 2017b] Wen, T.-H., Vandyke, D., Mrksic, N., Gašić, M., Rojas-Barahona, L. M., Su, P.-H., Ultes, S., and Young, S. (2017b). A network-based end-to-end trainable task-oriented dialogue system. EACL.[Young et al., 2013a] Young, S., Gašić, M., Thomson, B., and Williams, J.(2013a). POMDP-based Statistical Spoken Dialogue Systems: a Review. InProc of IEEE, volume 99, pages 1–20.[Young et al., 2013b] Young, S., Gašić, M., Thomson, B., and Williams, J. D.(2013b). Pomdp-based statistical spoken dialog systems: A review. Proceed-ings of the IEEE, 101(5):1160–1179.[Young, 2002] Young, S. J. (2002). Talking to machines (statistically speaking).9

Towards end-to-end multi-domain dialogue modelling

https://www.repository.cam.ac.uk/bitstream/handle/1810/284093/Cambridge_Report%282%29.pdf?sequence=1&isAllowed=y

Towards end-to-end multi-domain dialogue modelling

Abstract

Similar works

Full text

Available Versions

Apollo (Cambridge)