We showcase TODAY, a semanticsenhanced task-oriented dialogue translation system, whose novelties are: (i) taskoriented named entity (NE) definition and

a hybrid strategy for NE recognition and

translation; and (ii) a novel grounded semantic method for dialogue understanding

and task-order management. TODAY is a

case-study demo which can efficiently and

accurately assist customers and agents in

different languages to reach an agreement

in a dialogue for the hotel booking

Du, Jinhua

Li, Liangyou

Liu, Qun

Tu, Zhaopeng

Wang, Longyue

Way, Andy

We showcase TODAY, a semanticsenhanced task-oriented dialogue translation system, whose novelties are: (i) taskoriented named entity (NE) definition and
a hybrid strategy for NE recognition and
translation; and (ii) a novel grounded semantic method for dialogue understanding
and task-order management. TODAY is a
case-study demo which can efficiently and
accurately assist customers and agents in
different languages to reach an agreement
in a dialogue for the hotel booking

Irish Universities

Semantics-enhanced task-oriented dialogue translation:
a case Study on hotel booking

DCU Online Research Access Service

The Companion Volume of the IJCNLP 2017 Proceedings: System Demonstrations, pages 33–36,Taipei, Taiwan, November 27 – December 1, 2017. c©2017 AFNLPSemantics-Enhanced Task-Oriented Dialogue Translation:A Case Study on Hotel BookingLongyue Wang† Jinhua Du† Liangyou Li§ Zhaopeng Tu‡∗ Andy Way† Qun Liu††ADAPT Centre, School of Computing, Dublin City University, Ireland{longyue.wang, jinhua.du, andy.way, qun.liu}@adaptcentre.ie§Noah’s Ark Lab, Huawei Technologies, China ‡Tencent AI Lab, Chinaliliangyou@huawei.com zptu@@tencent.comAbstractWe showcase TODAY, a semantics-enhanced task-oriented dialogue transla-tion system, whose novelties are: (i) task-oriented named entity (NE) definition anda hybrid strategy for NE recognition andtranslation; and (ii) a novel grounded se-mantic method for dialogue understandingand task-order management. TODAY is acase-study demo which can efficiently andaccurately assist customers and agents indifferent languages to reach an agreementin a dialogue for the hotel booking.1 IntroductionApplications of machine translation (MT) in somehuman–human communication scenarios still existmany challenging problems due to the characteris-tics of spoken languages and dialogues. For exam-ple, general-purpose MT systems cannot performefficiently and effectively on specific tasks such ashotel booking because of the low accuracy of entityrecognition and translation in dialogues betweencustomers and hotel agents as shown below:Source: 我想定个房间，{十二月二十五号} (星期二) [三点]入住。Reference: I would like to reserve a room on {Decemberthe 25th}, (Tuesday) and I will check in at[three o‘clock].Google: I’d like to have a room, [three] on (Tuesday),{February 25}.App1: I want to book a room, (Two) or [Three]rooms at the {December 25} week.In this example, App1 is a commercialised trans-lation system for the travel domain. We found thatcheck-in date/time and week day were not trans-lated correctly either by Google or App1. Wrong∗ Work was done when Zhaopeng Tu was working atHuawei Noah’s Ark Lab.translations of these entities will impede communi-cation between the customer and agent.We showcase our task-oriented semantics-enhanced dialogue machine translation (DMT) sys-tem TODAY1 which alleviates these problems forthe hotel booking scenario.2 System DescriptionIn the hotel booking scenario, customers and agentsspeak different languages.2 Customers access thehotel website to request a conversation, and theagent accepts the customer’s request to start theconversion. Figure 1 shows the detailed workflowof TODAY. We first recognise entities by inferringtheir specific types based on information such ascontexts, speakers etc. (cf. Section 2.1 and 2.2).Then, the recognised entities will be representedas logical expressions or semantic templates usingthe grounded semantics module (cf. Section 2.2).Finally, candidate translations of semantically rep-resented entities will be marked up and fed into aunified bi-directional translation process.2.1 Task-Oriented Named Entity Recognitionand TranslationAs standard types of entities (e.g. people, organi-zations, locations) cannot exactly match our task-oriented entity types, we define a series of task-oriented entity types in TODAY, including {time,number, date, currency, room type, person name,hotel name, location, payment type }. We combinerule-based and dictionary-based methods for ourNE recognition and translation. For bilingual dic-tionary construction, we employ the ICE toolkit.3ICE can guide users through a series of linguis-1The demo system can be found at http://computing.dcu.ie/˜lwang/demo.html.2The rest of the paper will assume that customers speakEnglish and agents speak Chinese.3Available at http://nlp.cs.nyu.edu/ice.33Figure 1: Workflow of TODAY.tic processing steps, presents them with entitiesand dependency relations that are potential seeds,and helps them to expand the seeds by answeringyes/no questions. Other types of NEs such as time,data etc. can be recognised and translated usingrules because of their formulaic and common ex-pressions.Our hybrid strategy can generates multiple trans-lation candidates. For example, the date have vari-ous formats in both English and Chinese. Then therecognised entities and their candidate translationsare formalise by XML markup in the source-sidesentence. Finally the sentence is fed into the de-coder to compete with the translation model.2.2 Grounded Semantic Representation andForm FillingWe propose a specific semantic module for our task-oriented dialogues: grounded semantics modulewhich is in the form of Feature-Value (FV) pairs.In TODAY, we define two types of features (Ta-ble 1): customer features and room features. Whilethe customer features include information aboutthe customer of the current order, room featuresdescribe the details of the order, including check-inand check-out date, room type, room price, andpayment information. All these features and theirvalues can be recognised by the NE recogniser (cf.Section 2.1) but extended by adding extra patterns.These new patterns are used to label an entity withType Feature Example valueCustomerName JohnTel. NO. 1234567RoomCheck-In Date 1st April 2017Check-Out Date 2nd April 2017Room Type SinglePrice Per. Night 300 DollarsPayment Type VISACard NO. 987654321Table 1: An example of a booking order whichrepresents the grounded semantic of a dialogue.more detailed categories, such as determining adate to be one of the two features: check-in dateand check-out date. To achieve this, patterns takecontexts into consideration. For example, accord-ing to the phrase from 1st Jan to 2nd Jan, 1st Jan isa check-in date while 2nd Jan is a check-out date.After recognising all features and their values,we need to solve conflicts when a feature appearsmultiple times with different values.Customer: I’d like to have a single room.Hotel: Sorry. I only have double rooms available.Customer: OK. A double room would also be fine.In this example, the feature room type appears threetimes with values single and double. To determinewhich value should be chosen, we propose to scoreeach candidate value of a feature by comparing itscontexts with predefined attributes of the feature.The candidate with the highest score is then takenas the final feature value.We define four attributes on each feature:• Speaker: Its value is either hotel or customer.For example, because the room type is usuallychosen by a customer, we define the speakerattribute of this feature to be “customer”.• Position: It defines a range of positions in dia-logues that a feature should be within. For exam-ple, we define the position attribute of a customername as [1–3], because a dialogue usually startsfrom self-introduction and greetings, but in otherfeatures we set the position to infinity.• Patterncur: It consists of a set of patterns thatusually appear in the current utterance. For ex-ample, the customer name usually follows I amor my name is.• Patternpre: It consists of a set of patterns thatusually appear in the previous utterance.Given these predefined attributes on a feature, wecalculate a score for each candidate value of the34feature, according to Equation (1):Score(v) = S · P · (λc +Mc) · (λp +Mp) (1)where S ∈ {0, 1} indicates whether the currentspeaker equals to the speaker defined in the featureor not, P ∈ {0, 1} denotes whether the position ofthe feature is within the range given by its positionattributes or not, and Mc and Mp are the numberof matched patterns in the current utterance andprevious utterance, respectively. We use λc = 1and λp = 1 as smooth factors.3 Experiments and Analysis3.1 SetupFrom the IWSLT DIALOG corpus,we select 1,023and 1053 hotel booking sentences (34/36 dia-logues) as development set and test set, respectively.We combine our home-made travel domain corporaas in-domain training data (180K). We also usedomain adaptation techniques to select in-domaindata from movie subtitles (Wang et al., 2016b).We carry out our experiments using the phrase-based SMT model in Moses (Koehn et al., 2007) onChinese (ZH)–English (EN). Furthermore, we traina 5-gram language model using the SRI LanguageToolkit (Stolcke, 2002). We run GIZA++ (Och andNey, 2003) for alignment and use MERT (Och,2003) to optimize the feature weights. We de-velop TODAY on the basis of an open-sourcelive support application Mibew4 by integrating oursemantics-enhanced SMT system and the semanticform filling.3.2 Evaluation of Dialogue TranslationWe first evaluate the domain adaptation and NEapproaches on DMT, respectively. Then, we com-bine these best sub-models to further improve thetranslation quality.The baseline systems are trained on the in-domain corpus and the results show that an MTsystem trained on small-scale data can only obtain24.20 and 17.90 BLEU points on English–Chineseand Chinese–English, respectively. Combining themodels trained on the selected pseudo in-domaindata can improve the performance by at most +1.09and +1.24 on EN-ZH (top-50K) and ZH-EN (top-50K), respectively. However, bring more pseudoin-domain data (> top− 250K), the performancedrops sharply.4Available at https://mibew.org.System EN-ZH 4 ZH-EN 4In-domain 24.20 - 17.90 -1-best Entity 30.70 +6.5 20.30 +2.4N-best Entity 31.10 +6.9 20.20 +2.3Table 2: Performance with task-oriented NE recog-nition.Task SYS BLEU (%)ZH-ENGoogle 10.3App1 10.4TODAY 21.5EN-ZHGoogle 16.9App1 15.5TODAY 32.7Table 3: Overall performance.About NE component, we employ XML markuptechnique to insert bilingual entities into the trans-lation. As the entity may have multiple translations,we also explore N-best entity lists. After insertingentities into the MT system, the performance im-proves by +6.5 (EN-ZH) and +2.4 (ZH-EN) BLEUpoints as shown in Table 2. When using the N-bestentity method, it can further improve the perfor-mance by +0.4 BLEU on English–Chinese.Based on the individual performance of eachcomponent, we design our DMT: 1) build transla-tion models on selected top-50K data and combineit with baseline; 2) integrate N-best NE models toour MT our system. In Table 3, it shows that combi-nation further improve the translation performance.Comparing with App1 and Google Translate, oursystem significantly outperforms these systems by+17.2 and +11.2 BLEU points on EN-ZH and ZH-EN, respectively.3.3 Evaluation of Task-Oriented NamedEntity and TranslationWe manually annotated Chinese and English sen-tences in the test set to evaluate the proposed task-oriented NE recognition and translation in termsof accuracy, recall and F1. In Table 4, Recog indi-cates NE recognition on the source language, andTrans indicates translation task. All F1 scores areover 90% in terms of recognition and translation,Lang Task Acc. (%) Rec. (%) F1 (%)ZH-EN Recog 98.21 99.76 98.99Trans 91.86 93.33 92.59EN-ZH Recog 97.78 96.04 96.90Trans 97.24 95.52 96.37Table 4: Results of NE recognition and translation.35Task SYS Trans. Acc. (%)ZH-ENGoogle 72.08App1 85.28TODAY 97.24EN-ZHGoogle 58.11App1 66.42TODAY 95.52Table 5: Comparison on entity translation withdifferent systems.Feature Precision (%)Customer Name 97.1Customer Tel. NO. 91.2Check-In Date 100.0Check-Out Date 76.5Room Type 73.5Price Per. Night 79.4Payment Type 100.0Card NO. 97.1Average 89.4Table 6: Performance of form-filling (EN).which shows that the proposed fine-grained NE def-initions and hybrid strategy for NE recognition andtranslation is effective in TODAY.We also compared TODAY with Google andApp1 as shown in Table 5. Since we cannot obtainNE recognition information from both third-partapplications, we manually inspected the top-300sentences (in test set) and only calculate accuracyof translations of entities. If the translation of anentity matches the reference, we count it as correct;otherwise, it is regarded as incorrect. It shows thatTODAY significantly outperforms both Google andApp1 in terms of accuracy of entity translation.3.4 Evaluation of Grounded SemanticExtractionWe tested our grounded semantic module on thetest set in terms of feature recognition and formfilling. Each dialogue is manually annotated withsemantic features and has an associated order formas a reference. Since our feature recognizer is asimple extension of the NE recogniser, in this sec-tion we ignore the performance of the recognizer.Table 6 shows evaluation results of the form-fillinggiven golden feature annotations.Five features (customer name, customer Tel.NO., check-in date, payment type, and Card NO.)achieve an accuracy over 90%. The reason of suchhigh accuracy we analyse is that there are fewerconflicts for them in a single dialogue. By con-trast, on the other 3 features, (check-out date, roomtype, and price per. night) the accuracy is between70%–80%. By inspecting the dialogues, we foundthat (i) the check-out date is not always explicitlymentioned in the dialogues; (ii) the room type andprice per. night have a relatively higher repetition.These observations suggest that it is harder to solvethe conflicts on these three features.4 Conclusion and Future WorkIn this paper we described TODAY, a semantics-enhanced task-oriented dialogue translation systemfor hotel booking scenarios and evaluated its perfor-mance. In future work, we plan to integrate neuralMT into our demo system based on our advancedapproaches (Wang et al., 2016a, 2017).AcknowledgmentsThis work is supported by the Science Foun-dation of Ireland (SFI) ADAPT project (GrantNo.:13/RC/2106), and partly supported by theDCU-Huawei Joint Project (Grant No.:201504032-A (DCU), YB2015090061 (Huawei)).ReferencesPhilipp Koehn, Hieu Hoang, Alexandra Birch, ChrisCallison-Burch, Marcello Federico, Nicola Bertoldi,Brooke Cowan, and et al. 2007. Moses: Opensource toolkit for statistical machine translation. InProceedings of the 45th ACL: Demo and Poster Ses-sions, pages 177–180.Franz Josef Och. 2003. Minimum error rate trainingin statistical machine translation. In Proceedings ofthe 41st ACL, pages 160–167.Franz Josef Och and Hermann Ney. 2003. A systematiccomparison of various statistical alignment models.Computational Linguistics, 29(1):19–51.Andreas Stolcke. 2002. SRILM – an extensible lan-guage modeling toolkit. In Proceedings of the 7thICSLP, pages 901–904.Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu.2017. Exploiting cross-sentence context for neuralmachine translation. In Proceedings of the 2017EMNLP, pages 2816–2821.Longyue Wang, Zhaopeng Tu, Xiaojun Zhang, HangLi, Andy Way, and Qun Liu. 2016a. A novel ap-proach for dropped pronoun translation. In Proceed-ings of the 2016 NAACL, pages 983–993.Longyue Wang, Xiaojun Zhang, Zhaopeng Tu, AndyWay, and Qun Liu. 2016b. The automatic construc-tion of discourse corpus for dialogue translation. InProceedings of the 10th LREC, pages 2748–2754.36

Semantics-enhanced task-oriented dialogue translation:

a case Study on hotel booking

http://doras.dcu.ie/23369/1/Semantics-Enhanced_Task-Oriented_Dialogue_Translation%5B1%5D.pdf

Semantics-enhanced task-oriented dialogue translation: a case Study on hotel booking

Abstract

Similar works

Full text

Available Versions

Irish Universities

DCU Online Research Access Service