We evaluate a semantic parser based on a character-based sequence-to-sequence
model in the context of the SemEval-2017 shared task on semantic parsing for
AMRs. With data augmentation, super characters, and POS-tagging we gain major
improvements in performance compared to a baseline character-level model.
Although we improve on previous character-based neural semantic parsing models,
the overall accuracy is still lower than a state-of-the-art AMR parser. An
ensemble combining our neural semantic parser with an existing, traditional
parser, yields a small gain in performance.Comment: To appear in Proceedings of SemEval, 2017 (camera-ready

Bos, Johan

van Noord, Rik

English

arXiv

Bos, Johannes

ARTS repository - University of Groningen

   University of GroningenThe Meaning Factory at SemEval-2017 Task 9: Producing AMRs with Neural SemanticParsingvan Noord, Rik; Bos, JohannesPublished in:Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)IMPORTANT NOTE: You are advised to consult the publisher's version (publisher's PDF) if you wish to cite fromit. Please check the document version below.Document VersionPublisher's PDF, also known as Version of recordPublication date:2017Link to publication in University of Groningen/UMCG research databaseCitation for published version (APA):van Noord, R., & Bos, J. (2017). The Meaning Factory at SemEval-2017 Task 9: Producing AMRs withNeural Semantic Parsing. In Proceedings of the 11th International Workshop on Semantic Evaluation(SemEval-2017) (Vol. 11, pp. 929-933). Association for Computational Linguistics (ACL).CopyrightOther than for strictly personal use, it is not permitted to download or to forward/distribute the text or part of it without the consent of theauthor(s) and/or copyright holder(s), unless the work is under an open content license (like Creative Commons).The publication may also be distributed here under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license.More information can be found on the University of Groningen website: https://www.rug.nl/library/open-access/self-archiving-pure/taverne-amendment.Take-down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.Downloaded from the University of Groningen/UMCG research database (Pure): http://www.rug.nl/research/portal. For technical reasons thenumber of authors shown on this cover page is limited to 10 maximum.Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 929–933,Vancouver, Canada, August 3 - 4, 2017. c©2017 Association for Computational LinguisticsThe Meaning Factory at SemEval-2017 Task 9: Producing AMRs withNeural Semantic ParsingRik van NoordCLCGUniversity of Groningenr.i.k.van.noord@rug.nlJohan BosCLCGUniversity of Groningenjohan.bos@rug.nlAbstractWe evaluate a semantic parser based ona character-based sequence-to-sequencemodel in the context of the SemEval-2017 shared task on semantic parsingfor AMRs. With data augmentation,super characters, and POS-tagging wegain major improvements in performancecompared to a baseline character-levelmodel. Although we improve on previ-ous character-based neural semantic pars-ing models, the overall accuracy is stilllower than a state-of-the-art AMR parser.An ensemble combining our neural se-mantic parser with an existing, traditionalparser, yields a small gain in performance.1 IntroductionTraditional open-domain semantic parsers oftenuse statistical syntactic parsers to derive syntac-tic structure on which to build a meaning repre-sentation. Recently there have been interestingattempts to view semantic parsing as a transla-tion task, mapping a source language (here: En-glish) to a target language (a logical form of somekind). Dong and Lapata (2016) used sequence-to-sequence and sequence-to-tree neural translationmodels to produce logical forms from sentences,while Barzdins and Gosko (2016) and Peng et al.(2017) used a similar method to produce AMRs.From a purely engineering point of view, these areinteresting attempts as complex models of the se-mantic parsing process can be avoided. Yet littleis known about the performance and fine-tuning ofsuch parsers, and whether they can reach perfor-mance of traditional semantic parsers, or whetherthey could contribute to performance in an ensem-ble setting.In the context of SemEval-2017 Task 9 we aimto shed more light on these questions. In particularwe participated in Subtask 1, Parsing BiomedicalData, and work with parallel English-AMR train-ing data comprising extracts of scientific articlesabout cancer pathway discovery.More specifically, our objectives are (1) tryto reproduce the results of Barzdins and Gosko(2016), who used character-level models for neu-ral semantic parsing; (2) improve on their re-sults by employing several novel techniques; and(3) combine the resulting neural semantic parserwith an existing off-the-shelf AMR parser to reachstate-of-the-art results.2 Neural Semantic Parsing2.1 DatasetsOur training set consists of the second LDC AMRrelease (LDC2016E25) containing 39,620 AMRs,as well as the training set of the bio AMR corpusthat contains 5,452 AMRs. As development andtest set we use the designated development andtest partition of the bio AMR corpus, both con-taining 500 AMRs. HTML-tags are removed fromthe sentences.2.2 Basic Translation ModelWe use a seq2seq neural translation model totranslate English sentences into AMRs. This is abi-LSTM model with added attention mechanism,as described in Bahdanau et al. (2014). Similar toBarzdins and Gosko (2016), but contrasting withPeng et al. (2017), we train the model only oncharacter-level input. Model specifics are shownin Table 1.In a preprocessing step, we remove all vari-ables from the AMR and duplicate co-referringnodes. An example of this is shown in Figure1. The variables and co-referring nodes are re-stored after testing, using the restoring script from929Parameter ValueLayers 1Nodes 400Buckets (510,510)Epochs 25–35Vocabulary 150–200Learning rate 0.5Decay factor 0.99Gradient norm 5Table 1: Model specifics for the seq2seq model.Barzdins and Gosko (2016).1 Wikipedia links arealso removed from the training set, but get restoredin a separate Wikification post-processing step.(require-01:ARG0 (induce-01:ARG1 (cell):ARG2 (migrate-01:ARG0 cell)):ARG1 (bind-01:ARG1 (protein:name (name :op1 "Crk")):ARG2 (protein:name (name :op1 "CAS"))))Figure 1: “Crk binding to CAS is required for theinduction of cell migration” - seq2seq tree repre-sentation.2.3 ImprovementsIn this section we describe the methods used toimprove the neural semantic parser.Augmentation AMRs, as introduced by Ba-narescu et al. (2013), are rooted, directed, labeledgraphs, in which the different nodes and triples areunordered by definition. However, in our tree rep-resentation of AMRs (see Figure 1), there is anorder of branches. This means that we are able topermute this order into a more intuitive represen-tation of the sentence, by matching the word orderusing the AMR-sentence alignments. An exampleof this method is shown in Figure 2.This approach can also be used to augmentthe training data, since we are now able generate“new” AMR-sentence pairs that can be added toour training set. However, due to the exponentialgrowth, there are often more than 1,000 differentAMR permutations for long sentences. We ranmultiple experiments to find the best way to usethis oversupply of data. Ultimately, we found that1https://github.com/didzis/tensorflowAMR(require-01:ARG1 (bind-01:ARG1 (protein:name (name :op1 "Crk")):ARG2 (protein:name (name :op1 "CAS")))):ARG0 (induce-01:ARG1 (cell):ARG2 (migrate-01:ARG0 cell))Figure 2: “Crk binding to CAS is required for theinduction of cell migration” - seq2seq representa-tion that best matches the word order.it is most beneficial to “double” the training databy adding the best matching AMR (based on wordorder) to the existing data set.Super characters We do not necessarily have torestrict ourselves to using only individual charac-ters as input. For example, the AMR relations (e.g.:ARG0, :domain, :mod) can be seen as singleentities instead of a collection of characters. Thisdecreases the input length of the AMRs in featurespace, but increases the total vocabulary. We referto these entities as super characters. This way, weessentially create a model that is a combination ofcharacter and word level input.POS-tagging Character-level models might stillbe able to benefit from syntactic information, evenwhen this is added directly to the input structure.Especially POS-tags can easily be added as fea-tures to the input data, while also providing valu-able information. For example, proper nouns ina sentence often occur with the :name relationin the corresponding AMR, while adjectives cor-relate with the :mod relation. We append thecorresponding POS-tag to each word in the sen-tence (using the C&C POS-tagger by Clark et al.(2003)), creating a new super character for eachunique tag.Post-processing In a post-processing step, firstthe variables and co-referring nodes are restored.We try to fix invalidly produced AMRs by apply-ing a few simple heuristics, such as inserting spe-cial characters (e.g. parentheses, quotes) or re-moving unfinished edges. If the AMR is still in-valid, we output a smart default AMR.2We also remove all double nodes, i.e., relation-concept pairs that occur more than once in abranch of the AMR. This form of duplicate outputis a common problem with deep learning models,2This was not necessary for the evaluation data.930since the model does not keep track of what it hasalready output. We refer to this method as prun-ing.Wikification Our Wikification method is basedon Bjerva et al. (2016), using Spotlight (Daiberet al., 2013). They initially removed wiki linksfrom the input and then tried to restore them in theoutput by simply adding a wiki link to the AMRif it matches with the name in a :name relation.Even though this approach worked well for theLDC data, it did not work for the biomedical data.This is mainly due to the fact that :name nodesare not consistently annotated with a wiki linkin the gold biomedical data. 138 unique namesthat had a corresponding wiki link at least oncein the gold data did not have this wiki link 100%of the time. For example DNA occurred 86 timesas a :name in the gold data and was only anno-tated with a wiki link in 69 cases, while ERK oc-curred 228 times with only 3 annotated wiki links.For this reason we opt for a safe Wikification ap-proach: we only add wiki links to names that wereannotated with the same wiki link more than 50%of the time in the gold data. Following our previ-ous example, this means that DNA does still get awiki link, but ERK does not.2.4 CAMR ensembleAs we know that our neural semantic parser is un-likely to outperform a state-of-the-art AMR parser,but is likely to complement it, our strategy is to usean ensemble-based approach. The ensemble com-prises the off-the-shelf parser CAMR (Wang et al.,2015) and our neural semantic parser. The imple-mentation of this ensemble is similar to Barzdinsand Gosko (2016), choosing the AMR that ob-tains the highest pairwise Smatch (Cai and Knight,2013) score when compared to the other AMRsgenerated for a sentence. This method is de-signed to ultimately choose the AMR with themost prevalent relations and concepts.We train CAMR models based on the biomedi-cal data only, the LDC data only and the combina-tion of both data sets. Since CAMR is nondeter-ministic, we can also train multiple models on thesame data set. Ultimately, the best ensemble onthe test data consisted of three bio-only models,two bio + LDC models and one LDC-only model.This ensemble was used to parse the evaluationset.3 Results and Discussion3.1 Results on Test SetTable 2 shows the results of all improvement meth-ods tested in isolation on the test set of the biomed-ical data. Augmenting the data only helps veryslightly, while the super characters are responsi-ble for the largest increase in performance. Thisshows that we do not necessarily have to use onlycharacter or word level input in our models, butthat a combination of the two might be optimal.The best result was obtained by combining the dif-ferent methods. This model was then used to parsethe evaluation data. Table 3 shows the results ofretraining CAMR on different data sets, as well asan ensemble of those models. Adding our seq2seqmodel to the ensemble only yielded a very smallgain in performance.Feature F-score IncreaseBaseline 0.422Pruning 0.425 0.7%Wikification 0.423 0.2%Augmentation 0.424 0.5%Super characters 0.481 14.0%POS-tagging 0.436 3.3%All combined 0.504 19.4%Table 2: Results of the different seq2seq modelson the test set of the biomedical data.F-scoreCAMR retrained on LDC 0.399CAMR retrained on bio 0.585CAMR retrained on LDC + bio 0.582Ensemble CAMR 0.588Ensemble CAMR + seq2seq 0.589Table 3: Results of retraining CAMR and resultsof best ensemble models, tested on the biomedicaltest data.3.2 Official ResultsIn Table 4 we see the detailed results of the bestseq2seq model and best ensemble on the evalua-tion data, using the scripts from Damonte et al.(2017).3 While CAMR has similar scores on the3Unofficial score for seq2seq negation; due to a mistake,all :polarity nodes were removed in the official submis-sion. This had no influence on the final F-score.931test data, the score of the seq2seq model decreasesby 0.04. It is interesting to note that seq2seq scoresequally well without word sense disambiguation,while there is no separate module that handles this.Setting seq2seq EnsembleSmatch 0.460 0.576Unlabeled 0.504 0.623No WSD 0.463 0.579Named Entities 0.512 0.576Wikification 0.458 0.396Negation 0.141 0.244Concepts 0.630 0.759Reentrancies 0.290 0.352SRL 0.427 0.543Table 4: Official results on the evaluation set forboth the ensemble and the seq-to-seq neural se-mantic parser.Figure 3: Comparison of CAMR and our seq-to-seq model for different sentence lengths.3.3 Comparison with CAMRAlthough CAMR outperformed our neural seman-tic parser by a large margin, the seq-to-seq modeldid produce a better AMR for 108 out of the 500evaluation AMRs, based on Smatch score. If theCAMR + seq2seq ensemble was somehow able toalways choose the best AMR, it obtains an F-scoreof 0.601, an increase of 2.2% instead of the cur-rent 0.2%. This suggests that the current methodof combining neural semantic parsers with exist-ing parsers is far from optimal, but that the neuralmethods do provide complementary information.A different way to incorporate this informationwould be to pick the most suitable parser basedon the input sentence. A classifier that exploits thecharacteristics of the sentence could be trained toassign a parser to each individual (to be parsed)sentence.Figure 3 shows the performance of the neu-ral semantic parser and the CAMR ensemble permaximum sentence length. We see that seq-to-seqcan keep up with CAMR for very short sentences,but is clearly outperformed on longer sentences.As the sentences get longer, the difference in per-formance gets bigger, but not much.4 Conclusion and Future WorkWe were able to reproduce the results of thecharacter-level models for neural semantic pars-ing as proposed by Barzdins and Gosko (2016).Moreover, we showed improvement on their ba-sic setting by using data-augmentation, part-of-speech as additional input, and using super charac-ters. The latter setting showed that a combinationof character and word level input might be opti-mal for neural semantic parsers. Despite these en-hancements, the resulting AMR parser is still out-performed by more traditional, off-the-shelf AMRparsers. Adding our neural semantic parser to anensemble including CAMR (Wang et al., 2015), adependency-based parser, yielded no noteworthyimprovements on the overall performance.Do these results indicate that neural semanticparsers will never be competitive with more tra-ditional statistical parsers? We don’t think so. Wehave the feeling that we have just scratched thesurface of possibilities that neural semantic pars-ing can offer us, and how they possibly can com-plement parsers using different strategies. In fu-ture work we will explore these.AcknowledgementsWe thank the two anonymous reviewers for theircomments. We would also like to thank the Centerfor Information Technology of the University ofGroningen for their support and for providing ac-cess to the Peregrine high performance computingcluster. We also used a Tesla K40 GPU, which waskindly donated by the NVIDIA Corporation. Thiswork was funded by the NWO-VICI grant “Lost inTranslation – Found in Meaning” (288-89-003).932ReferencesDzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben-gio. 2014. Neural machine translation by jointlylearning to align and translate. arXiv preprintarXiv:1409.0473 .Laura Banarescu, Claire Bonial, Shu Cai, MadalinaGeorgescu, Kira Griffitt, Ulf Hermjakob, KevinKnight, Philipp Koehn, Martha Palmer, and NathanSchneider. 2013. Abstract meaning representa-tion for sembanking. In Proceedings of the7th Linguistic Annotation Workshop and Interoper-ability with Discourse. Association for Computa-tional Linguistics, Sofia, Bulgaria, pages 178–186.http://www.aclweb.org/anthology/W13-2322.Guntis Barzdins and Didzis Gosko. 2016. Rigaat semeval-2016 task 8: Impact of smatch ex-tensions and character-level neural translation onamr parsing accuracy. In Proceedings of the10th International Workshop on Semantic Evalu-ation (SemEval-2016). Association for Computa-tional Linguistics, San Diego, California, pages1143–1147. http://www.aclweb.org/anthology/S16-1176.Johannes Bjerva, Johan Bos, and Hessel Haagsma.2016. The meaning factory at semeval-2016 task8: Producing amrs with boxer. In Proceedingsof the 10th International Workshop on SemanticEvaluation (SemEval-2016). Association for Com-putational Linguistics, San Diego, California, pages1179–1184. http://www.aclweb.org/anthology/S16-1182.Shu Cai and Kevin Knight. 2013. Smatch: anevaluation metric for semantic feature structures.In Proceedings of the 51st Annual Meeting ofthe Association for Computational Linguistics (Vol-ume 2: Short Papers). Association for Computa-tional Linguistics, Sofia, Bulgaria, pages 748–752.http://www.aclweb.org/anthology/P13-2131.Stephen Clark, James R Curran, and Miles Os-borne. 2003. Bootstrapping pos taggers us-ing unlabelled data. In Proceedings of theseventh conference on Natural language learn-ing at HLT-NAACL 2003-Volume 4. Associa-tion for Computational Linguistics, pages 49–55.http://www.aclweb.org/anthology/W03-0407.Joachim Daiber, Max Jakob, Chris Hokamp, andPablo N. Mendes. 2013. Improving efficiency andaccuracy in multilingual entity extraction. In Pro-ceedings of the 9th International Conference on Se-mantic Systems (I-Semantics).Marco Damonte, Shay B. Cohen, and Giorgio Satta.2017. An incremental parser for abstract mean-ing representation. In Proceedings of the 15thConference of the European Chapter of the As-sociation for Computational Linguistics: Vol-ume 1, Long Papers. Association for Computa-tional Linguistics, Valencia, Spain, pages 536–546.http://www.aclweb.org/anthology/E17-1051.Li Dong and Mirella Lapata. 2016. Languageto logical form with neural attention. In Pro-ceedings of the 54th Annual Meeting of theAssociation for Computational Linguistics (Vol-ume 1: Long Papers). Association for Computa-tional Linguistics, Berlin, Germany, pages 33–43.http://www.aclweb.org/anthology/P16-1004.Xiaochang Peng, Chuan Wang, Daniel Gildea, andNianwen Xue. 2017. Addressing the data spar-sity issue in neural amr parsing. In Proceedingsof the 15th Conference of the European Chapterof the Association for Computational Linguistics:Volume 1, Long Papers. Association for Computa-tional Linguistics, Valencia, Spain, pages 366–375.http://www.aclweb.org/anthology/E17-1035.Chuan Wang, Nianwen Xue, and Sameer Pradhan.2015. A transition-based algorithm for amr pars-ing. In Proceedings of the 2015 Conferenceof the North American Chapter of the Associa-tion for Computational Linguistics: Human Lan-guage Technologies. Association for ComputationalLinguistics, Denver, Colorado, pages 366–375.http://www.aclweb.org/anthology/N15-1040.933

The Meaning Factory at SemEval-2017 Task 9: Producing AMRs with Neural Semantic Parsing

Rik van Noord

Johan Bos

Crossref

The Meaning Factory at SemEval-2017 Task 9: Producing AMRs with
            Neural Semantic Parsing

University of Groningen

University of Groningen Research Database

https://pure.rug.nl/ws/files/48621735/Noord_Bos_AMR_SemEval_2017.pdf

The Meaning Factory at SemEval-2017 Task 9: Producing AMRs with Neural
  Semantic Parsing

The Meaning Factory at SemEval-2017 Task 9: Producing AMRs with Neural Semantic Parsing

Abstract

Similar works

Full text

Available Versions

ARTS repository - University of Groningen

Crossref

ARTS repository - University of Groningen

University of Groningen

University of Groningen Research Database