Magdy, Walid

Meaney, Julie-Anne

Wilson, Steve R.

English

The use of pre-trained language models such as BERT and ULMFiT has become increasingly popular in shared tasks, due to their powerful language modelling capabilities. Our entry to SemEval uses ERNIE 2.0, a language model which is pre-trained on a large number of tasks to enrich the semantic and syntactic information learned. ERNIE’s knowledge masking pre-training task is a unique method for learning about named entities, and we hypothesise that it may be of use in a dataset which is built on news headlines and which contains many named entities. We optimize the hyperparameters in a regression and classification model and find that the hyperparameters we selected helped to make bigger gains in the classification model than the regression model

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Smash at SemEval-2020 Task 7: Optimizing the Hyperparametersof ERNIE 2.0 for Humor Ranking and RatingCitation for published version:Meaney, J-A, Wilson, SR & Magdy, W 2020, Smash at SemEval-2020 Task 7: Optimizing theHyperparameters of ERNIE 2.0 for Humor Ranking and Rating. in Proceedings of the Fourteenth Workshopon Semantic Evaluation. Association for Computational Linguistics, pp. 1049–1054, International Workshopon Semantic Evaluation 2020, Barcelona, Spain, 12/12/20.<https://www.aclweb.org/anthology/2020.semeval-1.137>Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Publisher's PDF, also known as Version of recordPublished In:Proceedings of the Fourteenth Workshop on Semantic EvaluationGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 23. Jul. 2021Proceedings of the 14th International Workshop on Semantic Evaluation, pages 1049–1054Barcelona, Spain (Online), December 12, 2020.1049Smash at SemEval-2020 Task 7: Optimizing the Hyperparameters ofERNIE 2.0 for Humor Ranking and RatingJ. A. Meaney Steven R. Wilson Walid MagdyInstitute for Language, Cognition and ComputationSchool of InformaticsUniversity of Edinburgh{jameaney, steven.wilson}@ed.ac.uk, wmagdy@inf.ed.ac.ukAbstractThe use of pre-trained language models such as BERT and ULMFiT has become increasinglypopular in shared tasks, due to their powerful language modelling capabilities. Our entry toSemEval uses ERNIE 2.0, a language model which is pre-trained on a large number of tasksto enrich the semantic and syntactic information learned. ERNIE’s knowledge masking pre-training task is a unique method for learning about named entities, and we hypothesise that itmay be of use in a dataset which is built on news headlines and which contains many namedentities. We optimize the hyperparameters in a regression and classification model and find thatthe hyperparameters we selected helped to make bigger gains in the classification model than theregression model.1 IntroductionVerbal humor uses a variety of linguistic features, such as synonymy, wordplay, and phonological sim-iliarities, as well non-linguistic features like world knowledge, to produce a comic effect. That such abroad set of skills are required to understand humor, has led several researchers to deem that computa-tional humor is an AI-complete problem (Stock and Strapparava, 2006; Binsted et al., 2006). There is arelatively longstanding body of research into humor detection in a limited domain, such as knock-knockjokes (Taylor and Mazlack, 2004), one-liners (Mihalcea and Strapparava, 2006) and humorous news ar-ticles from the satirical news publication The Onion (Mihalcea and Pulman, 2007). However, the use ofshared tasks has attracted more attention and interest in the field since 2017. While previous challengeshave focused on collecting Twitter data (Potash et al., 2017; Castro et al., 2018), SemEval 2020 (Hossainet al., 2020) took an original approach and generated the data by collecting news headlines and thenasking annotators to edit one word in the headline to make it humorous (Hossain et al., 2019). Theseheadlines emulate those of The Onion. The edits shown below indicate the location of the substitutionand the word to be inserted.Table 1: Example of Funlines HeadlineType Text Edit ScoreOriginalMitch McConnell thinks tax reform will take longerthan Trump claimedEdit 1Mitch McConnell thinks tax reform will take longerthan Trump <claimed/>haircut 2.8Edit 2Mitch McConnell thinks tax <reform/>will take longerthan Trump claimedreturn 1.6The edited headlines were then rated for humor by subsequent annotators. Sub-task A was to predictthe mean funniness score of the edited headline. In sub-task B, the systems saw two edits of the sameheadline, and predicted which one had achieved the higher mean funniness score.This work is licensed under a Creative Commons Attribution 4.0 International Licence.Licence details: http://creativecommons.org/licenses/by/4.0/.10502 Previous WorkExcluding work on puns, there have been three humor detection shared tasks in recent years: Semeval2017 (Potash et al., 2017), HAHA 2018 (Castro et al., 2018) and HAHA 2019 (Chiruzzo et al., 2019). Asthe tasks and data have varied between them, direct comparison is not possible. However, a comparisonof approaches to the tasks shows some interesting trends.Semeval 2017’s entries were evenly divided between feature engineering approaches and deep learningsystems, with both achieving competitive results. The highest ranking team in the official results for taskA, SVNIT (Mahajan and Zaveri, 2017), used an SVM with incongruity, ambiguity and stylistic features.The second highest-ranking team, Datastories (Baziotis et al., 2017) opted for a Siamese bi-LSTM withattention. Interestingly, a remarkably simple system prevailed in task B: Duluth (Yan and Pedersen,2017) used the probability assigned to the text by a bigram language model instead of the output of aclassifier to make predictions.Entries to HAHA 2018 were divided along similar lines. The winning system used Naive Bayes andridge regression models optimized with an evolutionary algorithm (Ortiz-Bejar et al., 2018) with therunner up using a bi-LSTM with attention (Ortega-Bueno et al., 2018).HAHA 2019 saw a sea change towards the use of transfer learning models, such as BERT (Devlinet al., 2018) and ULMFiT (Howard and Ruder, 2018). These models leverage large amounts of dataand transformer attention models to learn contextual relations between words. Adilism (Ismailov, 2019)used multilingual BERT base uncased and extended the language model training without labels, beforefinetuning their system with the dataset labels. The second place system used an ensemble of a BERTmodel and ULMFiT, with Naive Bayes and SVM classifiers. The majority of the top entries to this taskused BERT in some way, although one noted that it did not improve performance as expected (Ortega-Bueno et al., 2019).3 System Overview3.1 Why ERNIE 2.0?As BERT models are trained on a masked-language model and sentence prediction task, they capturemainly word-level and sentence-level information. By comparison, ERNIE 2.0 (Sun et al., 2020) - hence-forth ERNIE - aims to capture more lexical, syntactic and semantic information in corpora, by trainingon eight different tasks in a continual pre-training framework. Knowledge masking features among theseeight tasks, and is implemented by treating a phrase or entity as an entire unit, instead of masking theconstituent words. The distinction in how BERT and ERNIE learn is illustrated in how they learn thefollowing sentence: Harry Potter is a series of fantasy novels written by J. K. Rowling• Learned by BERT: [mask] Potter is a series [mask] fantasy novels [mask] by J.[mask] Rowling• Learned by ERNIE: Harry Potter is a series of [mask] [mask] written by [mask][mask] [mask]BERT captures co-occurrence information of ’J’ with ’K’ and ’Rowling’, however it does not captureinformation about the entity J. K. Rowling. By modelling this entity as a single unit, ERNIE claims to becapable of extrapolating the relationship between Harry Potter and J. K. Rowling (Sun et al., 2019). Fur-thermore, ERNIE is trained on a wide varieties of domains, including encyclopedias and news articles,giving the model a lot of knowledge of named entities. This is of great benefit in the Funlines dataset,which is built on news headlines, and therefore features a large number of named entities, particularlypoliticians. This may help the model to infer the relationship between Mitch McConnell and Trump inthe example from table 1.10513.2 Text PreprocessingThe dataset featured the original headline, with the word which had been replaced in angle brackets, andthe substitute word separate. We rendered the edited headlines by placing the word in angle brackets intothe sentence. This did not give our model access to the keyword, or to the original headlines.For ERNIE models, we preprocessed the data as follows: We lowercased the texts and tokenized theminto word pieces, this was implemented with a greedy longest-match-first system to tokenize them giventhe vocabulary. As is conventional for ERNIE, we then added a [CLS] token to the start of each text, anda [SEP] token to the end of each text, with an additional [SEP] replacing the [CLS] in the second text forpairs of texts (e.g. task 2). We also padded sequences to a maximum length of 128.3.3 BaselineFor task 1, we create two baselines, one which predicted a constant value, and the other which predictedthe mean value, using scikit learn (Pedregosa et al., 2011).Table 2: Baselines for Task 1System RMSEPredict Constant Value 0.7214Predict Mean Value 0.6968For task 2, we created three baselines. In the first, we always predicted the same label. The secondbaseline was a trigram language model built on KenLM (Heafield, 2011), using a dataset containingaround 200,000 news headlines from 2012-2018 editions of the Huffington Post1. Similarly to the ap-proach taken by the Duluth team (Yan and Pedersen, 2017) in SemEval 2017, we reasoned that thefunnier of the two headlines would be the least similar to real news headlines, so we selected the sen-tence that had a lower log probability according to the model. However, this performed worse than thefirst baseline.The third baseline was a trigram model built the headlines labelled as sarcastic from a sarcastic newsdataset (Misra and Arora, 2019). These headlines came from The Onion, which the competition datasetseeks to emulate. Here we reasoned that the funnier headline would have a higher log probability underthis language model. Predicting labels in this way was an improvement over the other two baselines,suggesting that the unique data generation methods in this challenge succeeded in emulating satiricalheadlines in some way.Table 3: Baselines for Task 2System AccuracyPredict Constant Value 0.4475200k Huff Post Headlines 0.431428k Onion Headlines 0.45463.4 Model ConfigurationFor the transfer learning models, we used ERNIE base which has 12 layers, a hidden size of 768 and12 self-attention heads. We used a maximum sequence length of 128, a dropout probability of 0.1 andthe Adam optimizer. To finetune for task 1, we built a fully connected layer with mean square error asthe loss function. For task 2, after the fully connected layer, we added a softmax layer and used crossentropy as the loss function.4 ExperimentsWe experimented with optimizing three hyperparameters: learning rate (1e-06, 0.0001 or 0.001), batchsize (16, 32 or 64) and number of epochs (3, 4 or 5). For the sake of brevity, we report only the three1 https://www.kaggle.com/rmisra/news-headlines-dataset-for-sarcasm-detection1052highest and lowest results for each task. The results reported are the mean of 5 runs, with standarddeviation in parentheses.Table 4: Highest and Lowest Performing Parameters for Task 1Learning Rate Batch Size Epochs RMSE (SD)0.0001 16 3 0.5806 (0.011)0.0001 64 3 0.5817 (0.005)0.0001 32 3 0.5829 (0.003)0.001 16 3 0.5966 (0)0.0001 16 5 0.6008 (0.010)1e-06 64 3 0.6009 (0.001)We noticed remarkably little variation in the task 1 results, regardless of the hyperparameter tweaking.Given that the same learning rate is observed in both high and low-scoring systems, and that there is noobservable pattern in terms of batch size, this suggests that another hyperparameter, or variable may helpto achieve better results.By contrast, in task 2, we saw much more variation, with a jump of almost 11% from the lowest to thehighest-scoring configuration. A small learning rate of 0.0001, along with a relatively large batch size of64 featured in all three top results, and the number of epochs was decisive, bringing a 5% increase overat the optimal number - 4. We observed that the lowest learning rate also achieved the lowest scores.However, with too small a learning rate, the network appears not to converge, and varying the otherhyperparameters does not impact this.Table 5: Highest and Lowest Performing Parameters for Task 2Learning Rate Batch Size Epochs Mean Accuracy (SD)0.0001 64 4 0.59408 (0.017)0.0001 64 5 0.54644 (0.051)0.0001 64 3 0.5150 (0.053)1e-06 32 3 0.4911 (0.006)1e-06 64 5 0.4880 (0.005)1e-06 64 3 0.4846 (0.002)5 ConclusionWhile transfer learning models have achieved very impressive results on a variety of NLP tasks, theperformance on this humor task was not as high as anticipated. Perhaps in a multi-task learning setup,we may have seen better performance. Nonetheless, our work demonstrates the importance of optimizingthe hyperparamters of the finetuning layers, which achieved improvements on both tasks, but specificallythe classification task.AcknowledgementsThis work was supported in part by the EPSRC Centre for Doctoral Training in Data Science, funded bythe UK Engineering and Physical Sciences Research Council (grant EP/L016427/1) and the Universityof Edinburgh.ReferencesChristos Baziotis, Nikos Pelekis, and Christos Doulkeridis. 2017. Datastories at semeval-2017 task 4: Deep lstmwith attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th InternationalWorkshop on Semantic Evaluation (SemEval-2017), pages 747–754, Vancouver, Canada, August. Associationfor Computational Linguistics.1053K Binsted, J Hendler, B van den Bergen, S Coulson, Antinus Nijholt, O Stock, C Strapparava, G Ritchie, R Manu-rung, H Pain, et al. 2006. Computational humor. IEEE intelligent systems, 21(suppl 2/2):59–69.Santiago Castro, Luis Chiruzzo, and Aiala Rosá. 2018. Overview of the haha task: Humor analysis based onhuman annotation at ibereval 2018. In IberEval@ SEPLN, pages 187–194.Luis Chiruzzo, S Castro, Mathias Etcheverry, Diego Garat, Juan José Prada, and Aiala Rosá. 2019. Overview ofhaha at iberlef 2019: Humor analysis based on human annotation. In Proceedings of the Iberian LanguagesEvaluation Forum (IberLEF 2019). CEUR Workshop Proceedings, CEUR-WS, Bilbao, Spain (9 2019).Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirec-tional transformers for language understanding. arXiv preprint arXiv:1810.04805.Kenneth Heafield. 2011. Kenlm: Faster and smaller language model queries. In Proceedings of the sixth workshopon statistical machine translation, pages 187–197. Association for Computational Linguistics.Nabil Hossain, John Krumm, and Michael Gamon. 2019. ”President Vows to Cut <Taxes> Hair”: Dataset andAnalysis of Creative Text Editing for Humorous Headlines. In Proceedings of the 2019 Conference of the NorthAmerican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1(Long and Short Papers), pages 133–142.Nabil Hossain, John Krumm, Michael Gamon, and Henry Kautz. 2020. Semeval-2020 Task 7: Assessing humorin edited news headlines. In Proceedings of International Workshop on Semantic Evaluation (SemEval-2020),Barcelona, Spain.Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXivpreprint arXiv:1801.06146.Adilzhan Ismailov. 2019. Humor analysis based on human annotation challenge at iberlef 2019: First-placesolution. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2019). CEUR Workshop Pro-ceedings, CEUR-WS, Bilbao, Spain (9 2019).Rutal Mahajan and Mukesh Zaveri. 2017. Svnit@ semeval 2017 task-6: Learning a sense of humor using super-vised approach. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017),pages 411–415.Rada Mihalcea and Stephen Pulman. 2007. Characterizing humour: An exploration of features in humoroustexts. In International Conference on Intelligent Text Processing and Computational Linguistics, pages 337–347. Springer.Rada Mihalcea and Carlo Strapparava. 2006. Learning to laugh (automatically): Computational models for humorrecognition. Computational Intelligence, 22(2):126–142.Rishabh Misra and Prahal Arora. 2019. Sarcasm detection using hybrid neural network. arXiv preprintarXiv:1908.07414.Reynier Ortega-Bueno, Carlos E Muniz-Cuza, José E Medina Pagola, and Paolo Rosso. 2018. Uo upv: Deeplinguistic humor detection in spanish social media. In Proceedings of the Third Workshop on Evaluation ofHuman Language Technologies for Iberian Languages (IberEval 2018) co-located with 34th Conference of theSpanish Society for Natural Language Processing (SEPLN 2018).Reynier Ortega-Bueno, Paolo Rosso, and José E Medina Pagola. 2019. Uo upv2 at haha 2019: Bigru neuralnetwork informed with linguistic features for humor recognition. In Proceedings of the Iberian LanguagesEvaluation Forum (IberLEF 2019). CEUR Workshop Proceedings, CEUR-WS, Bilbao, Spain (9 2019).José Ortiz-Bejar, Vladimir Salgado, Mario Graff, Daniela Moctezuma, Sabino Miranda-Jiménez, and Eric S Tellez.2018. Ingeotec at ibereval 2018 task haha: µtc and evomsa to detect and score humor in texts. In Proceedingsof the Third Workshop on Evaluation of Human Language Technologies for Iberian Languages (IberEval 2018)co-located with 34th Conference of the Spanish Society for Natural Language Processing (SEPLN 2018).F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss,V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825–2830.Peter Potash, Alexey Romanov, and Anna Rumshisky. 2017. Semeval-2017 task 6:# hashtagwars: Learning asense of humor. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017),pages 49–57.1054Oliviero Stock and Carlo Strapparava. 2006. Laughing with hahacronym, a computational humor system. In 21stconference of American Association for Artificial Intelligence (AAAI-06), pages 1675–1678. AAAI.Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2019. Ernie 2.0: Acontinual pre-training framework for language understanding. arXiv preprint arXiv:1907.12412.Yu Sun, Shuohuan Wang, Yu-Kun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020. Ernie 2.0: Acontinual pre-training framework for language understanding. In AAAI, pages 8968–8975.Julia M Taylor and Lawrence J Mazlack. 2004. Computationally recognizing wordplay in jokes. In Proceedingsof the Annual Meeting of the Cognitive Science Society, volume 26.Xinru Yan and Ted Pedersen. 2017. Duluth at semeval-2017 task 6: Language models in humor detection. arXivpreprint arXiv:1704.08390.

Smash at SemEval-2020 Task 7: Optimizing the Hyperparameters of ERNIE 2.0 for Humor Ranking and Rating

https://www.research.ed.ac.uk/files/197331474/Smash_at_SemEval_MEANEY_DOA24062020_VOR_CC_BY.pdf

Smash at SemEval-2020 Task 7: Optimizing the Hyperparameters of ERNIE 2.0 for Humor Ranking and Rating

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer