This paper presents a contribution to PolEval 2021 Task 1: Punctuation restoration from read text. The system was scored first and achieved 81.29% weighted F1

Wróbel, Krzysztof

Zhylko, Dmytro

Jagiellonian Univeristy Repository

Punctuation Restoration with TransformersKrzysztof Wróbel (Enelpol / Jagiellonian University / AGH Universityof Science and Technology)Dmytro Zhylko (AGH University of Science and Technology / Enelpol)AbstractThis paper presents a contribution1 to PolEval 2021 Task 1: Punctuation restoration fromread text. The system was scored first and achieved 81.29% weighted F1.Keywordsnatural language processing, punctuation restoration1. IntroductionPunctuation restoration is an important problem for automatic speech recognition (ASR)outputs. ASR systems usually produce output as a sequence of words without interpunction.In this article we aim to present capabilities of pre-trained BERT (Devlin et al. 2019) modelto restore missing punctuation using only textual data. We model this problem as sequencelabeling task. In Section 2 we describe datasets, preprocessing and used data splits. Section4 describes evaluation metrics. Sections 5–6 describe model architecture, parameters andapplied techniques. Sections 7–8 contain our results and conclusions.2. DataExperiments were conducted on different combinations and splits of two distinct datasets:WikiNews and WikiTalks.WikiNews is a Wikimedia Foundation project that is based on collaborative journalism, datasetcontains high quality, grammatically correct data, punctuation and other types of errors arerare.1https://github.com/enelpol/poleval2021-task134 Krzysztof Wróbel, Dmytro ZhylkoWikiTalks is a dataset of conversational based on the Wikipedia discussions pages associatedwith Wikipedia articles found in Wikipedia dump, punctuation and grammatical errors aremore common.Table 1 shows descriptions of all shared dataset splits. rest split contains original text fromWikiNews and WikiTalks. Other splits were read, recognized by ASR systems and manuallycorrected adding puntuation marks.Table 2 presents a number of texts in each split with the ratio of WikiNews and WikiTalks texts.Punctuation restoration is an important problem for automatic speech recognition (ASR)outputs. ASR systems usually produce output as a sequence of words without interpunction.Table 1: Dataset splits with descriptionsSplit name Descriptionrest All available text data except data used for model evaluationtrain Original training set, in out experiments mainly used as development settest-A Original development set, after initial experiments was combined with traintest-B/C/D Datasets are consecutive iterations of final test dataset (test-D was usedas final test set)Table 2: Data splits with number of texts from each datasetSplit name Total size WikiNews WikiTalksrest 22 186 13 757 (62.00%) 8429 (38.00%)train 800 632 (79.00%) 168 (21.00%)test-A 200 168 (84.00%) 32 (16.00%)test-B 185 185 (100%) 0 (0.00%)test-C 274 185 (67.52%) 89 (32.48%)test-D 244 155 (63.52%) 89 (36.47%)3. PreprocessingAll data splits except rest were processed by organizers and can be found in competitionrepository2.rest split has been processed to resemble train data:— Quotation marks and plus characters were separated.— Multiple whitespaces deleted.— HTML entities were replaced to its characters.Table 3 shows label statistics in rest and train splits for each dataset.2https://github.com/poleval/2021-punctuation-restoration.gitPunctuation Restoration with Transformers 35Table 3: Statistics of punctuation marks in rest and train splits divided into both datasets. About 15% ofall tokens are tokens with punctuation marksrest trainWikiNews WikiTalks WikiNews WikiTalks- 6.36% 5.73% 0.00% 0.52%, 42.75% 41.90% 46.53% 37.86%; 0.33% 0.58% 0.29% 0.72%: 3.48% 6.16% 3.55% 5.58%! 0.25% 1.09% 0.19% 1.43%? 0.37% 4.49% 0.26% 13.87%. 46.35% 38.34% 49.18% 39.96%... 0.12% 1.69% 0.00% 0.06%4. EvaluationWe used weighted F1 as our main metric as in task specifications.5. MethodsWe used pre-trained HerBERT model (Mroczkowski et al. 2021) as it is currently the bestmodel for Polish language according to KLEJ benchmark (Rybak et al. 2020). Additionally weused single linear layer to make predictions for each word in the text. More specifically wepredict punctuation mark that comes after given word, in case of word being split into fewtokens we predict punctuation only for first token.We also experimented with Conditional Random Fields (CRF) layer, but didn’t get any tangibleimprovements.6. ExperimentsFor all experiments data was previously splitted into chunks that can fit into experimentspecific parameter of maximum token sequence length (max_seq_len) with experiment specificoverlap between splits. Training data was also splitted, but without overlap to not worsenlabel disbalance in data.6.1. PolEval SubmissionsCommon model parameters for PolEval experiments:— model: HerBERT large— max sequence length: 25636 Krzysztof Wróbel, Dmytro ZhylkoFew models were trained on rest corpus with train data as validation. Then hyperparameteroptimization was performed using test-A combined with train corpus as validation overparameters:— model: one of few trained on rest corpus— batch size: 4, 8, 12— epochs: 1, 2, 4, 10— learning rate: 5e-6, 1e-5, 2e-5, 5e-5— evaluation every 50 steps70 models were trained and the best run achieved the best result on the final leaderboard.Final model parameters are following: batch size 12, epochs 10, learning rate 5e-6.Submitted solutions:— AE1 — models trained on rest, train and test-A corpora, without validation, majorityvoting— E1 — models trained on rest, with validation on train, majority voting— SE1 — further training of models trained on rest corpus with test-A corpus, with valida-tion on train, majority voting— S1 — further training of models trained on rest corpus with test-A corpus, with validationon train, the best model from 70 models7. ResultsTable 4 presents final results in PolEval competition. Our submission S1 won with smallmargin over other contestants.Table 4: PolEval best results and scores by different submissionstest-D weighted F1AE1 80.09E1 80.86SE1 81.27S1 81.29Norbert Ropiak 81.25Michał Marcińczuk 81.23Table 5 presents F1 scores for each punctuation marks. The most influence on weighted F1has F1 scores of comma and dot.Punctuation Restoration with Transformers 37Table 5: Results of S1 submission per labelPunctuation mark test-D F1- 66.91, 76.08... 0.00. 88.68? 80.61: 82.98! 36.368. ConclusionsThe top three contestants obtained very similar results. Our success depends on properpreprocessing, good correlation between scores on train and test-D and 70 experiments inhyperparameter optimization process.AcknowledgmentsThis work was supported in part by the Polish National Centre for Research and Development –LIDER Program under Grant LIDER/27/0164/L-8/16/NCBR/2017 titled “Lemkin – intelligentlegal information system” and in part by the PLGrid Infrastructure.ReferencesDevlin J., Chang M.-W., Lee K. and Toutanova K. (2019). BERT: Pre-training of deep bidirec-tional transformers for language understanding. In Proceedings of the 2019 Conference of theNorth American Chapter of the Association for Computational Linguistics: Human LanguageTechnologies, Volume 1 (Long and Short Papers), pp. 4171–4186, Minneapolis, Minnesota.Association for Computational Linguistics.Mroczkowski R., Rybak P., Wróblewska A. and Gawlik I. (2021). HerBERT: Efficiently pretrainedtransformer-based language model for Polish. In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pp. 1–10, Kiyv, Ukraine. Association for ComputationalLinguistics.Rybak P., Mroczkowski R., Tracz J. and Gawlik I. (2020). KLEJ: Comprehensive benchmark forPolish language understanding. In Proceedings of the 58th Annual Meeting of the Association forComputational Linguistics, pp. 1191–1201, Online. Association for Computational Linguistics.

Punctuation restoration with transformers

Abstract

Similar works

Full text

Available Versions

Jagiellonian Univeristy Repository