Conditional Neural Headline Generation for Finnish

Automatic headline generation has the potential to significantly assist editors charged with head- lining articles. Approaches to automation in the headlining process can range from tools as creative aids, to complete end to end automation. The latter is difficult to achieve as journalistic require- ments imposed on headlines must be met with little room for error, with the requirements depending on the news brand in question. This thesis investigates automatic headline generation in the context of the Finnish newsroom. The primary question I seek to answer is how well the current state of text generation using deep neural language models can be applied to the headlining process in Finnish news media. To answer this, I have implemented and pre-trained a Finnish generative language model based on the Transformer architecture. I have fine-tuned this language model for headline generation as autoregression of headlines conditioned on the article text. I have designed and implemented a variation of the Diverse Beam Search algorithm, with additional parameters, to perform the headline generation in order to generate a diverse set of headlines for a given text. The evaluation of the generative capabilities of this system was done with real world usage in mind. I asked domain-experts in headlining to evaluate a generated set of text-headline pairs. The task was to accept or reject the individual headlines in key criteria. The responses of this survey were then quantitatively and qualitatively analyzed. Based on the analysis and feedback, this model can already be useful as a creative aid in the newsroom despite being far from ready for automation. I have identified concrete improvement directions based on the most common types of errors, and this provides interesting future work

Findings of the 2019 Conference on Machine Translation (WMT19)

This paper presents the results of the premier shared task organized alongside the Conference on Machine Translation (WMT) 2019. Participants were asked to build machine translation systems for any of 18 language pairs, to be evaluated on a test set of news stories. The main metric for this task is human judgment of translation quality. The task was also opened up to additional test suites to probe specific aspects of translation

Deep learning in medical document classification

Text-based data is produced at an ever growing rate each year which has in turn increased the need for automatic text processing. Thus, to keep up with the amount of data, automatic natural language processing techniques have also been increasingly researched and developed, especially in the last decade or so. This has lead to substantial improvements in various natural language processing tasks such as classification, translation and information retrieval. A major breakthrough has been the utilization of deep neural networks and massive amounts of data to train them. Using such methods in areas where time is valuable, such as the medical field, could provide considerable value. In this thesis, an overview is given of natural language processing w.r.t deep learning and text classification. Additionally, a dataset of medical reports in Finnish was preprocessed and used to train and evaluate a number of text classifiers for diagnosis code prediction in order to define the feasibility of such methods for medical text classification. The chosen methods include deep learning -based FinBERT, ULMFiT and ELECTRA, and a simpler linear baseline classifier, fastText. The results show that with a limited dataset, linear methods like fastText work surprisingly well. Deep learning -based methods, on the other hand, seem work reasonably well, and show a lot of potential especially in utilizing larger amounts of training data. In order to define the full potential of such methods, further investigation is required with different datasets and classification tasks.Tekstipohjaista tietoa tuotetaan vuosi vuodelta enemmän mikä puolestaan on lisännyt tarvetta automaattiselle tekstinkäsittelylle. Täten myös automaattisia tekniikoita luonnollisen kielen käsittelyyn on enenevissä määrin tutkittu ja kehitetty, erityisesti viimeisen vuosikymmenen aikana. Tämä on johtanut huomattaviin parannuksiin erilaisissa luonnollisen kielen käsittelytehtävissä. Suuri läpimurto on ollut valtavilla tietomäärillä koulutettujen syvien neuroverkkojen käyttäminen. Tällaisten menetelmien käyttö alueilla joilla aika on arvokasta, kuten lääketiede, voisi tarjota huomattavaa lisäarvoa. Tämä tutkielma antaa yleiskuvan luonnollisen kielen käsittelystä keskittyen syväoppimiseen ja tekstinluokitteluun. Lisäksi erilaisten syväoppivien menetelmien käytettävyyttä arvioitiin kouluttamalla tekstiluokittelijoita ennustamaan suomenkielisten lääketieteellisten dokumenttien diagnoosikoodeja. Valittuihin menetelmiin kuuluvat syväoppimiseen perustuvat FinBERT, ULMFiT ja ELECTRA, sekä yksinkertaisempi lineaarinen luokittelija fastText. Tulokset osoittavat, että rajallisella aineistolla lineaariset menetelmät, kuten fastText, toimivat yllättävän hyvin. Syväoppimiselle perustuvat menetelmät taasen vaikuttavat toimivan kohtuullisen hyvin, vaikkakin niiden aito potentiaali pitäisi todentaa käyttäen suurempia datajoukkoja. Täten jatkotutkimusta syväoppiviin menetelmiin liittyen tarvitaan