Challenges and Opportunities of Using Transformer-Based Multi-Task Learning in NLP Through ML Lifecycle: A Survey

The increasing adoption of natural language processing (NLP) models across industries has led to practitioners' need for machine learning systems to handle these models efficiently, from training to serving them in production. However, training, deploying, and updating multiple models can be complex, costly, and time-consuming, mainly when using transformer-based pre-trained language models. Multi-Task Learning (MTL) has emerged as a promising approach to improve efficiency and performance through joint training, rather than training separate models. Motivated by this, we first provide an overview of transformer-based MTL approaches in NLP. Then, we discuss the challenges and opportunities of using MTL approaches throughout typical ML lifecycle phases, specifically focusing on the challenges related to data engineering, model development, deployment, and monitoring phases. This survey focuses on transformer-based MTL architectures and, to the best of our knowledge, is novel in that it systematically analyses how transformer-based MTL in NLP fits into ML lifecycle phases. Furthermore, we motivate research on the connection between MTL and continual learning (CL), as this area remains unexplored. We believe it would be practical to have a model that can handle both MTL and CL, as this would make it easier to periodically re-train the model, update it due to distribution shifts, and add new capabilities to meet real-world requirements

CURE: An Effective Algorithm for Clustering Large Datasets

U ovom završnom radu obrađuje se tehnika grupiranja skupova podataka s naglaskom na CURE algoritam. Započinjemo definiranjem pojma grupe i udaljenosti među grupama. Objašnjena je osnovna podjela strategija za grupiranje skupova podataka. Obrađeno je nekoliko algoritama koji pretpostavljaju Euklidski prostor i očekivan broj grupa. Prvo obrađujemo hijerarhijski algoritam, zatim obrađujemo k-means algoritam te naposljetku CURE algoritam. Za svaki od algoritama objašnjena je njegova prostorna i vremenska složenost, te je napisan detaljan pseudokod s dodatnim obrazloženjima. Rad zaključujemo eksperimentima nad nekoliko malih i velikih skupova podataka pomoću kojih ukazujemo na prednost CURE algoritma u odnosu na druga dva obrađena u konkretnim situacijama.In this bachelor thesis, we discuss data clustering technique with emphasis on CURE algorithm. We start by defining notation of the group and the distance between the groups. Then, we describe the basic classification of data clustering strategies. We process several algorithms that suggest Euclidean space and that the expected number of groups is known in advance. First, we analyze the hierarchical clustering algorithm, then we analyze k-means algorithm and at the last CURE algorithm. For every described algorithm, space and time complexity is given so as the detailed pseudocode with additional explanations. We conclude the bachelor thesis with experiments on a couple of small and big data sets with the aim to show the advantages of the CURE algorithm over the other two in concrete situations

Song Lyrics Generator

U ovom radu obrađen je problem stvaranja tekstova pjesama s određenom strukturom. Zbog nedostupnosti skupa tekstova pjesama s označenom strukturom definiran je jednostavan algoritam za označavanje strukture teksta pjesme koja je prethodno segmentirana na paragrafe. 55000+ Song Lyrics skup podataka označen je pomoću algoritma kako bi se mogao koristiti u svrhu učenja i vrednovanja modela. Korišteni su sljedeći modeli strojnog učenja: N-gram jezični model, RNN i LSTM neuronske mreže te generativna suparnička mreža SeqGAN. Modeli su vrednovani pomoću tri mjere: zbunjenosti, mjere kvalitete strukture teksta pjesme koja je definirana u ovom radu te Levenshteinove udaljenosti kako bi se mjerila udaljenost između određenih paragrafa. Rad zaključujemo pregledom ostvarenih rezultata navedenih modela nad označenim skupom tekstova pjesama. RNN i LSTM neuronske mreže ostvarile su najbolje rezultate, zatim mreža SeqGAN, a najlošije N-gram jezični model.In this thesis we discuss the problem of generating song lyrics with a certain structure. Due to unavailability of song lyrics dataset with marked lyrics structure, a simple algorithm is defined to mark the structure of the lyrics of a song that has been previously segmented into paragraphs. The algorithm is then used to mark lyrics structure on 55000+ Song Lyrics dataset so the dataset can be used for learning and evaluating models. Following machine learning models were used: N-gram language model, RNN and LSTM recurrent networks and generative adversarial network SeqGAN. The models where evaluated using three measures: perplexity, measure of song lyrics structure defined in this thesis and Levenshtein's distance to measure the distance between certain paragraphs. We conclude the paper by reviewing the achieved results of the mentioned models over the marked lyrics dataset. RNN and LSTM neural networks achieved the best results, followed by the SeqGAN network, and the worst by the N-gram language model

Song Lyrics Generator

U ovom radu obrađen je problem stvaranja tekstova pjesama s određenom strukturom. Zbog nedostupnosti skupa tekstova pjesama s označenom strukturom definiran je jednostavan algoritam za označavanje strukture teksta pjesme koja je prethodno segmentirana na paragrafe. 55000+ Song Lyrics skup podataka označen je pomoću algoritma kako bi se mogao koristiti u svrhu učenja i vrednovanja modela. Korišteni su sljedeći modeli strojnog učenja: N-gram jezični model, RNN i LSTM neuronske mreže te generativna suparnička mreža SeqGAN. Modeli su vrednovani pomoću tri mjere: zbunjenosti, mjere kvalitete strukture teksta pjesme koja je definirana u ovom radu te Levenshteinove udaljenosti kako bi se mjerila udaljenost između određenih paragrafa. Rad zaključujemo pregledom ostvarenih rezultata navedenih modela nad označenim skupom tekstova pjesama. RNN i LSTM neuronske mreže ostvarile su najbolje rezultate, zatim mreža SeqGAN, a najlošije N-gram jezični model.In this thesis we discuss the problem of generating song lyrics with a certain structure. Due to unavailability of song lyrics dataset with marked lyrics structure, a simple algorithm is defined to mark the structure of the lyrics of a song that has been previously segmented into paragraphs. The algorithm is then used to mark lyrics structure on 55000+ Song Lyrics dataset so the dataset can be used for learning and evaluating models. Following machine learning models were used: N-gram language model, RNN and LSTM recurrent networks and generative adversarial network SeqGAN. The models where evaluated using three measures: perplexity, measure of song lyrics structure defined in this thesis and Levenshtein's distance to measure the distance between certain paragraphs. We conclude the paper by reviewing the achieved results of the mentioned models over the marked lyrics dataset. RNN and LSTM neural networks achieved the best results, followed by the SeqGAN network, and the worst by the N-gram language model

Song Lyrics Generator

U ovom radu obrađen je problem stvaranja tekstova pjesama s određenom strukturom. Zbog nedostupnosti skupa tekstova pjesama s označenom strukturom definiran je jednostavan algoritam za označavanje strukture teksta pjesme koja je prethodno segmentirana na paragrafe. 55000+ Song Lyrics skup podataka označen je pomoću algoritma kako bi se mogao koristiti u svrhu učenja i vrednovanja modela. Korišteni su sljedeći modeli strojnog učenja: N-gram jezični model, RNN i LSTM neuronske mreže te generativna suparnička mreža SeqGAN. Modeli su vrednovani pomoću tri mjere: zbunjenosti, mjere kvalitete strukture teksta pjesme koja je definirana u ovom radu te Levenshteinove udaljenosti kako bi se mjerila udaljenost između određenih paragrafa. Rad zaključujemo pregledom ostvarenih rezultata navedenih modela nad označenim skupom tekstova pjesama. RNN i LSTM neuronske mreže ostvarile su najbolje rezultate, zatim mreža SeqGAN, a najlošije N-gram jezični model.In this thesis we discuss the problem of generating song lyrics with a certain structure. Due to unavailability of song lyrics dataset with marked lyrics structure, a simple algorithm is defined to mark the structure of the lyrics of a song that has been previously segmented into paragraphs. The algorithm is then used to mark lyrics structure on 55000+ Song Lyrics dataset so the dataset can be used for learning and evaluating models. Following machine learning models were used: N-gram language model, RNN and LSTM recurrent networks and generative adversarial network SeqGAN. The models where evaluated using three measures: perplexity, measure of song lyrics structure defined in this thesis and Levenshtein's distance to measure the distance between certain paragraphs. We conclude the paper by reviewing the achieved results of the mentioned models over the marked lyrics dataset. RNN and LSTM neural networks achieved the best results, followed by the SeqGAN network, and the worst by the N-gram language model