Application of pre-training and fine-tuning AI models to machine translation: a case study of multilingual text classification in Baidu

With the development of international information technology, we are producing a huge amount of information all the time. The processing ability of information in various languages is gradually replacing information and becoming a rarer resource. How to obtain the most effective information in such a large and complex amount of multilingual textual information is a major goal of multilingual information processing. Multilingual text classification helps users to break the language barrier and accurately locate the required information and triage information. At the same time, the rapid development of the Internet has accelerated the communication among users of various languages, giving rise to a large number of multilingual texts, such as book and movie reviews, online chats, product introductions and other forms, which contain a large amount of valuable implicit information and urgently need automated tools to categorize and process those multilingual texts. This work describes the Natural Language Process (NLP) sub-task known as Multilingual Text Classification (MTC) performed within the context of Baidu, a Chinese leading AI company with a strong Internet base, whose NLP division led the industry in deep learning technology to go online in Machine Translation (MT) and search. Multilingual text classification is an important module in NLP machine translation and a basic module in NLP tasks. It can be applied to many fields, such as Fake Reviews Detection, News Headlines Categories Classification, Analysis of positive and negative reviews and so on. In the following work, we will first define the AI model paradigm of 'pre-training and fine-tuning' in deep learning in the Baidu NLP department. Then investigated the application scenarios of multilingual text classification. Most of the text classification systems currently available in the Chinese market are designed for a single language, such as Alibaba's text classification system. If users need to classify texts of the same category in multiple languages, they need to train multiple single text classification systems and then classify them one by one. However, many internationalized products do not have a single text language, such as AliExpress cross-border e-commerce business, Airbnb B&B business, etc. Industry needs to understand and classify users’ reviews in various languages, and have conducted in-depth statistics and marketing strategy development, and multilingual text classification is particularly important in this scenario. Therefore, we focus on interpreting the methodology of multilingual text classification model of machine translation in Baidu NLP department, and capture sets of multilingual data of reviews, news headlines and other data for manual classification and labeling, use the labeling results for fine-tuning of multilingual text classification model, and output the quality evaluation data of Baidu multilingual text classification model after fine-tuning. We will discuss if the pre-training and fine-tuning of the large model can substantially improve the quality and performance of multilingual text classification. Finally, based on the machine translation-multilingual text classification model, we derive the application method of pre-training and fine-tuning paradigm in the current cutting-edge deep learning AI model under the NLP system and verify the generality and cutting-edge of the pre-training and fine-tuning paradigm in the deep learning-intelligent search field.Com o desenvolvimento da tecnologia de informação internacional, estamos sempre a produzir uma enorme quantidade de informação e o recurso mais escasso já não é a informação, mas a capacidade de processar informação em cada língua. A maior parte da informação multilingue é expressa sob a forma de texto. Como obter a informação mais eficaz numa quantidade tão considerável e complexa de informação textual multilingue é um dos principais objetivos do processamento de informação multilingue. A classificação de texto multilingue ajuda os utilizadores a quebrar a barreira linguística e a localizar com precisão a informação necessária e a classificá-la. Ao mesmo tempo, o rápido desenvolvimento da Internet acelerou a comunicação entre utilizadores de várias línguas, dando origem a um grande número de textos multilingues, tais como críticas de livros e filmes, chats, introduções de produtos e outros distintos textos, que contêm uma grande quantidade de informação implícita valiosa e necessitam urgentemente de ferramentas automatizadas para categorizar e processar esses textos multilingues. Este trabalho descreve a subtarefa do Processamento de Linguagem Natural (PNL) conhecida como Classificação de Texto Multilingue (MTC), realizada no contexto da Baidu, uma empresa chinesa líder em IA, cuja equipa de PNL levou a indústria em tecnologia baseada em aprendizagem neuronal a destacar-se em Tradução Automática (MT) e pesquisa científica. A classificação multilingue de textos é um módulo importante na tradução automática de PNL e um módulo básico em tarefas de PNL. A MTC pode ser aplicada a muitos campos, tais como análise de sentimentos multilingues, categorização de notícias, filtragem de conteúdos indesejados (do inglês spam), entre outros. Neste trabalho, iremos primeiro definir o paradigma do modelo AI de 'pré-treino e afinação' em aprendizagem profunda no departamento de PNL da Baidu. Em seguida, realizaremos a pesquisa sobre outros produtos no mercado com capacidade de classificação de texto — a classificação de texto levada a cabo pela Alibaba. Após a pesquisa, verificamos que a maioria dos sistemas de classificação de texto atualmente disponíveis no mercado chinês são concebidos para uma única língua, tal como o sistema de classificação de texto Alibaba. Se os utilizadores precisarem de classificar textos da mesma categoria em várias línguas, precisam de aplicar vários sistemas de classificação de texto para cada língua e depois classificá-los um a um. No entanto, muitos produtos internacionalizados não têm uma única língua de texto, tais como AliExpress comércio eletrónico transfronteiriço, Airbnb B&B business, etc. A indústria precisa compreender e classificar as revisões dos utilizadores em várias línguas. Esta necessidade conduziu a um desenvolvimento aprofundado de estatísticas e estratégias de marketing, e a classificação de textos multilingues é particularmente importante neste cenário. Desta forma, concentrar-nos-emos na interpretação da metodologia do modelo de classificação de texto multilingue da tradução automática no departamento de PNL Baidu. Colhemos para o efeito conjuntos de dados multilingues de comentários e críticas, manchetes de notícias e outros dados para classificação manual, utilizamos os resultados dessa classificação para o aperfeiçoamento do modelo de classificação de texto multilingue e produzimos os dados de avaliação da qualidade do modelo de classificação de texto multilingue da Baidu. Discutiremos se o pré-treino e o aperfeiçoamento do modelo podem melhorar substancialmente a qualidade e o desempenho da classificação de texto multilingue. Finalmente, com base no modelo de classificação de texto multilingue de tradução automática, derivamos o método de aplicação do paradigma de pré-formação e afinação no atual modelo de IA de aprendizagem profunda de ponta sob o sistema de PNL, e verificamos a robustez e os resultados positivos do paradigma de pré-treino e afinação no campo de pesquisa de aprendizagem profunda

AI-Generated Content (AIGC): A Survey

To address the challenges of digital intelligence in the digital economy, artificial intelligence-generated content (AIGC) has emerged. AIGC uses artificial intelligence to assist or replace manual content generation by generating content based on user-inputted keywords or requirements. The development of large model algorithms has significantly strengthened the capabilities of AIGC, which makes AIGC products a promising generative tool and adds convenience to our lives. As an upstream technology, AIGC has unlimited potential to support different downstream applications. It is important to analyze AIGC's current capabilities and shortcomings to understand how it can be best utilized in future applications. Therefore, this paper provides an extensive overview of AIGC, covering its definition, essential conditions, cutting-edge capabilities, and advanced features. Moreover, it discusses the benefits of large-scale pre-trained models and the industrial chain of AIGC. Furthermore, the article explores the distinctions between auxiliary generation and automatic generation within AIGC, providing examples of text generation. The paper also examines the potential integration of AIGC with the Metaverse. Lastly, the article highlights existing issues and suggests some future directions for application.Comment: Preprint. 14 figures, 4 table

Англійська мова для навчання і роботи. Т. 3. Дискусії та презентації

Розглянуто всі види діяльності студентів з вивчення англійської мови, спрямовані на розвиток мовної поведінки, необхідної для ефективного спілкування в академічному та професійному середовищах. Містить завдання і вправи, типові для різноманітних академічних та професійних сфер і ситуацій. Структура організації змісту – модульна, охоплює певні мовленнєві вміння залежно від мовної поведінки. Даний модуль має на меті розвиток у студентів умінь і навичок академічного і професійно-орієнтованого мовлення, необхідних для участі в дискусіях, семінарах, конференціях та при підготовці й проведенні презентацій (виступів-доповідей). Зразки текстів – автентичні, містять цікаву та актуальну інформацію із загальнонаукової та професійної тематики. Ресурси для самостійної роботи (частина ІІ) включають завдання та вправи для розвитку словникового запасу та розширення діапазону функціональних зразків, необхідних для виконання певних функцій, та завдання, які спрямовані на організацію самостійної роботи студентів. За допомогою засобів діагностики (частина ІІІ) студенти можуть самостійно перевірити засвоєння навчального матеріалу та оцінити свої досягнення. Граматичні явища і вправи для їх засвоєння наводяться в томі 5. Призначений для студентів технічних університетів гірничого профілю. Може використовуватися для викладання вибіркових курсів англійської мови, а також у самостійному вивченні англійської мови викладачами, фахівцями і науковцями різних інженерних галузей

Topical Mining of malaria Using Social Media. A Text Mining Approach

Malaria is a life-threatening parasitic disease, common in subtropical and tropical climates caused by mosquitoes. Each year, several hundred thousand of people die from malaria infections. However, with the rapid growth, popularity and global reach of social media usage, a myriad of opportunities arises for extracting opinions and discourses on various topics and issues. This research examines the public discourse, trends and emergent themes surrounding malaria discussion. We query Twitter corpus leveraging text mining algorithms to extract and analyze topical themes. Further, to investigate these dynamics, we use Crimson social media analytics software to analyze topical emergent themes and monitor malaria trends. The findings reveal the discovery of pertinent topics and themes regarding malaria discourses. The implications include shedding insights to public health officials on sentiments and opinions shaping public discourse on malaria epidemic. The multi-dimensional analysis of data provides directions for future research and informs public policy decisions

On cross-domain social semantic learning

Approximately 2.4 billion people are now connected to the Internet, generating massive amounts of data through laptops, mobile phones, sensors and other electronic devices or gadgets. Not surprisingly then, ninety percent of the world's digital data was created in the last two years. This massive explosion of data provides tremendous opportunity to study, model and improve conceptual and physical systems from which the data is produced. It also permits scientists to test pre-existing hypotheses in various fields with large scale experimental evidence. Thus, developing computational algorithms that automatically explores this data is the holy grail of the current generation of computer scientists. Making sense of this data algorithmically can be a complex process, specifically due to two reasons. Firstly, the data is generated by different devices, capturing different aspects of information and resides in different web resources/ platforms on the Internet. Therefore, even if two pieces of data bear singular conceptual similarity, their generation, format and domain of existence on the web can make them seem considerably dissimilar. Secondly, since humans are social creatures, the data often possesses inherent but murky correlations, primarily caused by the causal nature of direct or indirect social interactions. This drastically alters what algorithms must now achieve, necessitating intelligent comprehension of the underlying social nature and semantic contexts within the disparate domain data and a quantifiable way of transferring knowledge gained from one domain to another. Finally, the data is often encountered as a stream and not as static pages on the Internet. Therefore, we must learn, and re-learn as the stream propagates. The main objective of this dissertation is to develop learning algorithms that can identify specific patterns in one domain of data which can consequently augment predictive performance in another domain. The research explores existence of specific data domains which can function in synergy with another and more importantly, proposes models to quantify the synergetic information transfer among such domains. We include large-scale data from various domains in our study: social media data from Twitter, multimedia video data from YouTube, video search query data from Bing Videos, Natural Language search queries from the web, Internet resources in form of web logs (blogs) and spatio-temporal social trends from Twitter. Our work presents a series of solutions to address the key challenges in cross-domain learning, particularly in the field of social and semantic data. We propose the concept of bridging media from disparate sources by building a common latent topic space, which represents one of the first attempts toward answering sociological problems using cross-domain (social) media. This allows information transfer between social and non-social domains, fostering real-time socially relevant applications. We also engineer a concept network from the semantic web, called semNet, that can assist in identifying concept relations and modeling information granularity for robust natural language search. Further, by studying spatio-temporal patterns in this data, we can discover categorical concepts that stimulate collective attention within user groups.Includes bibliographical references (pages 210-214)

English for Study and Work

Подано всі види діяльності студентів з вивчення англійської мови, спрямовані на розвиток мовної поведінки, необхідної для ефективного спілкування в академічному та професійному середовищах. Містить завдання і ситуації, типові для різноманітних академічних та професійних сфер. Структура організації змісту – модульна, охоплює певні мовленнєві вміння залежно від мовної поведінки. Даний модуль має на меті розвиток у студентів умінь і навичок академічного і професійно- орієнтованого мовлення, необхідних для участі в дискусіях, семінарах, конференціях та при підготовці й проведенні презентацій (виступів-доповідей). Зразки текстів – автентичні, містять цікаву та актуальну інформацію з загальнонаукової та професійної тематики. Ресурси для самостійної роботи (Частина ІІ) містять завдання та вправи для розвитку словникового запасу та розширення діапазону функціональних зразків, необхідних для виконання певних функцій, та завдання, які спрямовані на організацію самостійної роботи студентів. За допомогою засобів діагностики студенти можуть самостійно перевірити засвоєння навчального матеріалу та оцінити свої досягнення. Наводяться граматичні явища і вправи для їх засвоєння. Призначений для студентів технічних університетів гірничого профілю. Може використовуватися для викладання вибіркових курсів з англійської мови, а також для самостійного вивчення англійської мови викладачами, фахівцями і науковцями різних інженерних галузей

Англійська мова для навчання і работи. Навчальний посібник з англійської мови за професійним спрямуванням для студентів і фахівців галузі знань 0503 Розробка корисних копалин Т 1

A coursebook includes all the activities of students’ work at ESP course aimed at development of language behaviour necessary for effective communication of students in their study and specialism areas. The tasks and activities given in the coursebook are typicalfor students’ academic and professional domains and situations. The content is organized in modules that covers generic job-related language skills of engineers. The authentic texts taken from real life contain interesting up-to-date information about mining, peculiarities of study abroad, customs and traditions of English-speaking countries. Pack of self-study resources given in Part II contains Glossary of mining terms, tasks and activities aimed at developing a range of vocabulary necessary for mining, different functions and functional exponents to be used in academic and professional environment as well as tasks developing self-awareness, self-assessment and self-organisation skills. Testing points for different grammar structuresare given in Part III. Indices at the end of each part easify the use of the coursebook. The coursebook contains illustrations, various samples of visualizing technical information. The coursebook is designed for ESP students of non-linguistic universities. It can be used as teaching/learning materials for ESP Courses for Mining Engineers as well as for self-study of subject and specialist teachers, practicing mining engineers and researchers in Engineering.У посібнику представлені всі види діяльності студентів з вивчення англійської мови, спрямовані на розвиток мовної поведінки, необхідної для ефективного спілкування в академічному та професійному середовищах. Навчальний посібник містить завдання і вправи, типові для різноманітних академічних та професійних сфер і ситуацій. Структура організації змісту– модульна і охоплює загальні мовленнєві вміння інженерів. Зразки текстів– автентичні, взяті з реального життя, містять цікаву та актуальну інформацію про видобувничу промисловість, особливості навчання за кордоном, традиції та звичаї країн, мова яких вивчається. Ресурси для самостійної роботи(Том ІІ) містять глосарій термінів, завдання та вправи для розвитку словарного запасу та розширення діапазону функціональних зразків, необхідних для виконання певних функцій, та завдання, які спрямовані на розвиток навичок самооцінювання і організації свого навчання. Граматичні явища і вправи для їх засвоєння наводяться в томі ІІІ. Наприкінці кожної частини наведено алфавітно-предметні покажчики. Багато ілюстрацій та різних візуальних засобів подання інформації. Навчальний посібник призначений для студентів технічних університетів гірничого профілю. Може використовуватися для самостійного вивчення англійської мови викладачами, фахівцями і науковцями різних інженерних галузей

Emotional Tendency Analysis of Twitter Data Streams

The web now seems to be an alive and dynamic arena in which billions of people across the globe connect, share, publish, and engage in a broad range of everyday activities. Using social media, individuals may connect and communicate with each other at any time and from any location. More than 500 million individuals across the globe post their thoughts and opinions on the internet every day. There is a huge amount of information created from a variety of social media platforms in a variety of formats and languages throughout the globe. Individuals define emotions as powerful feelings directed toward something or someone as a result of internal or external events that have a personal meaning. Emotional recognition in text has several applications in human-computer interface and natural language processing (NLP). Emotion classification has previously been studied using bag-of words classifiers or deep learning methods on static Twitter data. For real-time textual emotion identification, the proposed model combines a mix of keyword-based and learning-based models, as well as a real-time Emotional Tendency Analysi

Trialing project-based learning in a new EAP ESP course: A collaborative reflective practice of three college English teachers

Currently in many Chinese universities, the traditional College English course is facing the risk of being ‘marginalized’, replaced or even removed, and many hours previously allocated to the course are now being taken by EAP or ESP. At X University in northern China, a curriculum reform as such is taking place, as a result of which a new course has been created called ‘xue ke’ English. Despite the fact that ‘xue ke’ means subject literally, the course designer has made it clear that subject content is not the target, nor is the course the same as EAP or ESP. This curriculum initiative, while possibly having been justified with a rationale of some kind (e.g. to meet with changing social and/or academic needs of students and/or institutions), this is posing a great challenge for, as well as considerable pressure on, a number of College English teachers who have taught this single course for almost their entire teaching career. In such a context, three teachers formed a peer support group in Semester One this year, to work collaboratively co-tackling the challenge, and they chose Project-Based Learning (PBL) for the new course. This presentation will report on the implementation of this project, including the overall designing, operational procedure, and the teachers’ reflections. Based on discussion, pre-agreement was reached on the purpose and manner of collaboration as offering peer support for more effective teaching and learning and fulfilling and pleasant professional development. A WeChat group was set up as the chief platform for messaging, idea-sharing, and resource-exchanging. Physical meetings were supplementary, with sound agenda but flexible time, and venues. Mosoteach cloud class (lan mo yun ban ke) was established as a tool for virtual learning, employed both in and after class. Discussions were held at the beginning of the semester which determined only brief outlines for PBL implementation and allowed space for everyone to autonomously explore in their own way. Constant further discussions followed, which generated a great deal of opportunities for peer learning and lesson plan modifications. A reflective journal, in a greater or lesser detailed manner, was also kept by each teacher to record the journey of the collaboration. At the end of the semester, it was commonly recognized that, although challenges existed, the collaboration was overall a success and they were all willing to continue with it and endeavor to refine it to be a more professional and productive approach

Detection of Hateful Comments on Social Media

Social media usage has grown tremendously in the contemporary communication landscape. Along with its numerous benefits, some users abuse the channels by spreading hatred, far from the intended purpose of building connections on a personal level. To date, an empirical method for detecting, quantifying, and categorizing hateful comments on social networks comprehensively and proactively is still lacking. Besides, majority of the cases remain unreported due to social confounders such as fear of victimization and the psychological implications of hateful comments, leading to a situation whereby, the detrimental effect of the situation is underestimated. The ill-defined situation in the growing online space impedes progress towards developing mechanisms and policies to mitigate the harmful effects of hate on social media, ultimately reducing the effectiveness of the platforms as effective communication tools. This proposal suggests Naïve Bayes classifier as a novel approach for detecting and classifying hateful social media comments to bridge this gap. Data set was taken from set provided by Kaggle and consisted of 30,000 Tweets. From the results of the use of this method, it was calculated that Bayes method is 62.75% accurate, which is not satisfactory. However, to bridge accuracy gap, nural algorithm was used which gain an improved accuracy of 87%