Разработка алгоритма поиска ключевых слов в корпусе текста на казахском языке

The issue of semantic text analysis occupies a special place in computational linguistics. Researchers in this field have an increased interest in developing an algorithm that will improve the quality of text corpus processing and probabilistic determination of text content. The results of the study on the application of methods, approaches, algorithms for semantic text analysis in computational linguistics in international and Kazakhstan science led to the development of an algorithm of keyword search in a Kazakh text. The first step of the algorithm was to compile a reference dictionary of keywords for the Kazakh language text corpus. The solution to this problem was to apply the Porter (stemmer) algorithm for the Kazakh language text corpus. The implementation of the stemmer allowed highlighting unique word stems and getting a reference dictionary, which was subsequently indexed. The next step is to collect learning data from the text corpus. To calculate the degree of semantic proximity between words, each word is assigned a vector of the corresponding word forms of the reference dictionary, which results in a pair of a keyword and a vector. And the last step of the algorithm is neural network learning. During learning, the error backpropagation method is used, which allows a semantic analysis of the text corpus and obtaining a probabilistic number of words close to the expected number of keywords. This process automates the processing of text material by creating digital learning models of keywords. The algorithm is used to develop a neurocomputer system that will automatically check the text works of online learners. The uniqueness of the keyword search algorithm is the use of neural network learning for texts in the Kazakh language. In Kazakhstan, scientists in the field of computational linguistics conducted a number of studies based on morphological analysis, lemmatization and other approaches and implemented linguistic tools (mainly translation dictionaries). The scope of neural network learning for parsing of the Kazakh language remains an open issue in the Kazakhstan science.The developed algorithm involves solving one of the problems of effective semantic analysis of the text in the Kazakh languageВопрос семантического анализа текста занимает особое место в компьютерной лингвистике. Исследователи данной области имеют повышенный интерес к разработке алгоритма, использование которого позволит повысить качество обработки корпуса текста и вероятностное определение содержания текста. Результаты исследования применений методик, подходов, алгоритмов для семантического анализа текста в компьютерной лингвистике в международной и казахстанской науке привела к разработке алгоритма поиска ключевых слов в тексте на казахском языке. Первым этапом алгоритма было составление эталонного словаря ключевых слов для корпуса текста на казахском языке. Решением этой проблемы стало применение алгоритма Портера (стеммера) для корпуса текстов на казахском языке. Реализация стеммера позволила выделить уникальные основы слов и получить эталонный словарь, который впоследствии проиндексировали. Следующий шаг – это сбор данных по обучению из корпуса текстов. Для вычисления степени семантической близости между словами каждому слову присваивается вектор соответствующих ему словоформ эталонного словаря, в результате которого получается пара – ключевое слово и вектор. И последним шагом алгоритма является обучение нейронных сетей. При обучении применяется метод обратного распространения ошибок, что позволяет провести семантический анализ корпуса текста и получить вероятностное количество слов, близкое к ожидаемому количеству ключевых. Этот процесс позволяет автоматизировать обработку текстового материала путем создания цифровых обучающих моделей ключевых слов. Алгоритм используется для разработки нейрокомпьютерной системы, который будет производить автоматическую проверку текстовых работ обучающихся онлайн курсов. Уникальностью алгоритма поиска ключевых слов является применение обучения нейронной сети для текстов на казахском языке. В Казахстане учеными в области компьютерной лингвистики были проведены ряд исследований на основе применения морфологического анализа, лиммитизации и других подходов и реализованы лингвистические инструменты (в основном словари-переводчики). Область применения обучения нейронных сетей для синтаксического анализа казахского языков остается открытым вопросом в казахстанской науке.Разработанный алгоритм предполагает решение одной из проблем в получении эффективного семантического анализа текста на казахском язык

Розробка тематичної та нейромережевої моделі для навчання даних

Research in the field of semantic text analysis begins with the study of the structure of natural language. The Kazakh language is unique in that it belongs to agglutinative languages and requires careful study. The object of this study is the text in the Kazakh language. Existing approaches to the study of the semantic analysis of text in the Kazakh language do not consider text analysis using the methods of thematic modeling and learning of neural networks. The purpose of this study is to determine the quality of a topic model based on the LDA (Latent Dirichlet Allocation) method with Gibbs sampling, through neural network learning. The LDA model can determine the semantic probability of the keywords of a single document and give them a rating score. To build a neural network, one of the widely used LSTM architectures was used, which has proven itself well in working with NLP (Natural Language Processing). As a result of learning, it is possible to see to what extent the text was trained and how the semantic analysis of the text in the Kazakh language went. The system, developed on the basis of the LDA model and neural network learning, combines the detected keywords into separate topics. In general, the experimental results showed that the use of deep neural networks gives the expected results of the quality of the LDA model in the processing of the Kazakh language. The developed model of the neural network contributes to the assessment of the accuracy of the semantics of the used text in the Kazakh language. The results obtained can be applied in systems for processing text data, for example, when checking the compliance of the topic and content of the proposed texts (abstracts, term papers, theses, and other works).Дослідження в галузі семантичного аналізу тексту починаються з вивчення структури природної мови. Казахська мова унікальна тим, що відноситься до аглютинативних мов і потребує ретельного вивчення. Об'єктом цього дослідження є текст казахською мовою. Існуючі підходи щодо дослідження семантичного аналізу тексту казахською мовою не розглядають аналіз тексту за допомогою методів тематичного моделювання та навчання нейронних мереж. Метою даного дослідження є визначення якості тематичної моделі на основі методу LDA (Latent Dirichlet Allocation) із семплюванням Гібса, через навчання нейронної мережі. LDA модель може визначити семантичну можливість ключових слів одного документа і дати їм коефіцієнт оцінки. Для побудови нейронної мережі була використана одна з поширених архітектур LSTM, яка добре зарекомендувала себе в роботі з NLP (Natural Language Processing). В результаті навчання можна побачити, якою мірою текст навчився і як пройшов семантичний аналіз тексту казахською мовою. Система, розроблена на основі LDA моделі та навчання нейронної мережі, поєднує виявлені ключові слова в окремі теми. В цілому експериментальні результати показали, що використання глибоких нейронних мереж дають передбачувані результати якості LDA моделі в обробці казахської мови. Розроблена модель нейронної мережі сприяє оцінці визначення точності семантики тексту, що використовується казахською мовою. Отримані результати можна застосувати в системах обробки текстових даних, наприклад, при перевірці відповідності теми та змісту запропонованих текстів (рефератів, курсових, дипломних та інших робіт)

Development of the Algorithm of Keyword Search in the Kazakh Language Text Corpus

The issue of semantic text analysis occupies a special place in computational linguistics. Researchers in this field have an increased interest in developing an algorithm that will improve the quality of text corpus processing and probabilistic determination of text content. The results of the study on the application of methods, approaches, algorithms for semantic text analysis in computational linguistics in International and Kazakhstan science led to the development of an algorithm of keyword search in a Kazakh text. The first step of the algorithm was to compile a reference dictionary of keywords for the Kazakh language text corpus. The solution to this problem was to apply the Porter (stemmer) algorithm for the Kazakh language text corpus. The implementation of the stemmer allowed highlighting unique word stems and getting a reference dictionary, which was subsequently indexed. The next step is to collect learning data from the text corpus. To calculate the degree of semantic proximity between words, each word is assigned a vector of the corresponding word forms of the reference dictionary, which results in a pair of a keyword and a vector. And the last step of the algorithm is neural network learning. During learning, the error backpropagation method is used, which allows a semantic analysis of the text corpus and obtaining a probabilistic number of words close to the expected number of keywords. This process automates the processing of text material by creating digital learning models of keywords. The algorithm is used to develop a neurocomputer system that will automatically check the text works of online learners. The uniqueness of the keyword search algorithm is the use of neural network learning for texts in the Kazakh language. In Kazakhstan, scientists in the field of computational linguistics conducted a number of studies based on morphological analysis, lemmatization and other approaches and implemented linguistic tools (mainly translation dictionaries). The scope of neural network learning for parsing of the Kazakh language remains an open issue in the Kazakhstan science.The developed algorithm involves solving one of the problems of effective semantic analysis of the text in the Kazakh languag

Distribution of influenza virus types by age using case-based global surveillance data from twenty-nine countries, 1999-2014

Background: Influenza disease burden varies by age and this has important public health implications. We compared the proportional distribution of different influenza virus types within age strata using surveillance data from twenty-nine countries during 1999-2014 (N=358,796 influenza cases)

Temporal Patterns of Influenza A and B in Tropical and Temperate Countries: What Are the Lessons for Influenza Vaccination?

<div><p>Introduction</p><p>Determining the optimal time to vaccinate is important for influenza vaccination programmes. Here, we assessed the temporal characteristics of influenza epidemics in the Northern and Southern hemispheres and in the tropics, and discuss their implications for vaccination programmes.</p><p>Methods</p><p>This was a retrospective analysis of surveillance data between 2000 and 2014 from the Global Influenza B Study database. The seasonal peak of influenza was defined as the week with the most reported cases (overall, A, and B) in the season. The duration of seasonal activity was assessed using the maximum proportion of influenza cases during three consecutive months and the minimum number of months with ≥80% of cases in the season. We also assessed whether co-circulation of A and B virus types affected the duration of influenza epidemics.</p><p>Results</p><p>212 influenza seasons and 571,907 cases were included from 30 countries. In tropical countries, the seasonal influenza activity lasted longer and the peaks of influenza A and B coincided less frequently than in temperate countries. Temporal characteristics of influenza epidemics were heterogeneous in the tropics, with distinct seasonal epidemics observed only in some countries. Seasons with co-circulation of influenza A and B were longer than influenza A seasons, especially in the tropics.</p><p>Discussion</p><p>Our findings show that influenza seasonality is less well defined in the tropics than in temperate regions. This has important implications for vaccination programmes in these countries. High-quality influenza surveillance systems are needed in the tropics to enable decisions about when to vaccinate.</p></div

Distribution of influenza virus types by age using case-based global surveillance data from twenty-nine countries, 1999-2014

BACKGROUND : Influenza disease burden varies by age and this has important public health implications. We compared the proportional distribution of different influenza virus types within age strata using surveillance data from twenty-nine countries during 1999-2014 (N=358,796 influenza cases). METHODS : For each virus, we calculated a Relative Illness Ratio (defined as the ratio of the percentage of cases in an age group to the percentage of the country population in the same age group) for young children (0-4 years), older children (5-17 years), young adults (18-39 years), older adults (40-64 years), and the elderly (65+ years). We used random-effects meta-analysis models to obtain summary relative illness ratios (sRIRs), and conducted metaregression and sub-group analyses to explore causes of between-estimates heterogeneity. RESULTS : The influenza virus with highest sRIR was A(H1N1) for young children, B for older children, A(H1N1) pdm2009 for adults, and (A(H3N2) for the elderly. As expected, considering the diverse nature of the national surveillance datasets included in our analysis, between-estimates heterogeneity was high (I2>90%) for most sRIRs. The variations of countries’ geographic, demographic and economic characteristics and the proportion of outpatients among reported influenza cases explained only part of the heterogeneity, suggesting that multiple factors were at play. CONCLUSIONS : These results highlight the importance of presenting burden of disease estimates by age group and virus (sub)type.Table S1. Number of influenza cases caused by the difference influenza viruses that were included in the analysis. The Global Influenza B Study, 1999-2014.Figure S1. Forest plot of the Relative Illness Ratio for patients aged 0-4 years infected with A(H1N1) influenza virus. The Global Influenza B Study, 1999-2014. Figure S2. Forest plot of the Relative Illness Ratio for patients aged 5-17 years infected with A(H1N1) influenza virus. The Global Influenza B Study, 1999-2014. Figure S3. Forest plot of the Relative Illness Ratio for patients aged 18-39 years infected with A(H1N1) influenza virus. The Global Influenza B Study, 1999-2014. Figure S4. Forest plot of the Relative Illness Ratio for patients aged 40-64 years infected with A(H1N1) influenza virus. The Global Influenza B Study, 1999-2014. Figure S5. Forest plot of the Relative Illness Ratio for patients aged 65+ years infected with A(H1N1) influenza virus. The Global Influenza B Study, 1999-2014.Table S2. Summary Relative Illness Ratio (sRIR), 95% confidence intervals (95% CI) across age groups and influenza viruses by categories of country ageing index. The Global Influenza B Study, 1999- 2014. Table S3. Summary Relative Illness Ratio (sRIR), 95% confidence intervals (95% CI) across age groups and influenza viruses by percentage of outpatients among cases reported to the influenza surveillance system. The Global Influenza B Study, 1999-2014. Table S4. Summary Relative Illness Ratio (sRIR), 95% confidence intervals (95% CI) across age groups and influenza viruses by country latitude. The Global Influenza B Study, 1999-2014. Table S5. Summary Relative Illness Ratio (sRIR), 95% confidence intervals (95% CI) across age groups and influenza viruses by percentage of influenza cases caused by that influenza virus in the same season. The Global Influenza B Study, 1999-2014. Table S6. Summary Relative Illness Ratio (sRIR), 95% confidence intervals (95% CI) across age groups and influenza viruses by percentage of influenza cases caused by that influenza virus in the previous season. The Global Influenza B Study, 1999-2014. Table S7. Summary Relative Illness Ratio (sRIR), 95% confidence intervals (95% CI) across age groups and influenza viruses by categories of country gross domestic product (GDP) per capita. The Global Influenza B Study, 1999-2014.The Global Influenza B Study is funded by an unrestricted research grant from Sanofi Pasteur.https://bmcinfectdis.biomedcentral.comam2019Medical Virolog

Influenza cases reported to the national influenza surveillance system by each participating country (from southern- to northern-most) and percentages of cases due to influenza type B virus.

<p>Influenza cases reported to the national influenza surveillance system by each participating country (from southern- to northern-most) and percentages of cases due to influenza type B virus.</p

Mean percentage of influenza cases by month (black diamonds) and number of times the peak of the influenza season took place in each month (pink squares) for countries in the inter-tropical belt.

<p>Mean percentage of influenza cases by month (black diamonds) and number of times the peak of the influenza season took place in each month (pink squares) for countries in the inter-tropical belt.</p

Mean percentage of influenza cases by month (black diamonds) and number of times the peak of the influenza season took place in each month (pink squares) for countries in the Southern hemisphere.

<p>Mean percentage of influenza cases by month (black diamonds) and number of times the peak of the influenza season took place in each month (pink squares) for countries in the Southern hemisphere.</p