LEXICON-BASED APPROACH IN GENERALIZATION EVALUATION IN RUSSIAN LANGUAGE MEDIA

We consider generalization as a property of human thinking to make general conclusion based on authors’ own experience and observations and one of the techniques of authors use to manipulate the readership and present an algorithm for evaluation of the generalization in texts. The algorithm is based on the lexicon-based approach. To search the generalization we use ready-made dictionary (KEY-dictionary) and RuSentiLex dictionary. KEY-dictionary contains words and phrases (elements) that express the generalization. In RuSentiLex we take the words and phrases that express opinion and fact. The algorithm searches exact matches the elements from text with the elements from the dictionaries, it is also important that the elements from different dictionaries have their weights. New method is developed for automatic detection of generalization in texts from official media. Numerical calculations of generalization were performed using a special software application. To test the proposed approach the expert estimation were used

KazNewsDataset: Single Country Overall Digital Mass Media Publication Corpus

Mass media is one of the most important elements influencing the information environment of society. The mass media is not only a source of information about what is happening but is often the authority that shapes the information agenda, the boundaries, and forms of discussion on socially relevant topics. A multifaceted and, where possible, quantitative assessment of mass media performance is crucial for understanding their objectivity, tone, thematic focus and, quality. The paper presents a corpus of Kazakhstan media, which contains over 4 million publications from 36 primary sources (which has at least 500 publications). The corpus also includes more than 2 million texts of Russian media for comparative analysis of publication activity of the countries, also about 4000 sections of state policy documents. The paper briefly describes the natural language processing and multiple-criteria decision-making methods, which are the algorithmic basis of the text and mass media evaluation method, and describes the results of several research cases, such as identification of propaganda, assessment of the tone of publications, calculation of the level of socially relevant negativity, comparative analysis of publication activity in the field of renewable energy. Experiments confirm the general possibility of evaluating the socially significant news, identifying texts with propagandistic content, evaluating the sentiment of publications using the topic model of the text corpus since the area under receiver operating characteristics curve (ROC AUC) values of 0.81, 0.73 and 0.93 were achieved on abovementioned tasks. The described cases do not exhaust the possibilities of thematic, tonal, dynamic, etc., analysis of the considered corpus of texts. The corpus will be interesting to researchers considering both multiple publications and mass media analysis, including comparative analysis and identification of common patterns inherent in the media of different countries