Основная статьяNowadays Social Media becomes one of the major providers of Big Data for
analysis of users’ behaviour, focus, trends, and deviations. One user can be presented in several
social networks by various avatars. Most users have different dynamics of data processing and
generation. In order to provide a solution capable to deal with this, there was developed and
implemented a software library for integration with a number of social networks. This paper
describes the problem, solution architecture and technical details of its implementation
supported by the results of simulation and real data analysis for a number of popular social
networks.This work was supported by the Federal Agency of Scientific Organizations (agreement No 007-
GZ/Ch3363/26)

A.A. Khorina

A.V. Ivaschenko

D.N. Krupin

N.Yu. Ilyasova

P.V. Sitnikov

V.A. Bolotsky

V.A. Isayko

Samara University

Науки о данных IV Международная конференция и молодёжная школа «Информационные технологии и нанотехнологии» (ИТНТ-2018) Integration Issues of Big Data Analysis on Social Networks A.V. Ivaschenko1, N.Yu. Ilyasova1,2, A.A. Khorina1, V.A. Isayko1, D.N. Krupin3,          V.A. Bolotsky4, P.V. Sitnikov4 1Samara National Research University, Moskovskoye shosse, 34, Samara, Russia, 443086 2Image Processing Systems Institute of RAS - Branch of the FSRC "Crystallography and Photonics" RAS, Molodogvardejskaya street 151, Samara, Russia, 443001  3IPSI SEC “Open Code”, 55, Yarmarochnaya Str., Samara, Russia, 443001 4ITMO University, 14, Birzhevaya liniya, Saint-Petersburg, Russia, 199034  Abstract. Nowadays Social Media becomes one of the major providers of Big Data for analysis of users’ behaviour, focus, trends, and deviations. One user can be presented in several social networks by various avatars. Most users have different dynamics of data processing and generation. In order to provide a solution capable to deal with this, there was developed and implemented a software library for integration with a number of social networks. This paper describes the problem, solution architecture and technical details of its implementation supported by the results of simulation and real data analysis for a number of popular social networks. Keywords: social networks, big data, integration, architecture.  1. Introduction Modern social media can be treated as a major source of Big Data that describes the process of users’ interaction and various information exchanges. Analysis of this data turns out to become a complicated technical problem: it is required to integrate with multiple social media for data import, associate separate profiles of the same users in different networks, match the facts of their interaction across the real events and derive basic trends and deviations. To solve this problem there was developed a model of social media user behavior and a based on it software solution that provides capabilities for social networks analysis and simulation. 2. State of the Art New opportunities of interaction in virtual environments allow Internet users to exchange the ideas immediately. At the same time everybody needs to obtain and process lots of incoming events. This process can be described by the modern principles of distributed simulation and decision-making support powered by multi-agent technology [1]. The virtual world of social media should be treated as a complex network of continuously running and co-evolving intelligent agents. Such solutions are based on holons paradigm and bio-inspired approach [2], which requires development of new methods Науки о данных                           A.V. Ivaschenko et al. IV Международная конференция и молодёжная школа «Информационные технологии и нанотехнологии» (ИТНТ-2018)      2547 and tools for supporting fundamental mechanisms of self-organization and evolution similar to living organisms (colonies of ants, swarms of bees, etc) [3].  As for the human beings represented by actors or agents, social network user should consider a combination of human and time factors. Interaction of customers and service providers powered by intermediary services generate and can be characterized by a big number of events that form Big Data and require modern technologies for its analysis [4].  Modeling the Internet users’ behavior can be based on the modern principles of knowledge representation in the form of Ontologies [5]. These concepts allow formalizing self-organization and semantics, which is advantageous for abstract description of social concepts and their interaction in technical applications.  In the context of this paper there should be mentioned the papers on Internet development strategies [7], virtual communities and social networks studies [8 – 9]. Despite the successful application of mathematical statistics used to cluster and generalize the user’s behavior the problem of Big Data analysis of social networks remains open. This happens due to a necessity to personalize user activity models and understand individual features of human behavior.   Our experience in the area of integrated information space development and its users’ behavior analysis [10 – 12] can be used to build a software solution to derive basic trends in social media and provide intelligent functionality for social media big data analysis. The proposed abstract model and solution vision are given below. 3. Abstract Model Let us present a community of Internet users by iu , where 1 \ .. ui N=  – a number of users. The activity of users information exchange can be presented by posts, comments or messages jp , where 1 \ .. wj N=  – an absolute number of an informational object. Post generation is an event  ( )0, ,, ,i j i j i jg u p t= .  (1) Issue or processing of an information object can be presented by an event , ,i j ke that can be characterized by the combination of user, focus, and time:  ( )( ) { }, , . , ,, , , 0,1i j k j i i k i j ke e p u f t= = ,  (2) where focus ,i kf  presents the current user interest and can be described by a tag cloud, which is a set of pairs: ( ){ }, , ,,i k n n k i kf wτ= ,  (3) where nτ  is a tag (keyword) with weight ,n kw .  The sequence of interdependent user focuses represents the evolution of the user’s interest.  Each user has own ontology that forms the basis of his perception. It changes with time under the influence of learning and forgetting the information (presented by posts, comments or messages) and can be presented by a chain of contexts:  ( ){ }, , ,,i m l l m i mc wτ ′ ′= .  (4) This change is correlated with user focus. The focus cannot be considered new in order to provide positive perception, and at the same time it is not equal to the context to be able to excite interest. Considering this correlation let us synchronize the context and focus changes: ( )( ) { }, , . , ,, , , 0,1i j m k i i m i j me e p u c t′ ′= = .  (5)  The statements (2) and (5) are Boolean variables, which mean that appearance or perception of a post, comment or message does not guarantee changes in focus and context.  Events (2) and (5) can be used for analysis. One of the possible implementations is presented below. Study of the user’s focus and context trends allows identifying tendencies, variations and iterations that form the patterns of user creativity. Науки о данных                           A.V. Ivaschenko et al. IV Международная конференция и молодёжная школа «Информационные технологии и нанотехнологии» (ИТНТ-2018)      2548  In case new informational proposal remain suspended and does not make any effect over the user focus, this means that the user does not see any interest. Possible reasons are concerned with context: additional education is needed to provoke such interest. On the other side, lots of changes of the user context indicate the search for a stable interest that should be proposed for the user at a certain time.  Context and focus can be also influenced by negative intervention. In order to manage the user focus there can be generated a series of repeated affections partially covering the actual context and the targeted interest. Such patterns can also be identified applying cross-correlation analysis to the proposed model, which helps identification and resistance to negative informational influence. 4. Solution Architecture The proposed approach is based on simulation of focus and context. It was implemented in multi agent architecture, which is presented in Figure 1. Under the bounds of our proposed architecture we provide profile descriptor, post generator and navigator. These are methods generated and used to simulate real activity of users in the social networks.  Figure. 1. Solution multi-agent architecture.   Post generator used to create posts according to predefined logic. Navigator is used to process incoming data which is described by network and can be presented like a sorted graph there the nodes are informational objects, for example web sites, documents, posts, comments, and the links are references between these objects. Each object can refer to several other objects and documents; and the navigator according to all predefined logic decides which link to go.  In addition to navigator and post generator the informational frames are provided under the multi-agent architecture that correspond to a predefined above focus and context concepts. Focus is used to represent the current interest of Internet users. Context is used to formalize informational space in which the agent performs its negotiating activity.  Based on the provided model an algorithm has been developed for social media big data analysis. The model is used to formalize the social media user and integrate the analytical software with various social media open for data import and analysis.  This algorithm consists of 2 stages:   1. Calculation of the sample frequency vector for all users and development of the standard deviation vector for a variety of users.  You need to select topics and convert them to a view  ,1,ii pTN    ,  (6) Науки о данных                           A.V. Ivaschenko et al. IV Международная конференция и молодёжная школа «Информационные технологии и нанотехнологии» (ИТНТ-2018)      2549 where p  is an hour of publication, ,i tN  is a number of users that posted on a certain topic in a time period p , iT is an identificator of a topic.  Then process the received pairs and calculate the amount  0 ,1,iik k pTN=    ∑   (7) and calculate the standard deviation for each pair kjσ ;  The obtained values are divided by the period.  2. Calculating the deviation metric for a particular user.  You need to select topics and convert them to a key-value view.  After that you need to process the data pairs and count the sum of topics with the same key;  0, 3 ,1, 3i ii iσσ∆ ≤ ∆ >  (8)  Count the deviation of a particular user, summarize the deviations of a particular user, divide the sum of the deviations by the number of topics (n) for a particular user and generate the resulting CSV-file in the form of a table with user data and information of standard deviation of this user.  One of the main features of Internet users’ activity online that should be considered in the explored scope is mutual influence of contexts and focuses of communicating peers. This factor makes it possible to introduce the control loop: in addition to web content semantic analysis the platform starts to manage the users interest based on focus identification and context feedback.  This information is being collected in social networks and has all necessary details to get actual estimations. Still in this case it is required to provide integration with social networks and the data being processed contain tons of subjective assessments and perceptions. Online libraries and professional communities are more neutral. For example, Wikipedia enforces various groups of authors to update the articles targeting maximum objectivity. Analysis of this data can help adequate identification of significant trends of consumers’ focus identification that can be practically used e.g. for marketing and product placement.  Activation method is used to simulate multi agent activities in real time. The special agent dispatcher will call all the agents by using this activation method and after be activated each time an agent generates the time series period according to some distribution rule. Agent generates time series of  navigation calls and post generation methods. At these stages, we solve the proposed navigation and generation in such a way that we can model post-writers or readers and introduce some specific patterns of online activity, for example, the agent can be more active at night, or we can use some time frame for high / low activity.  Focus and contexts update the results of real agent behavior based on the influents of informational objects. We can generate focus and contexts according to our goals and in case we want agent to behave in a sort of specific way we introduce this control directly and formulate focus and context the agent will do that you want.   This approach allows simulating this influence and is introduced in the system. In this case we need to analyze the focus and context changes of the agent during the period of time. On the basis of analysis we introduce changes in focus by generating informational objects inside this network. This can be done in real systems using the contexts based advertisements. We can generate just the objects with certain informational context, which can be described by tag clouds.  The introduced architecture can be used to simulate online users in social networks and model realistic Internet behavior. In the area of simulation, practical application is generation of cognitive patterns of collective behavior based on self-organization. In this case the agent should be simple and the logic of focus and context should be close to very simple but generic behavior. This logic can correspond to know real users of social networks but it can represent some generalized behavior and the community of agent. Such behavior can be used to study and develop some visual cases. Науки о данных                           A.V. Ivaschenko et al. IV Международная конференция и молодёжная школа «Информационные технологии и нанотехнологии» (ИТНТ-2018)      2550  In another case it is implemented as a sort of a frame, using which the algorithms of syntactical analysis or other large data analysis can look onto the real world of social networks and filter the data for intelligent study. 5. Implementation To implement the proposed approach there was developed a software solution for social media focus identification based on knowledge discovery and Big Data analysis.  The solution can integrate with various data sources, pick out concepts, generate tag clouds for contexts and focuses and process their changes in time. Solution implementation architecture is presented in Fig. 2. The data imported from social networks is captured in database and can be processed either in real time or in batch mode.  Figure 2. Integration model.   Crawler addresses asynchronously to a web service with requests for data from social networks. After receiving the request, the web service starts processing it. Next, the web service accesses the integrator, which starts downloading data from the public social media API in different object models and converts it into one common model a  RDF / XML file type, storing the data in the form of a crawler to receive the data by the single block to transfer the already downloaded ones.  Then in the background, i.e. in a mode where there is no need to control the data unloading process, the integrator automatically continues the embedded process and uploads the data to the database and uses Apache JENA to generate RDF / XML files that will be transferred to the first crawler address.  The described model, software solution and its implementation was probated and tested using a typical data set derived from a number of social networks. In addition to a real regular result set of social media users’ negotiation there was introduced a peak batch of posts generated by an online bot from it’s social media profile. Apart from the social media (getting no a prior knowledge of a data structure) the big data analysis algorithms was able to identify the online bot influence.  The results are presented in Fig. 3. Gray lines represent the annual trends of users’ activity. The peak identified on Aug 15 corresponds the Bot activity and can be easily identified by the agent comparing the behavior of previous periods. The described research results show that the proposed model can be used for online behavior analysis and identification of negative informational influence.  Figure 3. Bot activity identification. Науки о данных                           A.V. Ivaschenko et al. IV Международная конференция и молодёжная школа «Информационные технологии и нанотехнологии» (ИТНТ-2018)      2551 6. Conclusion As shown above, the proposed model allows solve the complicated technical problem. It allows to integrate with multiple social media by converting data from different public API into the one common object model for import and to search matching the facts of interaction across the real events and derive basic trends and deviations based on analysing of Internet users activity considering a combination of human and time factors.  7. Acknowledgment This work was supported by the Federal Agency of Scientific Organizations (agreement No 007-GZ/Ch3363/26). 8. References [1] Wooldridge, M. An introduction to multi-agent systems / M. Wooldridge. – Chichester: John Wiley and Sons, 2002. – 340 p. [2] Leitao, P. Holonic rationale and self-organization on design of complex evolvable systems / P. Leitao // HoloMAS, LNAI. – 2009. – Vol. 5696. – P. 1-12. [3] Gorodetskii, V. Self-organization and multiagent systems: I. Models of multiagent self-organization / V. Gorodetskii // Journal of Computer and Systems Sciences International. – 2012. – Vol. 51(2). – P. 256-281. [4] Bessis, N. Big Data and Internet of Things: A roadmap for smart environments / N. Bessis, C. Dobre. – Berlin: Springer, 2014. – 450 p. [5] Mouromtsev, D. The simple, web-based tool for visualization and sharing of semantic data and ontologies / D. Mouromtsev, D. Pavlov, Y. Emelyanov, A. Morozov, D. Razdyakonov, M. Galkin // CEUR Workshop Proceedings. – 2015. – Vol. 1486. – P. 77. [6] One Internet. Global commission on Internet Governance. Report, 2016. – Access mode: https://www.cigionline.org/initiatives/global-commission-internet-governance (01.11.2017). [7] Balakrishnan, H. Discovering communities in complex networks / H. Balakrishnan, N. Deo // Proceedings of the 44th annual Southeast regional conference, 2006. – P. 280-285. [8] Wei, W. Exploring Characteristics of Suspended Users and Network Stability on Twitter / W. Wei, K. Joseph, H. Liu, K. Carley // Social Network Analysis and Mining, 2016. – P. 6-51. [9] Kadushin, C. Understanding social networks: theories, concepts, and findings / C. Kadushin. – Oxford: Oxford University Press, 2012. – 264 p. [10] Ivaschenko, A. Multi-agent solution for business processes management of 5PL transportation provider / A. Ivaschenko // Lecture Notes in Business Information Processing. – 2014. – Vol. 170. – P. 110-120. [11] Ivaschenko, A. Self-mediator software for sensor networks / A. Ivaschenko, A. Minaev, M. Spodobaev // Proceedings of the 2015 International Siberian Conference on Control and Communications (SIBCON), 2015. – P.1-4. [12] Ivaschenko, A. Agent-based outsourcing solution for agency service management / A. Ivaschenko, A. Lednev, A. Diyazitdinova, P. Sitnikov // Lecture Notes in Networks and Systems. – 2016. – Vol.  16. – P. 204-215. 

Integration Issues of Big Data Analysis on Social Networks

http://repo.ssau.ru/bitstream/Informacionnye-tehnologii-i-nanotehnologii/Integration-Issues-of-Big-Data-Analysis-on-Social-Networks-69652/1/Integration%20Issues%20of%20Big%20Data%20Analysis%20on%20Social%20Networks.pdf

Integration Issues of Big Data Analysis on Social Networks

Abstract

Similar works

Full text

Available Versions

Samara University