Sensing Human Sentiment via Social Media Images: Methodologies and Applications

abstract: Social media refers computer-based technology that allows the sharing of information and building the virtual networks and communities. With the development of internet based services and applications, user can engage with social media via computer and smart mobile devices. In recent years, social media has taken the form of different activities such as social network, business network, text sharing, photo sharing, blogging, etc. With the increasing popularity of social media, it has accumulated a large amount of data which enables understanding the human behavior possible. Compared with traditional survey based methods, the analysis of social media provides us a golden opportunity to understand individuals at scale and in turn allows us to design better services that can tailor to individuals’ needs. From this perspective, we can view social media as sensors, which provides online signals from a virtual world that has no geographical boundaries for the real world individual's activity. One of the key features for social media is social, where social media users actively interact to each via generating content and expressing the opinions, such as post and comment in Facebook. As a result, sentiment analysis, which refers a computational model to identify, extract or characterize subjective information expressed in a given piece of text, has successfully employs user signals and brings many real world applications in different domains such as e-commerce, politics, marketing, etc. The goal of sentiment analysis is to classify a user’s attitude towards various topics into positive, negative or neutral categories based on textual data in social media. However, recently, there is an increasing number of people start to use photos to express their daily life on social media platforms like Flickr and Instagram. Therefore, analyzing the sentiment from visual data is poise to have great improvement for user understanding. In this dissertation, I study the problem of understanding human sentiments from large scale collection of social images based on both image features and contextual social network features. We show that neither visual features nor the textual features are by themselves sufficient for accurate sentiment prediction. Therefore, we provide a way of using both of them, and formulate sentiment prediction problem in two scenarios: supervised and unsupervised. We first show that the proposed framework has flexibility to incorporate multiple modalities of information and has the capability to learn from heterogeneous features jointly with sufficient training data. Secondly, we observe that negative sentiment may related to human mental health issues. Based on this observation, we aim to understand the negative social media posts, especially the post related to depression e.g., self-harm content. Our analysis, the first of its kind, reveals a number of important findings. Thirdly, we extend the proposed sentiment prediction task to a general multi-label visual recognition task to demonstrate the methodology flexibility behind our sentiment analysis model.Dissertation/ThesisDoctoral Dissertation Computer Science 201

Econometrics meets sentiment : an overview of methodology and applications

The advent of massive amounts of textual, audio, and visual data has spurred the development of econometric methodology to transform qualitative sentiment data into quantitative sentiment variables, and to use those variables in an econometric analysis of the relationships between sentiment and other variables. We survey this emerging research field and refer to it as sentometrics, which is a portmanteau of sentiment and econometrics. We provide a synthesis of the relevant methodological approaches, illustrate with empirical results, and discuss useful software

Measuring relative opinion from location-based social media: A case study of the 2016 U.S. presidential election

Social media has become an emerging alternative to opinion polls for public opinion collection, while it is still posing many challenges as a passive data source, such as structurelessness, quantifiability, and representativeness. Social media data with geotags provide new opportunities to unveil the geographic locations of users expressing their opinions. This paper aims to answer two questions: 1) whether quantifiable measurement of public opinion can be obtained from social media and 2) whether it can produce better or complementary measures compared to opinion polls. This research proposes a novel approach to measure the relative opinion of Twitter users towards public issues in order to accommodate more complex opinion structures and take advantage of the geography pertaining to the public issues. To ensure that this new measure is technically feasible, a modeling framework is developed including building a training dataset by adopting a state-of-the-art approach and devising a new deep learning method called Opinion-Oriented Word Embedding. With a case study of the tweets selected for the 2016 U.S. presidential election, we demonstrate the predictive superiority of our relative opinion approach and we show how it can aid visual analytics and support opinion predictions. Although the relative opinion measure is proved to be more robust compared to polling, our study also suggests that the former can advantageously complement the later in opinion prediction

A Review on MAS-Based Sentiment and Stress Analysis User-Guiding and Risk-Prevention Systems in Social Network Analysis

[EN] In the current world we live immersed in online applications, being one of the most present of them Social Network Sites (SNSs), and different issues arise from this interaction. Therefore, there is a need for research that addresses the potential issues born from the increasing user interaction when navigating. For this reason, in this survey we explore works in the line of prevention of risks that can arise from social interaction in online environments, focusing on works using Multi-Agent System (MAS) technologies. For being able to assess what techniques are available for prevention, works in the detection of sentiment polarity and stress levels of users in SNSs will be reviewed. We review with special attention works using MAS technologies for user recommendation and guiding. Through the analysis of previous approaches on detection of the user state and risk prevention in SNSs we elaborate potential future lines of work that might lead to future applications where users can navigate and interact between each other in a more safe way.This work was funded by the project TIN2017-89156-R of the Spanish government.Aguado-Sarrió, G.; Julian Inglada, VJ.; García-Fornes, A.; Espinosa Minguet, AR. (2020). A Review on MAS-Based Sentiment and Stress Analysis User-Guiding and Risk-Prevention Systems in Social Network Analysis. Applied Sciences. 10(19):1-29. https://doi.org/10.3390/app10196746S1291019Vanderhoven, E., Schellens, T., Vanderlinde, R., & Valcke, M. (2015). Developing educational materials about risks on social network sites: a design based research approach. Educational Technology Research and Development, 64(3), 459-480. doi:10.1007/s11423-015-9415-4Teens and ICT: Risks and Opportunities. Belgium: TIRO http://www.belspo.be/belspo/fedra/proj.asp?l=en&COD=TA/00/08Risks and Safety on the Internet: The Perspective of European Children: Full Findings and Policy Implications From the EU Kids Online Survey of 9–16 Year Olds and Their Parents in 25 Countries http://eprints.lse.ac.uk/33731/Vanderhoven, E., Schellens, T., & Valcke, M. (2014). Educating teens about the risks on social network sites. An intervention study in Secondary Education. Comunicar, 22(43), 123-132. doi:10.3916/c43-2014-12Christofides, E., Muise, A., & Desmarais, S. (2012). Risky Disclosures on Facebook. Journal of Adolescent Research, 27(6), 714-731. doi:10.1177/0743558411432635George, J. M., & Dane, E. (2016). Affect, emotion, and decision making. Organizational Behavior and Human Decision Processes, 136, 47-55. doi:10.1016/j.obhdp.2016.06.004Thelwall, M. (2017). TensiStrength: Stress and relaxation magnitude detection for social media texts. Information Processing & Management, 53(1), 106-121. doi:10.1016/j.ipm.2016.06.009Thelwall, M., Buckley, K., Paltoglou, G., Cai, D., & Kappas, A. (2010). Sentiment strength detection in short informal text. Journal of the American Society for Information Science and Technology, 61(12), 2544-2558. doi:10.1002/asi.21416Shoumy, N. J., Ang, L.-M., Seng, K. P., Rahaman, D. M. M., & Zia, T. (2020). Multimodal big data affective analytics: A comprehensive survey using text, audio, visual and physiological signals. Journal of Network and Computer Applications, 149, 102447. doi:10.1016/j.jnca.2019.102447Zhang, C., Zeng, D., Li, J., Wang, F.-Y., & Zuo, W. (2009). Sentiment analysis of Chinese documents: From sentence to document level. Journal of the American Society for Information Science and Technology, 60(12), 2474-2487. doi:10.1002/asi.21206Lu, B., Ott, M., Cardie, C., & Tsou, B. K. (2011). Multi-aspect Sentiment Analysis with Topic Models. 2011 IEEE 11th International Conference on Data Mining Workshops. doi:10.1109/icdmw.2011.125Nasukawa, T., & Yi, J. (2003). Sentiment analysis. Proceedings of the international conference on Knowledge capture - K-CAP ’03. doi:10.1145/945645.945658Borth, D., Ji, R., Chen, T., Breuel, T., & Chang, S.-F. (2013). Large-scale visual sentiment ontology and detectors using adjective noun pairs. Proceedings of the 21st ACM international conference on Multimedia - MM ’13. doi:10.1145/2502081.2502282Deb, S., & Dandapat, S. (2019). Emotion Classification Using Segmentation of Vowel-Like and Non-Vowel-Like Regions. IEEE Transactions on Affective Computing, 10(3), 360-373. doi:10.1109/taffc.2017.2730187Deng, J., Zhang, Z., Marchi, E., & Schuller, B. (2013). Sparse Autoencoder-Based Feature Transfer Learning for Speech Emotion Recognition. 2013 Humaine Association Conference on Affective Computing and Intelligent Interaction. doi:10.1109/acii.2013.90Nicolaou, M. A., Gunes, H., & Pantic, M. (2011). Continuous Prediction of Spontaneous Affect from Multiple Cues and Modalities in Valence-Arousal Space. IEEE Transactions on Affective Computing, 2(2), 92-105. doi:10.1109/t-affc.2011.9Hossain, M. S., Muhammad, G., Alhamid, M. F., Song, B., & Al-Mutib, K. (2016). Audio-Visual Emotion Recognition Using Big Data Towards 5G. Mobile Networks and Applications, 21(5), 753-763. doi:10.1007/s11036-016-0685-9Zhou, F., Jianxin Jiao, R., & Linsey, J. S. (2015). Latent Customer Needs Elicitation by Use Case Analogical Reasoning From Sentiment Analysis of Online Product Reviews. Journal of Mechanical Design, 137(7). doi:10.1115/1.4030159Ceci, F., Goncalves, A. L., & Weber, R. (2016). A model for sentiment analysis based on ontology and cases. IEEE Latin America Transactions, 14(11), 4560-4566. doi:10.1109/tla.2016.7795829Vizer, L. M., Zhou, L., & Sears, A. (2009). Automated stress detection using keystroke and linguistic features: An exploratory study. International Journal of Human-Computer Studies, 67(10), 870-886. doi:10.1016/j.ijhcs.2009.07.005Feldman, R. (2013). Techniques and applications for sentiment analysis. Communications of the ACM, 56(4), 82-89. doi:10.1145/2436256.2436274Schouten, K., & Frasincar, F. (2016). Survey on Aspect-Level Sentiment Analysis. IEEE Transactions on Knowledge and Data Engineering, 28(3), 813-830. doi:10.1109/tkde.2015.2485209Ji, R., Cao, D., Zhou, Y., & Chen, F. (2016). Survey of visual sentiment prediction for social media analysis. Frontiers of Computer Science, 10(4), 602-611. doi:10.1007/s11704-016-5453-2Li, L., Cao, D., Li, S., & Ji, R. (2015). Sentiment analysis of Chinese micro-blog based on multi-modal correlation model. 2015 IEEE International Conference on Image Processing (ICIP). doi:10.1109/icip.2015.7351718Lee, P.-M., Tsui, W.-H., & Hsiao, T.-C. (2015). The Influence of Emotion on Keyboard Typing: An Experimental Study Using Auditory Stimuli. PLOS ONE, 10(6), e0129056. doi:10.1371/journal.pone.0129056Matsiola, M., Dimoulas, C., Kalliris, G., & Veglis, A. A. (2018). Augmenting User Interaction Experience Through Embedded Multimodal Media Agents in Social Networks. Information Retrieval and Management, 1972-1993. doi:10.4018/978-1-5225-5191-1.ch088Rosaci, D. (2007). CILIOS: Connectionist inductive learning and inter-ontology similarities for recommending information agents. Information Systems, 32(6), 793-825. doi:10.1016/j.is.2006.06.003Buccafurri, F., Comi, A., Lax, G., & Rosaci, D. (2016). Experimenting with Certified Reputation in a Competitive Multi-Agent Scenario. IEEE Intelligent Systems, 31(1), 48-55. doi:10.1109/mis.2015.98Rosaci, D., & Sarnè, G. M. L. (2014). Multi-agent technology and ontologies to support personalization in B2C E-Commerce. Electronic Commerce Research and Applications, 13(1), 13-23. doi:10.1016/j.elerap.2013.07.003Singh, A., & Sharma, A. (2017). MAICBR: A Multi-agent Intelligent Content-Based Recommendation System. Lecture Notes in Networks and Systems, 399-411. doi:10.1007/978-981-10-3920-1_41Villavicencio, C., Schiaffino, S., Diaz-Pace, J. A., Monteserin, A., Demazeau, Y., & Adam, C. (2016). A MAS Approach for Group Recommendation Based on Negotiation Techniques. Lecture Notes in Computer Science, 219-231. doi:10.1007/978-3-319-39324-7_19Rincon, J. A., de la Prieta, F., Zanardini, D., Julian, V., & Carrascosa, C. (2017). Influencing over people with a social emotional model. Neurocomputing, 231, 47-54. doi:10.1016/j.neucom.2016.03.107Aguado, G., Julian, V., Garcia-Fornes, A., & Espinosa, A. (2020). A Multi-Agent System for guiding users in on-line social environments. Engineering Applications of Artificial Intelligence, 94, 103740. doi:10.1016/j.engappai.2020.103740Aguado, G., Julián, V., García-Fornes, A., & Espinosa, A. (2020). Using Keystroke Dynamics in a Multi-Agent System for User Guiding in Online Social Networks. Applied Sciences, 10(11), 3754. doi:10.3390/app10113754Camara, M., Bonham-Carter, O., & Jumadinova, J. (2015). A multi-agent system with reinforcement learning agents for biomedical text mining. Proceedings of the 6th ACM Conference on Bioinformatics, Computational Biology and Health Informatics. doi:10.1145/2808719.2812596Lombardo, G., Fornacciari, P., Mordonini, M., Tomaiuolo, M., & Poggi, A. (2019). A Multi-Agent Architecture for Data Analysis. Future Internet, 11(2), 49. doi:10.3390/fi11020049Schweitzer, F., & Garcia, D. (2010). An agent-based model of collective emotions in online communities. The European Physical Journal B, 77(4), 533-545. doi:10.1140/epjb/e2010-00292-