Research on multi-modal sentiment feature learning of social media content

社交媒体已成为现代社会舆论交流和信息传递的主要平台。针对社交媒体的情感分析对于舆论监控、商业产品导向和股市预测等都具有重大应用价值。但社交媒体内容的多模态性（文本、图片等）让传统的单模态情感分析方法面临许多局限，多模态情感分析技术对跨媒体内容的理解与分析具有重大的理论价值。 多模态情感分析区别于单模态方法的关键问题在于，如何综合利用形态各异的多模态情感信息，来获取整体的情感倾向性，同时考虑单个模态本身在情感表达上的性质。针对该问题，利用社交媒体上的多模态内容在情感表达上所具有的关联性、抽象层级性的特点，提出了一套面向社交媒体的多模态情感特征学习与融合方法，实现多模态情感分析，主要内容和创新点...Social media has become a main platform of public communication and information transmission. Therefore, social media sentiment analysis has great application values in many fields, such as public opinion monitoring, production marking, stock forecasting and so on. But the multi-modal characteristic of social media content (e.g. texts and images) significantly challenges traditional text-based sen...学位：工学硕士院系专业：信息科学与技术学院_模式识别与智能系统学号：3152013115327

Memristor-Based Hierarchical Attention Network for Multimodal Affective Computing in Mental Health Monitoring

10.13039/501100001809-National Natural Science Foundation of China (Grant Number: 62001149 and U1909201); 10.13039/501100004731-Natural Science Foundation of Zhejiang Province (Grant Number: LQ21F010009)

Computational dynamic support model for social support assignments around stressed individuals among graduate students

Configuring the best resources for optimal overall performance is one of the challenging topics in Computer Science domains. Within the domain of intelligent social support assignment applications to help individuals with stress, it requires important aspects of configuring a possible set of input and parameters to obtain optimal solutions from both computational support provider and recipient models. However, the existing configuration algorithms are often randomized and static. Thus, their results can vary significantly between multiple runs. In the context of social support perspectives, the assigned support may not sufficient or cause a burden to the providers. Hence, this study aims to develop the dynamic configuration algorithm to provide an optimal support assignment based on information generated from both social support recipient and provision computational models. The computational models that simulate support providers and recipients behaviours were developed to generate several simulated patterns. These models explain the dynamics of support seeking and provision behaviours and were evaluated using equilibria analysis and automatic logical verification approaches for 14 selected empirical cases. Later, the dynamic configuration algorithm was designed to utilize possible support assignments based on support provision requirements. The algorithm complexity analysis was used to measure the execution time in the worst case. Finally, a prototype was developed and validated with 30 graduate students. This study allows to explore computational analysis in explicit comprehension of how seeking and giving support process can be obtained at different case conditions. Also, the study explicitly shows the psychological stress of support recipient can be reduced after the dynamic configuration algorithm process assigned selected social support providers from social support network members. Furthermore, this study provides an alternative method for software engineers in intelligent stress management systems to integrate social support-based concepts as one of the mechanisms in addressing the support of an individual with cognitive related stress

Tackling Mental Health by Integrating Unobtrusive Multimodal Sensing

Mental illness is becoming a major plague in modern societies and poses challenges to the capacity of current public health systems worldwide. With the widespread adoption of social media and mobile devices, and rapid advances in artificial intelligence, a unique opportunity arises for tackling mental health problems. In this study, we investigate how users’ online social activities and physiological signals detected through ubiquitous sensors can be utilized in realistic scenarios for monitoring their mental health states. First, we extract a suite of multimodal time-series signals using modern computer vision and signal processing techniques, from recruited participants while they are immersed in online social media that elicit emotions and emotion transitions. Next, we use machine learning techniques to build a model that establishes the connection between mental states and the extracted multimodal signals. Finally, we validate the effectiveness of our approach using two groups of recruited subjects