Multimodal Short Video Rumor Detection System Based on Contrastive Learning

With short video platforms becoming one of the important channels for news sharing, major short video platforms in China have gradually become new breeding grounds for fake news. However, it is not easy to distinguish short video rumors due to the great amount of information and features contained in short videos, as well as the serious homogenization and similarity of features among videos. In order to mitigate the spread of short video rumors, our group decides to detect short video rumors by constructing multimodal feature fusion and introducing external knowledge after considering the advantages and disadvantages of each algorithm. The ideas of detection are as follows: (1) dataset creation: to build a short video dataset with multiple features; (2) multimodal rumor detection model: firstly, we use TSN (Temporal Segment Networks) video coding model to extract video features; then, we use OCR (Optical Character Recognition) and ASR (Automatic Character Recognition) to extract video features. Recognition) and ASR (Automatic Speech Recognition) fusion to extract text, and then use the BERT model to fuse text features with video features (3) Finally, use contrast learning to achieve distinction: first crawl external knowledge, then use the vector database to achieve the introduction of external knowledge and the final structure of the classification output. Our research process is always oriented to practical needs, and the related knowledge results will play an important role in many practical scenarios such as short video rumor identification and social opinion control

소셜 미디어 속 루머 탐지를 위한 그래프 합성곱 신경망과 어텐션 메커니즘

학위논문 (석사) -- 서울대학교 대학원 : 공과대학 컴퓨터공학부, 2020. 8. 김종권.Social media has been a great disseminator for new information and thoughts. Due to its accessibility of sharing information, however, social media has also become an ideal platform for propagations of rumors, fake news, and misinformation. Rumors on social media not only mislead the users of online but also affects the real world immensely. Thus, detecting the rumors and preventing their spread became an essential task. Previous learning-based rumor detection methods adopted to use contents, users, or propagation features of rumors. However, the methods are limited to represent rumor propagation as static graphs, which arent optimal for capturing the dynamic information of the rumors. In this study, we propose a novel graph convolutional networks with attention mechanism model named, Dynamic GCN, for rumor detection. We first represent rumor posts with their responsive posts as dynamic graphs. The temporal information is used to generate a sequence of graph snapshots. The representation learning on graph snapshots with attention mechanism captures both structural and temporal information of rumor spreads. The conducted experiments on two real-world datasets demonstrate that our model, Dynamic GCN, achieves superior results over the state-of-the-art models in the rumor detection task.소셜 미디어는 강력한 정보 전달력을 가진 매체로 새로운 정보와 생각의 전파 창구이다. 소셜 미디어의 특징인 접근성은 때론 루머, 가짜 뉴스, 잘못된 정보의 전파에서도 이상적인 플랫폼이 된다. 소셜 미디어 속 루머는 온라인 사용자를 오도할 뿐만 아니라 때론 현실 세계에도 큰 영향을 미친다. 따라서, 루머를 탐지하고 그 전파를 막는 노력이 요구된다. 기존의 루머 탐지 방법은 루머의 내용, 사용자, 또는 전파 과정의 정보를 특성으로 이용한다. 이러한 방법은 루머의 전파를 정적 그래프로 표현하며 그 구조적 특성을 이용한다. 하지만 이는 루머의 동적 특성을 포착하지 못한다는 한계를 가지고 있다. 본 연구에서는 그래프 합성곱 신경망 (graph convolutional networks: GCN)과 어텐션 메커니즘 (attention mechanism)을 활용한 동적 그래프 합성곱 신경망 (Dynamic GCN) 루머 탐지 모델을 제시한다. 먼저, 소셜 미디어 속 루머 게시글들 (posts) 과 그의 답장이 되는 글들(responsive posts)을 이용하여 루머의 전파 과정을 정적 그래프로 표현한다. 시간 정보를 통해 전파 과정을 포함하고 있는 정적 그래프의 집합인 그래프 스냅숏 (graph snapshot) 시퀀스 (sequence)를 만들게 된다. 어텐션 메커니즘을 활용한 그래프 스냅숏 표현 학습은 루머 전파의 구조적 시간적 정보를 모두 효과적으로 반영한다. 실제 트위터 데이터를 이용한 실험을 통하여 제시된 모델의 성능이 다른 비교 모델들보다 높음을 확인할 수 있었다.Chapter I Introduction 1 Chapter II Related Work 5 2.1 Rumor Detection 5 2.2 Graph Convolutional Networks 6 2.3 Learning Sequences & Attention Mechanism 7 Chapter III Problem Definition 9 Chapter IV Dynamic GCN with Attention Mechanism 11 4.1 Snapshot Generation 13 4.2 Graph Convolutional Networks 14 4.3 Readout Layer 15 4.4 Attention Mechanism 16 4.5 Training & Prediction 17 Chapter V Experiments 18 5.1 Datasets 18 5.2 Baselines 20 5.3 Experimental Setup & Implementation Details 21 5.4 Performance Evaluations 24 5.5 Ablation Study 25 Chapter VI Conclusion 30 Bibliography 31 초 록 40Maste

Towards Evaluating Veracity of Textual Statements on the Web

The quality of digital information on the web has been disquieting due to the absence of careful checking. Consequently, a large volume of false textual information is being produced and disseminated with misstatements of facts. The potential negative influence on the public, especially in time-sensitive emergencies, is a growing concern. This concern has motivated this thesis to deal with the problem of veracity evaluation. In this thesis, we set out to develop machine learning models for the veracity evaluation of textual claims based on stance and user engagements. Such evaluation is achieved from three aspects: news stance detection engaged user replies in social media and the engagement dynamics. First of all, we study stance detection in the context of online news articles where a claim is predicted to be true if it is supported by the evidential articles. We propose to manifest a hierarchical structure among stance classes: the high-level aims at identifying relatedness, while the low-level aims at classifying, those identified as related, into the other three classes, i.e., agree, disagree, and discuss. This model disentangles the semantic difference of related/unrelated and the other three stances and helps address the class imbalance problem. Beyond news articles, user replies on social media platforms also contain stances and can infer claim veracity. Claims and user replies in social media are usually short and can be ambiguous; to deal with semantic ambiguity, we design a deep latent variable model with a latent distribution to allow multimodal semantic distribution. Also, marginalizing the latent distribution enables the model to be more robust in relatively smalls-sized datasets. Thirdly, we extend the above content-based models by tracking the dynamics of user engagement in misinformation propagation. To capture these dynamics, we formulate user engagements as a dynamic graph and extract its temporal evolution patterns and geometric features based on an attention-modified Temporal Point Process. This allows to forecast the cumulative number of engaged users and can be useful in assessing the threat level of an individual piece of misinformation. The ability to evaluate veracity and forecast the scale growth of engagement networks serves to practically assist the minimization of online false information’s negative impacts