Automatic fake news detection is an important, yet very challenging topic. Traditional methods using lexical features have only very limited success. This paper proposes a novel method to incorporate speaker profiles into an attention based LSTM model for fake news detection. Speaker profiles contribute to the model in two ways. One is to include them in the attention model. The other includes them as additional input data. By adding speaker profiles such as party affiliation, speaker title, location and credit history, our model outperforms the state-of-the-art method by 14.5% in accuracy using a benchmark fake news detection dataset. This proves that speaker profiles provide valuable information to validate the credibility of news articles

Long, Yunfei

English

University of Essex Research Repository

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 252–256,Taipei, Taiwan, November 27 – December 1, 2017 c©2017 AFNLPFake News Detection Through Multi-Perspective Speaker ProfilesYunfei Long1, Qin Lu1, Rong Xiang2, Minglei Li1 and Chu-Ren Huang31Department of Computing, The Hong Kong Polytechnic Universitycsylong,csluqin,csmli@comp.polyu.edu.hk2Advanced Micro Devices, Inc.(Shanghai)Rong.Xiang@amd.com3Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic Universitychuren.huang@polyu.edu.hkAbstractAutomatic fake news detection is an im-portant, yet very challenging topic. Tradi-tional methods using lexical features haveonly very limited success. This paperproposes a novel method to incorporatespeaker profiles into an attention basedLSTM model for fake news detection.Speaker profiles contribute to the model intwo ways. One is to include them in the at-tention model. The other includes them asadditional input data. By adding speakerprofiles such as party affiliation, speakertitle, location and credit history, our modeloutperforms the state-of-the-art method by14.5% in accuracy using a benchmark fakenews detection dataset. This proves thatspeaker profiles provide valuable informa-tion to validate the credibility of news ar-ticles.1 IntroductionFake news is written and published with the intentto mislead its readers in order to gain financially orpolitically, often with sensationalist, exaggerated,or patently false headlines attract readers’ atten-tion 1. Fake news is more dangerous than news-paper gaffes, especially in social media. One ofthe worst effect of fake news in China is an inci-dence after the Fukushima Daiichi nuclear disasterin 2011. A fake news in microbiology claims thatsalt can prevent radiation, but the nuclear disastermakes salt contaminated. This fake news triggeredpeople stockpiling table salt in China, resulted ina huge market disorder. 21https : //en.wikipedia.org/wiki/Fakenews2https://blogs.wsj.com/chinarealtime/2011/03/17/fearing-radiation-chinese-rush-to-buy-table-salt/Fake news detection is defined as the task ofcategorizing news along a continuum of veracitywith an associated measure of certainty (Conroyet al., 2015). Detecting fake news in social me-dia has been an extremely important, yet tech-nically very challenging problem. The difficultycomes partly from the fact that even human beingsmay have difficulty identifying between real newsand fake news. In one study, human judges, by arough measure of comparison, achieved only 50-63 % success rates in identifying fake news (Ru-bin, 2017).The most of fake news detection algorithmstry to linguistic cues (Feng and Hirst, 2013;Markowitz and Hancock, 2014; Ruchansky et al.,2017). Several successful studies on fake news de-tection have demonstrated the effectiveness of lin-guistic cue identification, as the language of truthnews is known to differ from that of fake news(Bachenko et al., 2008; Larcker and Zakolyuk-ina, 2012). For example, deceivers are likelyto use more sentiment words, more sense-basedwords (e.g., seeing, touching), and other-orientedpronouns, but less self-oriented pronouns. Com-pared to real news, fake news shows lower cog-nitive complexity and uses more negative emotionwords. However, the linguistic indicators of fakenews across different topics and media platformsare not well understood. Rubin (2015) points outthat there are many types of fake news, each withdifferent potential textual indicators. This indi-cates that using linguistic features is not only la-borious but also topic/media dependent domainknowledge, thus limiting the scalability of thesesolutions.In addition to lexical features, speaker profileinformation can be useful (Gottipati et al., 2013;Long et al., 2016). Speaker profiles, includingparty affiliations, job title of speaker, as well astopical information which can also be used to in-252dicate the credibility of a piece of news. For ex-ample, a conservative might neglect the impact ofclimate change, while a progressive might exag-gerate inequality. On some occasions, it is hard tomake fake claims like congressional inquiry. Butin other cases, the speaker tend to exaggerate factslike the campaign rally. For the study use profileinformation, Wang (2017) proposes a hybrid CNNmodel to detect fake news, which uses speakerprofiles as a part of the input data. Wang (2017)also made the first large scale fake news detec-tion benchmark dataset with speaker informationsuch as party affiliation, location of speech, job ti-tle, credit history as well as topic.The Long-Short memory network (LSTM), as aneural network model, is proven to work better forlong sentences (Tang et al., 2015). Attention mod-els are also proposed to weigh the importance ofdifferent words in their context. Current attentionmodels are either based on local semantic atten-tions (Yang et al., 2016) or user attentions(Chenet al., 2016).In this paper, we propose a model to incorpo-rate speaker profile into fake news detection. Pro-file information is used in two ways. The firstway includes profiles as an attention factor whilethe second one includes profiles as additional in-put data. Evaluations are conducted using thedataset provided by Wang (2017). Experimentalresults show that incorporating speaker profile canimprove performance dramatically. Our accuracycan reach 0.415 in the benchmark dataset, about14.5% higher than state-of-the-art hybrid CNNmodel.2 Proposed ModelOur proposed model uses LSTM model (Gers,2001) as the basic classifier. The attention mod-els and speaker profile information are added intoLSTM to form a hybrid model.Figure 1 shows the hybrid LSTM. Note that thefirst LSTM is for obtaining the representation ofnews articles. The speaker profile is used to con-struct two attention factors. One uses only thespeaker profile and the other uses topic informa-tion of the news articles. The second LSTM sim-ply uses speaker profiles to obtain the vector pre-sentations of speakers. The two representationsare then concatenated in the soft-max function forclassification.Let D be a collection of news and P be a col-lection of profiles. A piece of news, dk, dk ∈ D,includes both its text as a sentence, denoted by sk,and a speaker profile , denoted by pk, pk ∈ P .A sentence sk is formed by a sequence of wordssk = wk1wk2...wklk , where lk is the length of sk.The features of a wordwi ∈ Sk form a word vector~vwi with length N , ~vwi = [F1wi , F2wi ....FNwi ].Every wk ∈ sk and pk ∈ P are fed into the firstLSTM. The speaker based profile vector ~s and thetopic based profile vector ~t serve as the two atten-tion factors for sk. The output of the two LSTMmodels are ~sk and ~pk. Finally, ~sk and ~pk are con-catenated to form the representation ~dk. The finallayer projects ~dk onto the target space of L classlabels though a soft-max layer.A LSTM model has five types of gates in eachnode i represented by five vectors including an in-put gate ~ii, a forget gate ~fi, an output gate ~oi, acandidate memory cell gate ~c′i, and a memory cellgate ~ci. ~fi and ~oi are used to indicate which valueswill be updated, either to forget or to keep. ~c′i and~ci are used to keep the candidate features and theactual accepted features, respectively.Each node i corresponds to each word wi in agiven sentence Sk, represented by its word embed-ding ~wi. The LSTM cell state ~ci and the hiddenstate ~hSk:wi can be updated in two steps. In thefirst step, the previous hidden state ~hSk:wi−1 uses ahyperbolic function to form ~c′i as defined below.~c′i = tanh(Wˆc ∗ [~hSk:wi−1 ∗ ~wi] + bˆ), (1)where Wˆc is a parameter matrix, ~hSk:wi−1 is theprevious hidden state and ~wi is the word vector. bˆis the regularization parameter matrix. In the sec-ond step, ~ci is updated by ~c′i and its previous state~ci−1 according to the below formula:~ci = ~fi  ~ci−1 + ~ii  ~c′i. (2)The hidden state of wi can be obtained by~hS:wi = ~oitanh(~fi  ~ci). (3)The forget gate ~fi is for keeping the long termmemory. A series of hidden states ~h1~h2...~hi canserve as input to the average pooling layer to ob-tain the representation ~sk. ~pk, the representationof pk, can be obtained similarly through the sameLSTM model. Details will not be repeated here.Similar to other attention models, speaker pro-file based attention factors are included in the253first LSTM for each text, sk. Rather than feed-ing speaker profiles as hidden states to an averagepooling layer, we use attention mechanism whichuses a weighting scheme to indicate importance ofdifferent words to the meaning of the news.We use speaker profile and topic information forweight training, which are represented by contin-uous and real-valued vectors ~s and ~t, respectively.Let αwi denote the weight to measure the impor-tance of wi in sk. The attention weight αwi foreach hidden state can be defined as:αwi =exp(e(~hSk:wi , ~s,~t))∑lik=1 exp(e(~hSk:wi , ~s,~t))(4)where e is score function to indicate the impor-tance of words defined by:exp(e(~hSk:wi , ~s,~t) = vT tanh(WˆH~hSk:wi+Wˆs~s+ Wˆt~t+~b)(5)where WH ,WS ,WT are weight matrices, v isweight vector and vT denotes its transpose. Thismodel can train speaker vector ~s and topic vector~t at the same time.Formally, the enhanced sentence representation~sk is a weighted sum of hidden states as Formula6~sk =∑tαwi~hSk:wi . (6)Similarly, we use the second LSTM model to getthe representation of profile ~pk. The final newsrepresentation ~dk is computed by concatenating ~pkand ~sk, represented in Formula 7:~dk = ~sk ⊕ ~pk. (7)In LSTM model, we use a hidden layer to projectthe final news vector ~dfk through a hyperbolic func-tion.~dfk = tanh(Wˆh~dk + bˆh), (8)where Wˆh is the weight matrix of the hidden layerand bˆh is the regularization matrix.Finally, prediction for any label lL obtained bythe soft-max function is defined as:P (y = l|~dfk) =e~dfTk~Wl∑Ll=1 e~dfTk~Wl(9)where ~Wl is the soft-max weight for each label.Figure 1: Hybrid fake news detection model withattention mechanismStatistics Num Labels NumTraining Set 10,269 Pants on fire 1,050Develop Set 1,284 False 2,511Testing Size 1,283 Barely-true 2,108Democrats 4,150 Half-true 2,638Republicans 5,687 Mostly-true 2,466Others 2,185 True 2,063Table 1: The statistics of the LIAR data set3 Performance EvaluationEvaluations are performed using the LIAR datasetby Wang (2017). The dataset contains 12,836short statements from 3,341 speakers covering141 topics in POLITIFACT.COM3. Each news in-cludes text content, topic, and speaker profile.Speaker profiles include speaker name, title, partyaffiliation, current job, location of speech, andcredit history. The credit history includes the his-torical records of inaccurate statements for eachspeaker. Annotation is based on evaluations byprofessional editors. The labels take discrete val-ues from 1 to 6 corresponding to pants-fire, false,barely-true, half-true, mostly-true, and true. Thestatistics are listed in the Table 1.Models Dev. TestMajority 0.204 0.208LSTM 0.241 0.245CNN-Wang 0.247 0.247CNN-WangP 0.247 0.270Table 2: The results for baseline models3http://www.politifact.com/truth-o-meter/254The performance of four baseline models areshown in Table 2 including a simple majoritymodel, the LSTM model without using profile in-formation, the hybrid CNN model proposed byWang (2017) without profile information(CNN-Wang), and the hybrid CNN model by Wang withprofile information(CNN-WangP).Note that firstly, LSTM without profile doesnot perform better than CNN-Wang. However,other studies show that when attention model isincorporated, LSTM generally outperforms thatof CNN model (Chen et al., 2016; Yang et al.,2016) which will be shown later. Secondly, CNN-WangP, which uses speaker profiles has the bestperformance. For word representation, we trainthe skip-gram word embedding (Mikolov et al.,2013) on each dataset separately to initialize theword vectors. All embedding sizes on the modelare set to N = 200, a commonly used size.In speaker profiles, there are four basicattributes: party affiliation(Pa), location ofspeech(La), job title of speaker(Ti), and credit his-tory(Ch) which counts the inaccurate statementsfor speaker in past speeches. Note that credit his-tory is not a commonly available data. It is in-cluded here for comparison to CNN-Wang(P). Weconduct experiment on using these attributes indi-vidually and in combinations.Without Att With AttProfile Dev Test Dev TestCNN-Wang 0.247 0.247 N/A N/ACNN-WangP 0.247 0.274 N/A N/ABase LSTM 0.241 0.245 0.250 0.255Pa 0.247 0.246 0.253 0.257La 0.243 0.245 0.266 0.268Ti 0.241 0.247 0.258 0.257Ch 0.363 0.368 0.378 0.385Pa+La 0.250 0.252 0.255 0.261Pa+Ti 0.261 0.264 0.267 0.265La+Ti 0.267 0.264 0.268 0.271Ti+Ch 0.378 0.380 0.381 0.387Pa+Ti+Ch 0.385 0.387 0.397 0.395Pa+La+Ti 0.270 0.275 0.274 0.279La+Ti+Ch 0.388 0.401 0.398 0.405Pa+La+Ti+Ch 0.392 0.399 0.407 0.415Table 3: Evaluation on accuracy using differentcombinations of profile attributesTable 3 shows the performance of our pro-posed model with the top performers of the base-line systems put in the first two lines. The ba-sic LSTM model shown as Base-LSTM in Table3 performs less than CNN-WangP and similar toCNN-Wang without profile information. In otherwords, LSTM has no obvious model advantage inthis set of training data. We may also infer that thelexicon and style differences between fake newsand true news are not large enough for detection.And, the difference in the choice of deep neuralnetwork models are also not significant if profileinformation is not supplied.Table 3 also shows that speaker profile infor-mation can improve fake news detection signif-icantly. Besides credit history, which gives thelargest improvement of 3%, location of speakergives more improvements than part affiliation andjob title with improvement of 2.3%. When all at-tributes are included in detection, the performancesurge to over 40% in accuracy. Obviously, if credithistory of a speaker is available, it is not hard tosee how useful it is for fake new detection. Inpractice, however, we cannot expect the credit his-tory information to be available all the time forfake news detection. Therefore, it is more impor-tant to observe those combinations without Ch forcredit history. The best performers without Ch aremarked with underlines. The combination of usingall three attributes still outperforms CNN-WangPby 16.7% even though CNN-WangP has credit his-tory included. This further proves the effective-ness of our proposed method.4 ConclusionThis paper proposes a hybrid attention-basedLSTM model for fake news detection. In ourmodel, speaker profiles can contribute to fakenews detection in two ways: One is to includethem as attention factors for the learning of newstext; and the other is to use them as additional in-puts to provide more information. Experimentalresults show that both methods of using speakerprofiles can contribute to the improvement offake news detection. This can be interpreted asspeaker’s intention to speak the truth or fake itlargely depends on his/her, profiles, especiallyhis/hers credit history. Adopting speaker profilesinto an attention based LSTM detection model canreach over 41.5% in accuracy with net increase of14.5% in accuracy compared to the state-of-the-artmodel. Even without the use of credit history, theperformance net increase is still by 0.5%.255AcknowledgmentsThe work is partially supported by the researchgrants from Hong Kong Polytechnic University(PolyU RTVU) and GRF grant( CERG PolyU15211/14E).ReferencesJoan Bachenko, Eileen Fitzpatrick, and MichaelSchonwetter. 2008. Verification and implementa-tion of language-based deception indicators in civiland criminal narratives. In Proceedings of the22nd International Conference on ComputationalLinguistics-Volume 1, pages 41–48. Association forComputational Linguistics.Huimin Chen, Maosong Sun, Cunchao Tu, Yankai Lin,and Zhiyuan Liu. 2016. Neural sentiment classifica-tion with user and product attention. EMNLP.Niall J Conroy, Victoria L Rubin, and Yimin Chen.2015. Automatic deception detection: methods forfinding fake news. Proceedings of the Associationfor Information Science and Technology, 52(1):1–4.Vanessa Wei Feng and Graeme Hirst. 2013. Detect-ing deceptive opinions with profile compatibility. InIJCNLP, pages 338–346.Felix Gers. 2001. Long short-term memory in recur-rent neural networks. Ph.D. thesis, Universita¨t Han-nover.Swapna Gottipati, Minghui Qiu, Liu Yang, Feida Zhu,and Jing Jiang. 2013. Predicting users politicalparty using ideological stances. In InternationalConference on Social Informatics, pages 177–191.Springer.David F Larcker and Anastasia A Zakolyukina. 2012.Detecting deceptive discussions in conference calls.Journal of Accounting Research, 50(2):495–540.Yunfei Long, Qin Lu, Yue Xiao, MingLei Li, and Chu-Ren Huang. 2016. Domain-specific user preferenceprediction based on multiple user activities. In BigData (Big Data), 2016 IEEE International Confer-ence on, pages 3913–3921. IEEE.David M Markowitz and Jeffrey T Hancock. 2014.Linguistic traces of a scientific fraud: The case ofdiederik stapel. PloS one, 9(8):e105937.Tomas Mikolov, Kai Chen, Greg Corrado, and Jef-frey Dean. 2013. Efficient estimation of wordrepresentations in vector space. arXiv preprintarXiv:1301.3781.Victoria L Rubin. 2017. Deception detection and ru-mor debunking for social media.Victoria L Rubin, Yimin Chen, and Niall J Conroy.2015. Deception detection for news: three types offakes. Proceedings of the Association for Informa-tion Science and Technology, 52(1):1–4.Natali Ruchansky, Sungyong Seo, and Yan Liu. 2017.Csi: A hybrid deep model for fake news. arXivpreprint arXiv:1703.06959.Duyu Tang, Bing Qin, and Ting Liu. 2015. Documentmodeling with gated recurrent neural network forsentiment classification. In Proceedings of the 2015Conference on Empirical Methods in Natural Lan-guage Processing, pages 1422–1432.William Yang Wang. 2017. ” liar, liar pants on fire”:A new benchmark dataset for fake news detection.arXiv preprint arXiv:1705.00648.Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He,Alex Smola, and Eduard Hovy. 2016. Hierarchi-cal attention networks for document classification.In Proceedings of the 2016 Conference of the NorthAmerican Chapter of the Association for Computa-tional Linguistics: Human Language Technologies.256

Fake News Detection Through Multi-Perspective Speaker Profiles

http://repository.essex.ac.uk/27757/1/I17-2043.pdf

Fake News Detection Through Multi-Perspective Speaker Profiles

Abstract

Similar works

Full text

Available Versions

University of Essex Research Repository