People rely on their attitudes more than the source when judging the accuracy of news stories on Facebook

The role of 'fake news' in shaping political behaviour has received extensive attention in recent years, with Facebook and other websites undertaking a number of measures to try and address the problem. Drawing on an experimental study during the 2017 German federal election campaign, Bernhard Clemm von Hohenberg illustrates that people rely far more on their pre-existing political attitudes when judging the accuracy of a story on Facebook than they do on the reputation of a news source

Horseshoe patterns: visualizing partisan media trust in Germany

A trusted media is crucial for a politically informed citizenry, yet media trust has become fragile in many Western countries. An underexplored aspect is the link between media (dis)trust and populism. The authors visualize media trust across news outlets and partisanship in Germany, for both mainstream and “alternative” news sources. For each source, average trust is grouped by partisanship and sorted from left to right, allowing within-source comparisons. The authors find an intriguing horseshoe pattern for mainstream media sources, for which voters of both populist left-wing and right-wing parties express lower levels of trust. The underlying distribution of individual responses reveals that voters of the right-wing populist party are especially likely to “not at all” trust the mainstream outlets that otherwise enjoy high levels of trust. The media trust gap between populist and centrist voters disappears for alternative sources, for which trust is generally low

Incentivizing News Consumption on Social Media Platforms Using Large Language Models and Realistic Bot Accounts

Polarization, declining trust, and wavering support for democratic norms are pressing threats to U.S. democracy. Exposure to verified and quality news may lower individual susceptibility to these threats and make citizens more resilient to misinformation, populism, and hyperpartisan rhetoric. This project examines how to enhance users' exposure to and engagement with verified and ideologically balanced news in an ecologically valid setting. We rely on a large-scale two-week long field experiment (from 1/19/2023 to 2/3/2023) on 28,457 Twitter users. We created 28 bots utilizing GPT-2 that replied to users tweeting about sports, entertainment, or lifestyle with a contextual reply containing two hardcoded elements: a URL to the topic-relevant section of quality news organization and an encouragement to follow its Twitter account. To further test differential effects by gender of the bots, treated users were randomly assigned to receive responses by bots presented as female or male. We examine whether our over-time intervention enhances the following of news media organization, the sharing and the liking of news content and the tweeting about politics and the liking of political content. We find that the treated users followed more news accounts and the users in the female bot treatment were more likely to like news content than the control. Most of these results, however, were small in magnitude and confined to the already politically interested Twitter users, as indicated by their pre-treatment tweeting about politics. These findings have implications for social media and news organizations, and also offer direction for future work on how Large Language Models and other computational interventions can effectively enhance individual on-platform engagement with quality news and public affairs

Analysis of Web Browsing Data: A Guide

The use of individual-level browsing data, that is, the records of a person’s visits to online content through a desktop or mobile browser, is of increasing importance for social scientists. Browsing data have characteristics that raise many questions for statistical analysis, yet to date, little hands-on guidance on how to handle them exists. Reviewing extant research, and exploring data sets collected by our four research teams spanning seven countries and several years, with over 14,000 participants and 360 million web visits, we derive recommendations along four steps: preprocessing the raw data; filtering out observations; classifying web visits; and modelling browsing behavior. The recommendations we formulate aim to foster best practices in the field, which so far has paid little attention to justifying the many decisions researchers need to take when analyzing web browsing data.Die Verwendung von Browsing-Daten auf individueller Ebene, d.h. die Aufzeichnungen der Besuche einer Person bei Online-Inhalten über einen Desktop- oder mobilen Browser, ist für Sozialwissenschaftler*innen von zunehmender Bedeutung. Browsing-Daten haben Eigenschaften, die viele Fragen für die statistische Analyse aufwerfen, doch bisher gibt es nur wenige praktische Anleitungen für den Umgang mit ihnen. Nach Durchsicht bestehender Forschungsarbeiten und der Untersuchung von Datensätzen, die von vier Forschungsteams in sieben Ländern und über mehrere Jahre hinweg gesammelt wurden, mit über 14.000 Teilnehmenden und 360 Millionen Webbesuchen, leiten die Autor*innen Empfehlungen in vier Schritten ab: Vorverarbeitung der Rohdaten, Herausfiltern von Beobachtungen, Klassifizierung von Webbesuchen und Modellierung des Surfverhaltens

Truth and Bias: Robust findings?

Differences in information processing across ideological groups have been a recurring theme in political science. The recent debate about “fake news” has brought attention to the question whether US liberals and conservatives differ in how they evaluate the truth of information. Researchers have asked, first, which side is better at discerning true from false information (truth discernment), and second, whether liberals and conservatives are driven to different degrees by the ideological congruence of information (assimilation bias). The paradigmatic designs to study these question require selecting or constructing informational stimuli. As I show empirically with two robustness tests and one extended replication of previous studies, this selection/construction necessarily affects the resulting (a)symmetries. When it is unclear how well the selection represents some universe of real-world information, we should thus be wary of results about asymmetries. Ideally, studies should find ways to randomly sample stimuli from the target population of information

Large Language Models as a Substitute for Human Experts in Annotating Political Text

Large-scale text analysis has grown rapidly as a method in political science and beyond. To date, text-as-data methods rely on large volumes of human-annotated training examples, which places a premium on researcher resources. However, advances in large language models (LLMs) may make automated annotation increasingly viable. This paper tests the performance of GPT-4 across a range of scenarios relevant for analysis of political text. We compare GPT-4 coding with human expert coding of tweets and news articles across four variables (whether text is political, negativity, sentiment, and ideology) and across four countries (the United States, Chile, Germany, and Italy). GPT-4 coding is highly accurate, especially for shorter texts such as tweets, correctly classifying texts up to 95\% of the time. Performance drops for longer news articles, and very slightly for non-English text. We introduce a ``hybrid'' coding approach, in which disagreements of multiple GPT-4 runs are adjudicated by a human expert, which boosts accuracy. Finally, we explore downstream effects, finding that transformer models trained on hand-coded or GPT-4-coded data yield almost identical outcomes. Our results suggests that LLM-assisted coding is a viable and cost-efficient approach, although consideration should be given to task complexity

Large language models as a substitute for human experts in annotating political text

Large-scale text analysis has grown rapidly as a method in political science and beyond. To date, text-as-data methods rely on large volumes of human-annotated training examples, which place a premium on researcher resources. However, advances in large language models (LLMs) may make automated annotation increasingly viable. This paper tests the performance of GPT-4 across a range of scenarios relevant for analysis of political text. We compare GPT-4 coding with human expert coding of tweets and news articles across four variables (whether text is political, its negativity, its sentiment, and its ideology) and across four countries (the United States, Chile, Germany, and Italy). GPT-4 coding is highly accurate, especially for shorter texts such as tweets, correctly classifying texts up to 95% of the time. Performance drops for longer news articles, and very slightly for non-English text. We introduce a ‘hybrid’ coding approach, in which disagreements of multiple GPT-4 runs are adjudicated by a human expert, which boosts accuracy. Finally, we explore downstream effects, finding that transformer models trained on hand-coded or GPT-4-coded data yield almost identical outcomes. Our results suggest that LLM-assisted coding is a viable and cost-efficient approach, although consideration should be given to task complexity