Large Language Models vs. Search Engines: Evaluating User Preferences Across Varied Information Retrieval Scenarios

This study embarked on a comprehensive exploration of user preferences between Search Engines and Large Language Models (LLMs) in the context of various information retrieval scenarios. Conducted with a sample size of 100 internet users (N=100) from across the United States, the research delved into 20 distinct use cases ranging from factual searches, such as looking up COVID-19 guidelines, to more subjective tasks, like seeking interpretations of complex concepts in layman's terms. Participants were asked to state their preference between using a traditional search engine or an LLM for each scenario. This approach allowed for a nuanced understanding of how users perceive and utilize these two predominant digital tools in differing contexts. The use cases were carefully selected to cover a broad spectrum of typical online queries, thus ensuring a comprehensive analysis of user preferences. The findings reveal intriguing patterns in user choices, highlighting a clear tendency for participants to favor search engines for direct, fact-based queries, while LLMs were more often preferred for tasks requiring nuanced understanding and language processing. These results offer valuable insights into the current state of digital information retrieval and pave the way for future innovations in this field. This study not only sheds light on the specific contexts in which each tool is favored but also hints at the potential for developing hybrid models that leverage the strengths of both search engines and LLMs. The insights gained from this research are pivotal for developers, researchers, and policymakers in understanding the evolving landscape of digital information retrieval and user interaction with these technologies.Comment: 7 pages, 20 figures, conference pape

Global-Liar: Factuality of LLMs over Time and Geographic Regions

The increasing reliance on AI-driven solutions, particularly Large Language Models (LLMs) like the GPT series, for information retrieval highlights the critical need for their factuality and fairness, especially amidst the rampant spread of misinformation and disinformation online. Our study evaluates the factual accuracy, stability, and biases in widely adopted GPT models, including GPT-3.5 and GPT-4, contributing to reliability and integrity of AI-mediated information dissemination. We introduce 'Global-Liar,' a dataset uniquely balanced in terms of geographic and temporal representation, facilitating a more nuanced evaluation of LLM biases. Our analysis reveals that newer iterations of GPT models do not always equate to improved performance. Notably, the GPT-4 version from March demonstrates higher factual accuracy than its subsequent June release. Furthermore, a concerning bias is observed, privileging statements from the Global North over the Global South, thus potentially exacerbating existing informational inequities. Regions such as Africa and the Middle East are at a disadvantage, with much lower factual accuracy. The performance fluctuations over time suggest that model updates may not consistently benefit all regions equally. Our study also offers insights into the impact of various LLM configuration settings, such as binary decision forcing, model re-runs and temperature, on model's factuality. Models constrained to binary (true/false) choices exhibit reduced factuality compared to those allowing an 'unclear' option. Single inference at a low temperature setting matches the reliability of majority voting across various configurations. The insights gained highlight the need for culturally diverse and geographically inclusive model training and evaluation. This approach is key to achieving global equity in technology, distributing AI benefits fairly worldwide.Comment: 24 pages, 12 figures, 9 table

Design and Implementation of a Business-driven Threat Quantification Framework

Nowadays, companies and organizations invest in cybersecurity more and more as they are operating with digital information systems. Cyber risk management presents a welldefined path toward the management of critical assets, threats, and countermeasures. Within cyber risk management, threat modeling is a structured process to identify potential threats, and in this process, it is significant to evaluate each threat and estimate its potential impacts. Although threat modeling methodologies have been developed in depth, most of them focus on threat identification in di↵erent contexts, while how to quantify their impact for further inspection is less discussed. This thesis works on designing a framework to fill in this gap. The main outcome of this thesis is a framework that guides users to evaluate and quantify cyber threats in business contexts. The framework integrates applicable business impacts, calculates and visualizes the impacts of cyber threats, providing users with an intuitive picture of cyber threats analysis in the view of business. The prototype is well developed and properly evaluated, and the usability of the prototype is of satisfaction

The Social Media Machines: An Investigation of the Effect of Trust Moderated by Disinformation on Users’ Decision-Making Process

Social media networking sites (SMNS) have become a popular communications medium where users share information, knowledge, and persuasion. In less than two decades, social media\u27s (SM) dominance as a communication medium can\u27t be disputed, for good or evil. Combined with the newly found immediacy and pervasiveness, these SM applications\u27 persuasive power are useful weapons for organizations, angry customers, employees, actors, and activists bent on attacking or hacking other individuals, institutions, or systems. Consequently, SM has become the preferred default mechanism of news sources; however, users are unsure if the information gathered is true or false. According to the literature, SMNS generates large amounts of fake news or disinformation. The rapid proliferation of disinformation, information disseminated with the intent to harm, through SMNS has dramatically influenced and reduced people\u27s trust in the story and hints at hand. Disinformation has caused data breaches and many injured individuals and organizations, resulting in a lack of confidence in SMNS. While irrefutable that SMNS has become the new news outlet, trust remains the foundation of all communication. Since SM has changed the communication process, it is perceived as the most dangerous information dissemination vehicle known to society. Unfortunately, no one is safe from its lethality. Users must approach their usage with extreme care by understanding the technical capabilities and increasing their competence in detecting disinformation campaigns\u27 powerful influence. The continuous spread of disinformation has caused the credibility and trust of behemoths like Facebook, Twitter, and Instagram, to be significantly affected. Since trust is an essential factor in SMNS, mistrust hinders users\u27 abilities to make informed decisions. Research suggests that people make decisions based on the available information; therefore, it can be deduced that the decision-making process of SMNS users has been forever altered. Consequently, monitoring the spread of disinformation has become a front-burner priority for the government and society. By examining the effect of trust moderated by disinformation, this study aimed to investigate the factors that affect SMNS users\u27 decision-making behaviors. Factors influencing trust were also examined using the Conformity Group Norm Theory (CGNT) and Self Concept Theory (SCT). A theoretical model was created, and there were seven constructs; decision-making (DM), trust (TR), and the trust influencing factors: identification (ID), compliance (CP), internalization (IN), agency (AG), and community (CM). The theoretical model tested was based on the linear directional relationship of trust and decision making moderated by disinformation. This research tested three social media networking sites, Facebook, Twitter, and Instagram, with disinformation empirically. This quantitative study employed a role-play scenario web survey methodology and adopted a two-step Pearson r correlation coefficient procedure for data analysis. Before collecting data, an expert panel reviewed, and pilot tested the survey. The expert review recommended changes to the wording, length, and formatting of the instrument, allowing the pilot test to be easily tested by participants. The web-based scenario survey was designed with a 5- point Likert scale and distributed to SMNS users through Qualtrics XM to gather data on their decision-making process. The data analysis results revealed the moderating effect of disinformation between trust and the decision-making process of SMNS users. The data supported the conformity group norm theory (CGNT) and self-concept theory (SCT) factors. The results indicated that identification (ID), compliance (CP), internalization (IN), agency (AG), and community (CM) influence trust. Since the spread of disinformation through SMNS has much broader implications for democracy and society as a whole, this research\u27s results contribute to the knowledge of SM users\u27 behavior and decision-making processes. This study also contributes to the IS body of knowledge on social cybersecurity and has implications for practitioners and academics. This study offers a model by integrating behavioral and cognitive theories better to understand the directional relationship of trust and decision-making when exposed to disinformation. The model also identifies essential elements that influence SMNS users\u27 trust and engage them in risky cybersecurity behaviors. Furthermore, this study provides evidence of the need for future US social media governance

An Investigation of Non-Evidence Based Autism Intervention Representations in the Media: A Content Analysis

ABSTRACTDaniel Quinn Earixson: An Investigation of Non-Evidence Based Autism Intervention Representations in the Media: A Content Analysis(Under the direction of Kara Hume, PhD) Non-evidence-based practices (NEBPs) are interventions that have not been indicated by research to be effective in treating the core deficits of autism or the related behavioral challenges across developmental domains. Under the umbrella of NEBPs are the interventions for autism that are physically and/or emotionally harmful, as well as those that are not necessarily harmful, but are ineffective. Caregivers for people with autism often choose to use NEBPs either alongside or instead of evidence-based practices (EBPs) (Goin-Kochel et al., 2007). Caregivers also report reliance on the internet to find information about interventions (Law, 2009; Grant et. al., 2015). The online search engine Google, which powers YouTube, is by far the largest provider of online health information to the public (Curfman, 2020). Additionally, medical misinformation on social media, including YouTube, is rampant; some studies have found up to 87% of posts regarding certain health topics to contain misinformation (Suarez-Lledo & Alvarez-Galvez, 2021). Videos containing misinformation are often viewed, liked, and shared more than those containing EBP-related information (Bora et al., 2018), indicating these types of videos are highly engaging. Considering this landscape of internet-based medical information, it is critically important for researchers and clinicians to understand the array of information caregivers are exposed to when first starting their search for autism interventions, to inform prevention and research dissemination practices.This study used a simple, common search of YouTube to create a sample of 150 videos caregivers may see when researching autism interventions. Quantitative and qualitative data were used to describe the videos in the sample, specifically the proportion of NEBP to EBP videos in the total sample, as well as the actual content of the NEBP related videos. Videos from the search were categorized by EBP (n=34) and NEBP (n=62), as well as other categories, and the proportion of the total videos in these two categories to the sample were compared for significance. This was important to understand the overall likelihood of caregivers finding information regarding NEBPs or EBPs. Additionally, the content of NEBP-related videos was coded qualitatively and analyzed using the pre-existing warning signs of pseudoscience in autism interventions (Association of Science in Autism Treatment, 1999; Thyer, 2019). Additionally, the NEBP-related videos were coded using a constant-comparative method (Glaser, 1965) within a coding dyad to detect new possible warning signs for pseudoscience. Results indicated a significant difference between the proportion of NEBP to EBP-related videos in the sample, indicating there were more NEBP-related videos in this generic search and these videos are more likely to be seen by caregivers. Additionally, the results confirmed the presence of warning signs for pseudoscience in the videos in the sample. Results also helped to identify new warning signs from the content of NEBP videos. The study contributes to the research by highlighting the prevalence of NEBP-related information on an extremely popular search engine/social media site, as well as better understanding how modern NEBPs are advertised to the public searching for medical-health information.Doctor of Philosoph