Framework-Based Qualitative Analysis of Free Responses of Large Language Models: Algorithmic Fidelity

Today, using Large-scale generative Language Models (LLMs) it is possible to simulate free responses to interview questions like those traditionally analyzed using qualitative research methods. Qualitative methodology encompasses a broad family of techniques involving manual analysis of open-ended interviews or conversations conducted freely in natural language. Here we consider whether artificial "silicon participants" generated by LLMs may be productively studied using qualitative methods aiming to produce insights that could generalize to real human populations. The key concept in our analysis is algorithmic fidelity, a term introduced by Argyle et al. (2023) capturing the degree to which LLM-generated outputs mirror human sub-populations' beliefs and attitudes. By definition, high algorithmic fidelity suggests latent beliefs elicited from LLMs may generalize to real humans, whereas low algorithmic fidelity renders such research invalid. Here we used an LLM to generate interviews with silicon participants matching specific demographic characteristics one-for-one with a set of human participants. Using framework-based qualitative analysis, we showed the key themes obtained from both human and silicon participants were strikingly similar. However, when we analyzed the structure and tone of the interviews we found even more striking differences. We also found evidence of the hyper-accuracy distortion described by Aher et al. (2023). We conclude that the LLM we tested (GPT-3.5) does not have sufficient algorithmic fidelity to expect research on it to generalize to human populations. However, the rapid pace of LLM research makes it plausible this could change in the future. Thus we stress the need to establish epistemic norms now around how to assess validity of LLM-based qualitative research, especially concerning the need to ensure representation of heterogeneous lived experiences.Comment: 46 pages, 5 tables, 5 figure

Perceived discrimination and type 2 diabetes incidence: findings from the Health and Retirement Study

Objectives: To examine the prospective association between perceived everyday discrimination and type 2 diabetes incidence in a large population-based sample.Methods: Data were from the Health and Retirement Study of 14900 individuals aged ≥50 years without a diabetes diagnosis. Participants self-reported experiences of everyday discrimination and diabetes status. Associations between baseline perceived everyday discrimination (one time point) and incident diabetes in the following ten years were modelled using Cox regression, adjusting for potential confounders. Exploratory analyses assessed the association between repeated reports of perceived everyday discrimination (reports of everyday discrimination at more than one time point) and later diabetes onset. Results: A total of 917 (6.15%) of the 14900 participants developed type 2 diabetes over a maximum 10-year follow-up (median = 6). Baseline perceived everyday discrimination was prospectively associated with an increased risk of diabetes (HR = 1.37, 95% CI: [1.15; 1.63], p&lt;0.001) independent of age, sex, wealth, race and ethnicity, and education. This association was robust to further adjustment for body mass index (BMI), hypertension, physical activity, smoking, alcohol consumption, and depression. In exploratory analyses, repeated reports of everyday discrimination were not significantly associated with incident diabetes. Conclusions: Individuals who perceive everyday discrimination are more likely to develop type 2 diabetes than those who do not perceive everyday discrimination. Further research is needed to investigate the potential pathways linking discrimination and diabetes onset. <br/

Perceived discrimination in middle-aged and older adults: Comparison between England and the United States

OBJECTIVES: This study examined differences in perceived discrimination across multiple characteristics in England and the United States (US), in middle- and older-aged adults. METHODS: Using data from the English Longitudinal Study of Aging (N = 8,671) and the US-based Health and Retirement Study (N = 7,927), we assessed cross-national differences in perceived discrimination attributed to disability, financial status, sex, race, sexual orientation, and weight. We also compared how perceived discrimination varied with socioeconomic position (SEP) based on wealth. RESULTS: Perceived discrimination due to financial status was more common in England (6.65%) than in the US (2.14%) adjusting for age, sex, and wealth [Odds Ratio (OR) = 1.09, 95% CI (1.07; 1.10)]. This affected people of low but not high SEP. Sexual orientation discrimination was more common in England [0.72 vs. 0.15%, OR = 4.61, 95% CI (2.48; 8.57)]. Sex-based perceived discrimination was more prevalent in the US (12.42%) than England (9.07%) adjusting for age and wealth [OR = 0.87, 95% CI (0.86; 0.89)]. Cross-national differences in sex discrimination did not vary with SEP. Racism was the most common type of perceived discrimination reported in both samples (England: 17.84%, US: 19.80%), with no significant cross-national differences after adjustment for sex. DISCUSSION: Perceived discrimination attributed to financial status and sexual orientation were more prevalent in England, while more women perceived sex discrimination in the US. This study suggests that country-specific and socioeconomic factors affect the prevalence of perceived discrimination. This may be relevant when targeting interventions aimed at reducing perceived discrimination

Framework-based qualitative analysis of free responses of Large Language Models: Algorithmic fidelity

Today, with the advent of Large-scale generative Language Models (LLMs) it is now possible to simulate free responses to interview questions such as those traditionally analyzed using qualitative research methods. Qualitative methodology encompasses a broad family of techniques involving manual analysis of open-ended interviews or conversations conducted freely in natural language. Here we consider whether artificial “silicon participants” generated by LLMs may be productively studied using qualitative analysis methods in such a way as to generate insights that could generalize to real human populations. The key concept in our analysis is algorithmic fidelity, a validity concept capturing the degree to which LLM-generated outputs mirror human sub-populations’ beliefs and attitudes. By definition, high algorithmic fidelity suggests that latent beliefs elicited from LLMs may generalize to real humans, whereas low algorithmic fidelity renders such research invalid. Here we used an LLM to generate interviews with “silicon participants” matching specific demographic characteristics one-for-one with a set of human participants. Using framework-based qualitative analysis, we showed the key themes obtained from both human and silicon participants were strikingly similar. However, when we analyzed the structure and tone of the interviews we found even more striking differences. We also found evidence of a hyper-accuracy distortion. We conclude that the LLM we tested (GPT-3.5) does not have sufficient algorithmic fidelity to expect in silico research on it to generalize to real human populations. However, rapid advances in artificial intelligence raise the possibility that algorithmic fidelity may improve in the future. Thus we stress the need to establish epistemic norms now around how to assess the validity of LLM-based qualitative research, especially concerning the need to ensure the representation of heterogeneous lived experiences.</p

Conversion of carboxylic acids to amides under the action of tantalum(V) chloride

It was found that the reaction of aliphatic carboxylic acids with secondary amines under the action of tantalum (V) chloride leads to the selective formation of carboxamides. N,N-Diethyladamantane-1-carboxamide were synthesized with a yield of 73% as well

Informing behaviour change intervention design using systematic review with Bayesian meta-analysis: physical activity in heart failure

Embracing the Bayesian approach, we aimed to synthesise evidence regarding barriers and enablers to physical activity in heart failure (HF) in a way that can inform behaviour change intervention development. This approach helps in estimating and quantifying the uncertainty in the evidence and facilitates the synthesis of qualitative and quantitative studies. Qualitative and observational studies investigating barriers and enablers to physical activity in adults diagnosed with HF were included in this systematic review with a Bayesian meta-analysis. Qualitative evidence was annotated using the Theoretical Domains Framework and represented as a prior distribution using an expert elicitation task. The maximum a posteriori probability (MAP) was calculated as a summary statistic for the probability distribution for the log OR value estimating the relationship between physical activity and each determinant, according to qualitative evidence alone, quantitative evidence, and qualitative and quantitative evidence combined. The dispersion in the probability distribution for log OR associated with each barrier or enabler was used to evaluate the level of uncertainty in the evidence. Wide, medium, and narrow dispersion (SD) corresponded to high, moderate, and low uncertainty in the evidence, respectively. Evidence from three qualitative and 16 (N = 2739) quantitative studies was synthesised. High pro-b-type natriuretic peptide, pro-BNP (MAP value for log OR = -1.16; 95% CrI: [-1.21; -1.11]) and self-reported symptoms (MAP for log OR = 0.48; 95% CrI: [0.40; 0.55]) were suggested as barriers to physical activity with narrow distribution dispersion (SD = 0.18 and 0.19, respectively). Modifiable barriers were symptom distress (MAP for log OR = -0.46; 95% CrI: [-0.68; -0.24]), and negative attitude (MAP for log OR = -0.40; 95% CrI: [-0.49; -0.31]), SD = 0.36 and 0.26, respectively. Modifiable enablers were social support (MAP for log OR = 0.56; 95% CrI: [0.48; 0.63]), self-efficacy (MAP for log OR = 0.43; 95% CrI: [0.32; 0.54]), positive physical activity attitude (MAP for log OR = 0.92; 95% CrI: [0.77; 1.06]), SD = 0.26, 0.37, and 0.36, respectively. This work extends the limited research on the modifiable barriers and enablers for physical activity by individuals living with HF

Using virtual representations in mHealth application interventions for health-related behaviour change: A systematic review

Many mobile health (mHealth) application interventions include virtual representations of the self in varying forms, such as agents, or avatars to initiate health behaviour change. This review aimed to determine: (i) which virtual representations are effective in mHealth application interventions, and (ii) whether any studies implemented specific mechanisms (the psychological causes of change) and behaviour change techniques (BCTs) to influence positive behaviour change. Following PRISMA guidelines, a narrative systematic review of empirical studies was conducted from ten different databases (ranging from MEDLINE to Cochrane Library) from inception to December 2021. This included articles reporting a virtual representation of mHealth application intervention, published in English from any time point, which addressed and reported a variety of outcome health behaviours. Information on sample characteristics, study conditions, mechanisms and BCTs, results, and conclusions were extracted. Quality and risk of bias were assessed using the Mixed Methods Appraisal and Cochrane Risk of Bias Tools. Out of the 2,579 original search results, five eligible studies (total participants = 509), with low to moderate quality were included. It was found that customisable virtual representation mHealth interventions were most effective in encouraging behaviour change and app adherence. These interventions included mechanisms such as motivation, feedback, self-image, and BCTs such as goal setting, and self-monitoring. The current evidence suggests that virtual representations in mHealth app interventions may positively influence health behaviour change. However, there is limited evidence available to determine to which these influences are a result of the virtual representations, or the intervention design

Identifying physical activity mobile applications for people with rheumatoid arthritis : a content analysis and quality appraisal

Background Physical activity and exercise (PA) are recommended for people with rheumatoid arthritis (RA), although adherence to PA tends to be low. Accessible, tailored interventions are needed to help people with RA change their behaviour to achieve public health PA recommendations and improve health outcomes. Use of smartphones and mobile applications (apps) is increasing and apps may assist people to reach PA recommendations. However, the availability, quality and content of evidence-based PA apps for people with RA is unknown. This study assessed the features, content and quality of apps targeting PA for adults with RA by i) systematically identifying apps ii) summarising their features and content iii) comparing their content to physical activity and exercise guidelines and iv) rating app quality. Methods A systematic search of the UK Apple AppStore and Google Play store was conducted to identify apps designed to facilitate PA in adults with RA between 19th-20th June 2019. Apps were excluded if they were i) not in English ii) for use by clinicians only or ii) solely focused on advertising a clinic/product. No fee restrictions were applied. Android and iOS apps were downloaded to smartphones, features/functionality described, content coded using Consensus on Exercise Reporting Template (CERT) and behaviour change technique taxonomy V1 (BCTTv1) and apps rated using the Mobile App Rating Scale (MARS range 0-5) by two independent reviewers. App features were compared with public health PA guidelines (150 minutes moderate PA/week or equivalent plus twice weekly resistance exercise) and American College of Sports Medicine recommendations for exercise prescription. Results Initially, 14,047 apps were identified. Following de-duplication, 2,737 apps were screened for eligibly and six apps were downloaded (two UK Apple AppStore, four Google Play store), yielding four unique apps. Only one app provided PA recommendations broadly aligned with public health PA guidelines and no apps aligned to ACSM exercise prescription guidelines or offered information on tailoring of PA to disease symptoms or health status. Apps included between 4-13 behaviour change techniques (BCTs) and three BCTs were common to all apps (demonstration of behaviour, instructions on how to perform behaviour, information about health consequences). Overall, MARS scores ranged between 2.25-4.17. Conclusion There is a lack of high-quality mobile apps which can be tailored to support PA for people with RA. Whilst all included apps incorporated some BCTs previously identified as effective for PA promotion, only one, high quality app provided PA recommendations which broadly aligned with public health PA guidelines but offered limited options for tailoring PA or exercise. Collaboration between adults with RA, clinicians, and app developers is needed to produce evidence informed apps, with embedded BCTs, which can be tailored to support people with RA achieve PA recommendations

Smartphone Apps Targeting Physical Activity in People With Rheumatoid Arthritis:Systematic Quality Appraisal and Content Analysis

Background: Rheumatoid arthritis (RA) is a disabling, inflammatory joint condition affecting 0.5%-1% of the global population. Physical activity (PA) and exercise are recommended for people with RA, but uptake and adherence tend to be low. Smartphone apps could assist people with RA to achieve PA recommendations. However, it is not known whether high quality, evidence-informed PA apps that include behavior change techniques (BCTs) previously identified as effective for PA adherence are available for people with RA. Objective: This study aims to systematically identify apps that include goals to facilitate PA for adults with RA and assess app quality and content for the inclusion of relevant BCTs against recommendations for cardiorespiratory, resistance, flexibility, and neuromotor PA and exercise. Methods: A systematic search of the Apple App Store and Google Play Store in the United Kingdom was conducted to identify English language apps that promote PA for adults with RA. Two researchers independently assessed app quality (mobile app rating scale [MARS]; range 0-5) and content (BCT Taxonomy version 1, World Health Organization, the American College of Sports Medicine, and the European League against Rheumatism recommendations for PA). The completeness of reporting of PA prescription was evaluated using a modified version of the Consensus on Exercise Reporting Template (CERT; range 0-14). Results: A total of 14,047 apps were identified. Following deduplication, 2737 apps were screened for eligibility; 6 apps were downloaded (2 on the Apple App Store and 4 on the Google Play Store), yielding 4 unique apps. App quality varied (MARS score 2.25-4.17). Only 1 app was congruent with all aspects of the PA recommendations. All apps completely or partially recommended flexibility and resistance exercises, 3 apps completely or partially advised some form of neuromotor exercise, but only 2 offered full or partial guidance on cardiorespiratory exercise. Completeness of exercise reporting was mixed (CERT scores 7-14 points) and 3-7 BCTs were identified. Two BCTs were common to all apps (information about health consequences and instruction on how to perform behavior). Higher quality apps included a greater number of BCTs and were more closely aligned to PA guidance. No published trials evaluating the effect of the included apps were identified. Conclusions: This review identifies 4 PA apps of mixed quality and content for use by people with RA. Higher quality apps were more closely aligned to PA guidance and included a greater number of BCTs. One high-quality app (Rheumatoid Arthritis Information Support and Education) included 7 BCTs and was fully aligned with PA and exercise guidance. The effect of apps on PA adherence should be established before implementation

Applying a theoretical framework to assess the acceptability of therapist training and delivery of the Gait Rehabilitation in Early Rheumatoid Arthritis Trial intervention (GREAT Strides): a qualitative analysis

Background/Purpose: GREAT Strides is a novel gait rehabilitation intervention with an embedded psychological component aimed at improving walking ability in people with early rheumatoid arthritis. Therapists received two days of gait rehabilitation training, motivational interviewing (MI) and behaviour change techniques (BCTs). This study explored therapists’ acceptability of: (1) the training received and (2) delivering GREAT Strides within a feasibility study. Methods: Nine therapists (four physiotherapists, five podiatrists) participated in semi-structured interviews. The topic guide was informed by the Theoretical Framework of Acceptability (TFA). Interviews were professionally transcribed, and a deductive thematic analysis was applied. Data were coded into six TFA constructs (Affective Attitude; Burden; Intervention Coherence; Opportunity Costs; Perceived Effectiveness; Self-efficacy). Results: Key barriers and enablers with regards to the acceptability of the training and intervention delivery were identified. Training: Therapists liked the supportive training environment (affective attitude) and reported that role play exercises aided their confidence in applying MI and BCTs (self-efficacy). The lack of time available to attend training was considered unacceptable (opportunity costs). Delivery: All therapists valued the opportunity to provide individualised care (intervention coherence). Barriers associated with acceptability included the use of trial-related materials (e.g. checklist) during intervention delivery (burden) and the time delay between receiving training and intervention delivery (perceived effectiveness). Conclusions and implications: The bespoke training and Great Strides delivery was acceptable to most therapists. Key refinements (e.g. timing of training in relation to intervention delivery) have the potential to improve aspects of the training and delivery of the intervention, maximising efficiency and potential for effectiveness