5 research outputs found

    Framework-Based Qualitative Analysis of Free Responses of Large Language Models: Algorithmic Fidelity

    Full text link
    Today, using Large-scale generative Language Models (LLMs) it is possible to simulate free responses to interview questions like those traditionally analyzed using qualitative research methods. Qualitative methodology encompasses a broad family of techniques involving manual analysis of open-ended interviews or conversations conducted freely in natural language. Here we consider whether artificial "silicon participants" generated by LLMs may be productively studied using qualitative methods aiming to produce insights that could generalize to real human populations. The key concept in our analysis is algorithmic fidelity, a term introduced by Argyle et al. (2023) capturing the degree to which LLM-generated outputs mirror human sub-populations' beliefs and attitudes. By definition, high algorithmic fidelity suggests latent beliefs elicited from LLMs may generalize to real humans, whereas low algorithmic fidelity renders such research invalid. Here we used an LLM to generate interviews with silicon participants matching specific demographic characteristics one-for-one with a set of human participants. Using framework-based qualitative analysis, we showed the key themes obtained from both human and silicon participants were strikingly similar. However, when we analyzed the structure and tone of the interviews we found even more striking differences. We also found evidence of the hyper-accuracy distortion described by Aher et al. (2023). We conclude that the LLM we tested (GPT-3.5) does not have sufficient algorithmic fidelity to expect research on it to generalize to human populations. However, the rapid pace of LLM research makes it plausible this could change in the future. Thus we stress the need to establish epistemic norms now around how to assess validity of LLM-based qualitative research, especially concerning the need to ensure representation of heterogeneous lived experiences.Comment: 46 pages, 5 tables, 5 figure

    Psychosocial Risk Factors for Health-Related Quality of Life in Adult Congenital Heart Disease

    Get PDF
    Background: There is variability in the impact of adult congenital heart disease (ACHD) on health-related quality of life (HRQoL). A greater insight into the impact of ACHD may be gained from investigating HRQoL in various diagnostic groups and considering the importance of psychosocial risk factors for poor HRQoL. / Objective: We compared the HRQoL of people with ACHD with normative data from the general population and among 4 diagnostic groups and identified risk factors for poor HRQoL in ACHD from a comprehensive set of sociodemographic, clinical, and psychosocial factors. / Methods: We conducted a cross-sectional study with 303 participants from 4 diagnostic groups Simple, Tetralogy of Fallot, Transposition of the Great Arteries, Single Ventricle who completed measures of illness perceptions, coping, social support, mood, and generic and disease-specific HRQoL. Data were analyzed using 1-sample t tests, analysis of variance, and hierarchical multiple regressions. / Results: There was diminished psychosocial HRQoL in the Simple group compared with the general population. Consistently significant risk factors for poor HRQoL included younger age, a perception of more severe symptoms due to ACHD, depression, and anxiety. Clinical factors were poor predictors of HRQoL. / Conclusions: The findings highlight the need to develop intervention studies aiming to improve HRQoL in people with ACHD and the routine assessment of illness perceptions and mood problems during key periods in people's lives. This will help address patient misconceptions that could be tackled by clinicians or specialist nurses during routine outpatient appointments and identify people in need of psychological support

    Informing behaviour change intervention design using systematic review with Bayesian meta-analysis: physical activity in heart failure

    Get PDF
    Embracing the Bayesian approach, we aimed to synthesise evidence regarding barriers and enablers to physical activity in heart failure (HF) in a way that can inform behaviour change intervention development. This approach helps in estimating and quantifying the uncertainty in the evidence and facilitates the synthesis of qualitative and quantitative studies. Qualitative and observational studies investigating barriers and enablers to physical activity in adults diagnosed with HF were included in this systematic review with a Bayesian meta-analysis. Qualitative evidence was annotated using the Theoretical Domains Framework and represented as a prior distribution using an expert elicitation task. The maximum a posteriori probability (MAP) was calculated as a summary statistic for the probability distribution for the log OR value estimating the relationship between physical activity and each determinant, according to qualitative evidence alone, quantitative evidence, and qualitative and quantitative evidence combined. The dispersion in the probability distribution for log OR associated with each barrier or enabler was used to evaluate the level of uncertainty in the evidence. Wide, medium, and narrow dispersion (SD) corresponded to high, moderate, and low uncertainty in the evidence, respectively. Evidence from three qualitative and 16 (N = 2739) quantitative studies was synthesised. High pro-b-type natriuretic peptide, pro-BNP (MAP value for log OR = -1.16; 95% CrI: [-1.21; -1.11]) and self-reported symptoms (MAP for log OR = 0.48; 95% CrI: [0.40; 0.55]) were suggested as barriers to physical activity with narrow distribution dispersion (SD = 0.18 and 0.19, respectively). Modifiable barriers were symptom distress (MAP for log OR = -0.46; 95% CrI: [-0.68; -0.24]), and negative attitude (MAP for log OR = -0.40; 95% CrI: [-0.49; -0.31]), SD = 0.36 and 0.26, respectively. Modifiable enablers were social support (MAP for log OR = 0.56; 95% CrI: [0.48; 0.63]), self-efficacy (MAP for log OR = 0.43; 95% CrI: [0.32; 0.54]), positive physical activity attitude (MAP for log OR = 0.92; 95% CrI: [0.77; 1.06]), SD = 0.26, 0.37, and 0.36, respectively. This work extends the limited research on the modifiable barriers and enablers for physical activity by individuals living with HF
    corecore