11 research outputs found

    Framework-Based Qualitative Analysis of Free Responses of Large Language Models: Algorithmic Fidelity

    Full text link
    Today, using Large-scale generative Language Models (LLMs) it is possible to simulate free responses to interview questions like those traditionally analyzed using qualitative research methods. Qualitative methodology encompasses a broad family of techniques involving manual analysis of open-ended interviews or conversations conducted freely in natural language. Here we consider whether artificial "silicon participants" generated by LLMs may be productively studied using qualitative methods aiming to produce insights that could generalize to real human populations. The key concept in our analysis is algorithmic fidelity, a term introduced by Argyle et al. (2023) capturing the degree to which LLM-generated outputs mirror human sub-populations' beliefs and attitudes. By definition, high algorithmic fidelity suggests latent beliefs elicited from LLMs may generalize to real humans, whereas low algorithmic fidelity renders such research invalid. Here we used an LLM to generate interviews with silicon participants matching specific demographic characteristics one-for-one with a set of human participants. Using framework-based qualitative analysis, we showed the key themes obtained from both human and silicon participants were strikingly similar. However, when we analyzed the structure and tone of the interviews we found even more striking differences. We also found evidence of the hyper-accuracy distortion described by Aher et al. (2023). We conclude that the LLM we tested (GPT-3.5) does not have sufficient algorithmic fidelity to expect research on it to generalize to human populations. However, the rapid pace of LLM research makes it plausible this could change in the future. Thus we stress the need to establish epistemic norms now around how to assess validity of LLM-based qualitative research, especially concerning the need to ensure representation of heterogeneous lived experiences.Comment: 46 pages, 5 tables, 5 figure

    The association between loneliness and other constructs of social connectedness and the probability of desire for hastened death, euthanasia and assisted suicide: systematic review

    Get PDF
    Background Euthanasia and assisted suicide (EAS) requests are common in countries where they are legal. Loneliness and social isolation are modifiable risk factors for mental illness and suicidal behaviour and are common in terminal illness. Our objective was to summarise available literature to clarify whether these and related measures of social connectedness might contribute to requests for EAS. Methods We conducted a pre-registered (PROSPERO CRD42019160508) systematic review and narrative synthesis of quantitative literature investigating associations between social connectedness and a) requested/actual EAS, b) attitudes towards EAS, and c) a desire for hastened death (DHD) by searching six databases (PsycINFO, MEDLINE, EMBASE, Scopus, Web of Science, Google Scholar) from inception to November 2022, rating eligible peer-reviewed, empirical studies using the QATSO quality assessment tool. Results We identified 37 eligible studies that investigated associations with a) requested/actual EAS (n = 9), b) attitudes to EAS (n = 16), and c) DHD (n = 14), with limited overlap, including 17,359 participants. The majority (62%) were rated at medium/high risk of bias. Focussing our narrative synthesis on the more methodologically sound studies, we found no evidence to support an association between different constructs of social connectedness and requested or actual EAS, and very little evidence to support an association with attitudes to EAS or an association with DHD. Conclusions Our findings for all age groups are consistent with a those of a previous systematic review focussed on older adults and suggest that poor social connectedness is not a clear risk factor for EAS or for measures more distally related to EAS. However, we acknowledge low study quality in some studies in relation to sampling, unvalidated exposure/outcome measures, cross-sectional design, unadjusted analyses, and multiple testing. Clinical assessment should focus on modifying established risk factors for suicide and EAS, such as hopelessness and depression, as well as improving any distressing aspects of social disconnectedness to improve quality of life

    'Beyond places of safety' - a qualitative study exploring the implementation of mental health crisis care innovations across England

    Get PDF
    BACKGROUND: Mental health acute and crisis care consumes a large share of mental health budgets internationally but is often experienced as unsatisfactory and difficult to access. As a result, there is an increasing move towards developing innovative community crisis services, to improve patient experience and relieve pressure on inpatient and emergency services. This study aims to understand what helps and hinders the implementation of innovative mental health crisis care projects in England. METHODS: Using a qualitative approach, 18 interviews were conducted with crisis care service managers exploring their experiences and views of the development and implementation of their service developed with support from an English national capital funding programme. A framework analysis was conducted informed by implementation science. RESULTS: Key facilitators to implementation of innovative crisis services included bottom-up development, service user involvement, strong collaborative working, and leadership and management buy-in. Key barriers that affected the projects implementation included the complexities of crisis care, workforce challenges and resourcing issues. CONCLUSION: There is a recognised need to improve, update, and innovate current crisis care offers. Results from this study suggest that a range of models can help address the heterogenous needs of local populations and that new approaches can be implemented where they utilise a whole-systems approach, involving service users and relevant professional stakeholders beyond mental health services in planning and developing the service

    Mental health in Europe during the COVID-19 pandemic: a systematic review

    Get PDF
    The COVID-19 pandemic caused immediate and far-reaching disruption to society, the economy, and health-care services. We synthesised evidence on the effect of the pandemic on mental health and mental health care in high-income European countries. We included 177 longitudinal and repeated cross-sectional studies comparing prevalence or incidence of mental health problems, mental health symptom severity in people with pre-existing mental health conditions, or mental health service use before versus during the pandemic, or between different timepoints of the pandemic. We found that epidemiological studies reported higher prevalence of some mental health problems during the pandemic compared with before it, but that in most cases this increase reduced over time. Conversely, studies of health records showed reduced incidence of new diagnoses at the start of the pandemic, which further declined during 2020. Mental health service use also declined at the onset of the pandemic but increased later in 2020 and through 2021, although rates of use did not return to pre-pandemic levels for some services. We found mixed patterns of effects of the pandemic on mental health and social outcome for adults already living with mental health conditions

    Mental health in Europe during the COVID-19 pandemic: a systematic review

    Get PDF
    The COVID-19 pandemic caused immediate and far-reaching disruption to society, the economy, and health-care services. We synthesised evidence on the effect of the pandemic on mental health and mental health care in high-income European countries. We included 177 longitudinal and repeated cross-sectional studies comparing prevalence or incidence of mental health problems, mental health symptom severity in people with pre-existing mental health conditions, or mental health service use before versus during the pandemic, or between different timepoints of the pandemic. We found that epidemiological studies reported higher prevalence of some mental health problems during the pandemic compared with before it, but that in most cases this increase reduced over time. Conversely, studies of health records showed reduced incidence of new diagnoses at the start of the pandemic, which further declined during 2020. Mental health service use also declined at the onset of the pandemic but increased later in 2020 and through 2021, although rates of use did not return to pre-pandemic levels for some services. We found mixed patterns of effects of the pandemic on mental health and social outcome for adults already living with mental health conditions

    Framework-based qualitative analysis of free responses of Large Language Models: Algorithmic fidelity

    No full text
    Today, with the advent of Large-scale generative Language Models (LLMs) it is now possible to simulate free responses to interview questions such as those traditionally analyzed using qualitative research methods. Qualitative methodology encompasses a broad family of techniques involving manual analysis of open-ended interviews or conversations conducted freely in natural language. Here we consider whether artificial “silicon participants” generated by LLMs may be productively studied using qualitative analysis methods in such a way as to generate insights that could generalize to real human populations. The key concept in our analysis is algorithmic fidelity, a validity concept capturing the degree to which LLM-generated outputs mirror human sub-populations’ beliefs and attitudes. By definition, high algorithmic fidelity suggests that latent beliefs elicited from LLMs may generalize to real humans, whereas low algorithmic fidelity renders such research invalid. Here we used an LLM to generate interviews with “silicon participants” matching specific demographic characteristics one-for-one with a set of human participants. Using framework-based qualitative analysis, we showed the key themes obtained from both human and silicon participants were strikingly similar. However, when we analyzed the structure and tone of the interviews we found even more striking differences. We also found evidence of a hyper-accuracy distortion. We conclude that the LLM we tested (GPT-3.5) does not have sufficient algorithmic fidelity to expect in silico research on it to generalize to real human populations. However, rapid advances in artificial intelligence raise the possibility that algorithmic fidelity may improve in the future. Thus we stress the need to establish epistemic norms now around how to assess the validity of LLM-based qualitative research, especially concerning the need to ensure the representation of heterogeneous lived experiences.</p

    Informing behaviour change intervention design using systematic review with Bayesian meta-analysis: physical activity in heart failure

    Get PDF
    Embracing the Bayesian approach, we aimed to synthesise evidence regarding barriers and enablers to physical activity in heart failure (HF) in a way that can inform behaviour change intervention development. This approach helps in estimating and quantifying the uncertainty in the evidence and facilitates the synthesis of qualitative and quantitative studies. Qualitative and observational studies investigating barriers and enablers to physical activity in adults diagnosed with HF were included in this systematic review with a Bayesian meta-analysis. Qualitative evidence was annotated using the Theoretical Domains Framework and represented as a prior distribution using an expert elicitation task. The maximum a posteriori probability (MAP) was calculated as a summary statistic for the probability distribution for the log OR value estimating the relationship between physical activity and each determinant, according to qualitative evidence alone, quantitative evidence, and qualitative and quantitative evidence combined. The dispersion in the probability distribution for log OR associated with each barrier or enabler was used to evaluate the level of uncertainty in the evidence. Wide, medium, and narrow dispersion (SD) corresponded to high, moderate, and low uncertainty in the evidence, respectively. Evidence from three qualitative and 16 (N = 2739) quantitative studies was synthesised. High pro-b-type natriuretic peptide, pro-BNP (MAP value for log OR = -1.16; 95% CrI: [-1.21; -1.11]) and self-reported symptoms (MAP for log OR = 0.48; 95% CrI: [0.40; 0.55]) were suggested as barriers to physical activity with narrow distribution dispersion (SD = 0.18 and 0.19, respectively). Modifiable barriers were symptom distress (MAP for log OR = -0.46; 95% CrI: [-0.68; -0.24]), and negative attitude (MAP for log OR = -0.40; 95% CrI: [-0.49; -0.31]), SD = 0.36 and 0.26, respectively. Modifiable enablers were social support (MAP for log OR = 0.56; 95% CrI: [0.48; 0.63]), self-efficacy (MAP for log OR = 0.43; 95% CrI: [0.32; 0.54]), positive physical activity attitude (MAP for log OR = 0.92; 95% CrI: [0.77; 1.06]), SD = 0.26, 0.37, and 0.36, respectively. This work extends the limited research on the modifiable barriers and enablers for physical activity by individuals living with HF
    corecore