Framework-Based Qualitative Analysis of Free Responses of Large Language Models: Algorithmic Fidelity

Today, using Large-scale generative Language Models (LLMs) it is possible to simulate free responses to interview questions like those traditionally analyzed using qualitative research methods. Qualitative methodology encompasses a broad family of techniques involving manual analysis of open-ended interviews or conversations conducted freely in natural language. Here we consider whether artificial "silicon participants" generated by LLMs may be productively studied using qualitative methods aiming to produce insights that could generalize to real human populations. The key concept in our analysis is algorithmic fidelity, a term introduced by Argyle et al. (2023) capturing the degree to which LLM-generated outputs mirror human sub-populations' beliefs and attitudes. By definition, high algorithmic fidelity suggests latent beliefs elicited from LLMs may generalize to real humans, whereas low algorithmic fidelity renders such research invalid. Here we used an LLM to generate interviews with silicon participants matching specific demographic characteristics one-for-one with a set of human participants. Using framework-based qualitative analysis, we showed the key themes obtained from both human and silicon participants were strikingly similar. However, when we analyzed the structure and tone of the interviews we found even more striking differences. We also found evidence of the hyper-accuracy distortion described by Aher et al. (2023). We conclude that the LLM we tested (GPT-3.5) does not have sufficient algorithmic fidelity to expect research on it to generalize to human populations. However, the rapid pace of LLM research makes it plausible this could change in the future. Thus we stress the need to establish epistemic norms now around how to assess validity of LLM-based qualitative research, especially concerning the need to ensure representation of heterogeneous lived experiences.Comment: 46 pages, 5 tables, 5 figure

The association between loneliness and other constructs of social connectedness and the probability of desire for hastened death, euthanasia and assisted suicide: systematic review

Background Euthanasia and assisted suicide (EAS) requests are common in countries where they are legal. Loneliness and social isolation are modifiable risk factors for mental illness and suicidal behaviour and are common in terminal illness. Our objective was to summarise available literature to clarify whether these and related measures of social connectedness might contribute to requests for EAS. Methods We conducted a pre-registered (PROSPERO CRD42019160508) systematic review and narrative synthesis of quantitative literature investigating associations between social connectedness and a) requested/actual EAS, b) attitudes towards EAS, and c) a desire for hastened death (DHD) by searching six databases (PsycINFO, MEDLINE, EMBASE, Scopus, Web of Science, Google Scholar) from inception to November 2022, rating eligible peer-reviewed, empirical studies using the QATSO quality assessment tool. Results We identified 37 eligible studies that investigated associations with a) requested/actual EAS (n = 9), b) attitudes to EAS (n = 16), and c) DHD (n = 14), with limited overlap, including 17,359 participants. The majority (62%) were rated at medium/high risk of bias. Focussing our narrative synthesis on the more methodologically sound studies, we found no evidence to support an association between different constructs of social connectedness and requested or actual EAS, and very little evidence to support an association with attitudes to EAS or an association with DHD. Conclusions Our findings for all age groups are consistent with a those of a previous systematic review focussed on older adults and suggest that poor social connectedness is not a clear risk factor for EAS or for measures more distally related to EAS. However, we acknowledge low study quality in some studies in relation to sampling, unvalidated exposure/outcome measures, cross-sectional design, unadjusted analyses, and multiple testing. Clinical assessment should focus on modifying established risk factors for suicide and EAS, such as hopelessness and depression, as well as improving any distressing aspects of social disconnectedness to improve quality of life

'Beyond places of safety' - a qualitative study exploring the implementation of mental health crisis care innovations across England

BACKGROUND: Mental health acute and crisis care consumes a large share of mental health budgets internationally but is often experienced as unsatisfactory and difficult to access. As a result, there is an increasing move towards developing innovative community crisis services, to improve patient experience and relieve pressure on inpatient and emergency services. This study aims to understand what helps and hinders the implementation of innovative mental health crisis care projects in England. METHODS: Using a qualitative approach, 18 interviews were conducted with crisis care service managers exploring their experiences and views of the development and implementation of their service developed with support from an English national capital funding programme. A framework analysis was conducted informed by implementation science. RESULTS: Key facilitators to implementation of innovative crisis services included bottom-up development, service user involvement, strong collaborative working, and leadership and management buy-in. Key barriers that affected the projects implementation included the complexities of crisis care, workforce challenges and resourcing issues. CONCLUSION: There is a recognised need to improve, update, and innovate current crisis care offers. Results from this study suggest that a range of models can help address the heterogenous needs of local populations and that new approaches can be implemented where they utilise a whole-systems approach, involving service users and relevant professional stakeholders beyond mental health services in planning and developing the service

Mental health in Europe during the COVID-19 pandemic: a systematic review

The COVID-19 pandemic caused immediate and far-reaching disruption to society, the economy, and health-care services. We synthesised evidence on the effect of the pandemic on mental health and mental health care in high-income European countries. We included 177 longitudinal and repeated cross-sectional studies comparing prevalence or incidence of mental health problems, mental health symptom severity in people with pre-existing mental health conditions, or mental health service use before versus during the pandemic, or between different timepoints of the pandemic. We found that epidemiological studies reported higher prevalence of some mental health problems during the pandemic compared with before it, but that in most cases this increase reduced over time. Conversely, studies of health records showed reduced incidence of new diagnoses at the start of the pandemic, which further declined during 2020. Mental health service use also declined at the onset of the pandemic but increased later in 2020 and through 2021, although rates of use did not return to pre-pandemic levels for some services. We found mixed patterns of effects of the pandemic on mental health and social outcome for adults already living with mental health conditions

Mental health in Europe during the COVID-19 pandemic: a systematic review

The COVID-19 pandemic caused immediate and far-reaching disruption to society, the economy, and health-care services. We synthesised evidence on the effect of the pandemic on mental health and mental health care in high-income European countries. We included 177 longitudinal and repeated cross-sectional studies comparing prevalence or incidence of mental health problems, mental health symptom severity in people with pre-existing mental health conditions, or mental health service use before versus during the pandemic, or between different timepoints of the pandemic. We found that epidemiological studies reported higher prevalence of some mental health problems during the pandemic compared with before it, but that in most cases this increase reduced over time. Conversely, studies of health records showed reduced incidence of new diagnoses at the start of the pandemic, which further declined during 2020. Mental health service use also declined at the onset of the pandemic but increased later in 2020 and through 2021, although rates of use did not return to pre-pandemic levels for some services. We found mixed patterns of effects of the pandemic on mental health and social outcome for adults already living with mental health conditions

Approaches to improving mental healthcare for autistic people: systematic review.

Autistic people have a high likelihood of developing mental health difficulties but a low chance of receiving effective mental healthcare. Therefore, there is a need to identify and examine strategies to improve mental healthcare for autistic people. To identify strategies that have been implemented to improve access, experiences of care and mental health outcomes for autistic adults, and to examine evidence on their acceptability, feasibility and effectiveness. A co-produced systematic review was conducted. MEDLINE, PsycINFO, CINHAL, medRxiv and PsyArXiv were searched. We included all study designs reporting acceptability or feasibility outcomes and empirical quantitative study designs reporting effectiveness outcomes. Data were synthesised using a narrative approach. A total of 30 articles were identified. These included 16 studies of adapted mental health interventions, eight studies of service improvements and six studies of bespoke mental health interventions developed for autistic people. There was no conclusive evidence on effectiveness. However, most bespoke and adapted approaches appeared to be feasible and acceptable. Identified adaptations appeared to be acceptable and feasible, including increasing knowledge and detection of autism, providing environmental adjustments and communication accommodations, accommodating individual differences and modifying the structure and content of interventions. Many identified strategies are feasible and acceptable, and can be readily implemented in services with the potential to make mental healthcare more suitable for autistic people, but important research gaps remain. Future research should address these and investigate a co-produced package of service improvement measures

Shared Decision-Making in Risk Assessment and Risk Management with People with Severe Mental Illness

Risk Assessment (RA) and Risk Management (RM) are fundamental processes in the delivery of safe and effective mental health care, yet studies have shown that service users are often not directly involved or are unaware that an assessment has taken place. Shared Decision-Making (SDM) in mental health systems is supported by research and advocated in policy. The primary aim of this programme of research was to identify the barriers and enablers to SDM in RA and RM,from the perspective of mental health professionals and service users. Study 1 synthesised the best available evidence on mental health professionals’ experiences of SDM in RA and RM. The findings of this review indicate that SDM is not a concept commonly used in mental health services when exploring processes of RA and RM. The key barriers identified were ‘power and best interest’ and ‘my professional role and responsibility’. Key enablers were ‘therapeutic relationship’ and ‘value collaboration’. An exploratory sequential mixed-methods design and psychological theory were applied in study 2 (a qualitative interview study) and study 3 (a cross-sectional survey), which explored mental health professionals and service users’ perceived barriers and enablers to SDM in RA and RM. The findings show that stakeholders are motivated to work collaboratively but experience a complex range of social, cultural, and environmental factors that influence theiropportunity and capability to achieve SDM in RA and RM. The findings of this thesis can be used to develop targeted interventions aiming to increase SDM in RA and RM. However, further empirical work is needed to validate these findings and determine the acceptability and feasibility of an intervention with key stakeholders

How do speech-and-language therapists address the psychosocial well-being of people with aphasia? Results of a UK online survey:Addressing psychosocial needs following aphasia

Background The psychosocial impact of stroke and aphasia is considerable. Aims To explore UK speech‐and‐language therapists’ (SLTs) clinical practice in addressing the psychological and social needs of people with aphasia, including their experiences of working with mental health professionals. Methods & Procedures A 22‐item online survey was distributed to UK SLTs via the British Aphasiology Society mailing list and Clinical Excellence Networks. Results were analysed using descriptive statistics and qualitative content analysis. Outcomes & Results UK SLTs (n = 124) overwhelmingly considered that addressing psychological well‐being (93%) and social participation (99%) was part of their role. To achieve this, they frequently/very frequently used supportive listening (100%) and selected holistic goals collaboratively with clients (87%), including social goals (83%). However, only 42% felt confident in addressing the psychological needs of clients. The main barriers to addressing psychosocial well‐being were time/caseload pressures (72%); feeling under‐skilled/lack of training (64%), and lack of ongoing support (61%). The main barriers to referring on to mental health professionals were that mental health professionals were perceived as under‐skilled when working with people with aphasia (44%); were difficult to access (41%); and provided only a limited service (37%). A main theme from the free‐text responses was a concern that those with aphasia, particularly more severe aphasia, received inadequate psychological support due to the stretched nature of many mental health services; mental health professionals lacking skills working with aphasia; and SLTs lacking the necessary time, training and support. The main enablers to addressing psychosocial well‐being were collaborative working between SLTs and stroke‐specialist clinical psychologists; SLTs with training in providing psychological and social therapy; and ongoing support provided by the voluntary sector. Conclusions & Implications The vast majority of SLTs consider the psychosocial well‐being of their clients, and work collaboratively with people with aphasia in selecting holistic goals. It is, however, of concern that most respondents felt they lacked confidence and received insufficient training to address psychological well‐being. In order to improve psychological services for this client group, there is a strong case that stroke‐specialist mental health professionals should strive to make their service truly accessible to people with even severe aphasia, which may involve working more closely with SLTs. Further, improving the skills and confidence of SLTs may be an effective way of addressing psychological distress in people with aphasia

Framework-based qualitative analysis of free responses of Large Language Models: Algorithmic fidelity

Today, with the advent of Large-scale generative Language Models (LLMs) it is now possible to simulate free responses to interview questions such as those traditionally analyzed using qualitative research methods. Qualitative methodology encompasses a broad family of techniques involving manual analysis of open-ended interviews or conversations conducted freely in natural language. Here we consider whether artificial “silicon participants” generated by LLMs may be productively studied using qualitative analysis methods in such a way as to generate insights that could generalize to real human populations. The key concept in our analysis is algorithmic fidelity, a validity concept capturing the degree to which LLM-generated outputs mirror human sub-populations’ beliefs and attitudes. By definition, high algorithmic fidelity suggests that latent beliefs elicited from LLMs may generalize to real humans, whereas low algorithmic fidelity renders such research invalid. Here we used an LLM to generate interviews with “silicon participants” matching specific demographic characteristics one-for-one with a set of human participants. Using framework-based qualitative analysis, we showed the key themes obtained from both human and silicon participants were strikingly similar. However, when we analyzed the structure and tone of the interviews we found even more striking differences. We also found evidence of a hyper-accuracy distortion. We conclude that the LLM we tested (GPT-3.5) does not have sufficient algorithmic fidelity to expect in silico research on it to generalize to real human populations. However, rapid advances in artificial intelligence raise the possibility that algorithmic fidelity may improve in the future. Thus we stress the need to establish epistemic norms now around how to assess the validity of LLM-based qualitative research, especially concerning the need to ensure the representation of heterogeneous lived experiences.</p

Informing behaviour change intervention design using systematic review with Bayesian meta-analysis: physical activity in heart failure

Embracing the Bayesian approach, we aimed to synthesise evidence regarding barriers and enablers to physical activity in heart failure (HF) in a way that can inform behaviour change intervention development. This approach helps in estimating and quantifying the uncertainty in the evidence and facilitates the synthesis of qualitative and quantitative studies. Qualitative and observational studies investigating barriers and enablers to physical activity in adults diagnosed with HF were included in this systematic review with a Bayesian meta-analysis. Qualitative evidence was annotated using the Theoretical Domains Framework and represented as a prior distribution using an expert elicitation task. The maximum a posteriori probability (MAP) was calculated as a summary statistic for the probability distribution for the log OR value estimating the relationship between physical activity and each determinant, according to qualitative evidence alone, quantitative evidence, and qualitative and quantitative evidence combined. The dispersion in the probability distribution for log OR associated with each barrier or enabler was used to evaluate the level of uncertainty in the evidence. Wide, medium, and narrow dispersion (SD) corresponded to high, moderate, and low uncertainty in the evidence, respectively. Evidence from three qualitative and 16 (N = 2739) quantitative studies was synthesised. High pro-b-type natriuretic peptide, pro-BNP (MAP value for log OR = -1.16; 95% CrI: [-1.21; -1.11]) and self-reported symptoms (MAP for log OR = 0.48; 95% CrI: [0.40; 0.55]) were suggested as barriers to physical activity with narrow distribution dispersion (SD = 0.18 and 0.19, respectively). Modifiable barriers were symptom distress (MAP for log OR = -0.46; 95% CrI: [-0.68; -0.24]), and negative attitude (MAP for log OR = -0.40; 95% CrI: [-0.49; -0.31]), SD = 0.36 and 0.26, respectively. Modifiable enablers were social support (MAP for log OR = 0.56; 95% CrI: [0.48; 0.63]), self-efficacy (MAP for log OR = 0.43; 95% CrI: [0.32; 0.54]), positive physical activity attitude (MAP for log OR = 0.92; 95% CrI: [0.77; 1.06]), SD = 0.26, 0.37, and 0.36, respectively. This work extends the limited research on the modifiable barriers and enablers for physical activity by individuals living with HF