How to communicate evidence to patients.

All medical treatments have potential harms as well as benefits, and it’s vital that everyone has a good understanding of what these might be, how dramatic they might be, and how likely. In fact, in the UK, the Montgomery judgement in the supreme court in 2015 has made it a legal necessity for patients to be given comprehensible, personally relevant information about all reasonable treatment options – including none. So, how should we ensure this good, clear communication of relevant evidence?Funded by the Winton Centre (which is supported by the David & Claudia Harding Foundation

How well did experts and laypeople forecast the size of the COVID-19 pandemic?

Throughout the COVID-19 pandemic, social and traditional media have disseminated predictions from experts and nonexperts about its expected magnitude. How accurate were the predictions of 'experts'-individuals holding occupations or roles in subject-relevant fields, such as epidemiologists and statisticians-compared with those of the public? We conducted a survey in April 2020 of 140 UK experts and 2,086 UK laypersons; all were asked to make four quantitative predictions about the impact of COVID-19 by 31 Dec 2020. In addition to soliciting point estimates, we asked participants for lower and higher bounds of a range that they felt had a 75% chance of containing the true answer. Experts exhibited greater accuracy and calibration than laypersons, even when restricting the comparison to a subset of laypersons who scored in the top quartile on a numeracy test. Even so, experts substantially underestimated the ultimate extent of the pandemic, and the mean number of predictions for which the expert intervals contained the actual outcome was only 1.8 (out of 4), suggesting that experts should consider broadening the range of scenarios they consider plausible. Predictions of the public were even more inaccurate and poorly calibrated, suggesting that an important role remains for expert predictions as long as experts acknowledge their uncertainty

Do colored cells in risk matrices affect decision-making and risk perception? Insights from randomized controlled studies

Risk matrices communicate the likelihood and potential impact of risks and are often used to inform decision-making around risk mitigations. The merits and demerits of risk matrices in general have been discussed extensively, yet little attention has been paid to the potential influence of color in risk matrices on their users. We draw from fuzzy-trace theory and hypothesize that when color is present, individuals are likely to place greater value on reducing risks that cross color boundaries (i.e., the boundary-crossing effect), leading to sub-optimal decision making. In two randomized controlled studies, employing forced-choice and willingness-to-pay measures to investigate the boundary-crossing effect in two different color formats for risk matrices, we find preliminary evidence to support our hypotheses that color can influence decision making. The evidence also suggests that the boundary-crossing effect is only present in, or is stronger for, higher numeracy individuals. We therefore recommend that designers should consider avoiding color in risk matrices, particularly in situations where these are likely to be used by highly numerate individuals, if the communication goal is to inform in an unbiased way

The adverse effects of bisphosphonates in breast cancer: A systematic review and network meta-analysis.

BACKGROUND: Bisphosphonate drugs can be used to improve the outcomes of women with breast cancer. Whilst many meta-analyses have quantified their potential benefits for patients, attempts at comprehensive quantification of potential adverse effects have been limited. We undertook a meta-analysis with novel methodology to identify and quantify these adverse effects. METHODS: We systematically reviewed randomised controlled trials in breast cancer where at least one of the treatments was a bisphosphonate (zoledronic acid, ibandronate, pamidronate, alendronate or clodronate). Neoadjuvant, adjuvant and metastatic settings were examined. Primary outcomes were adverse events of any type or severity (excluding death). We carried out pairwise and network meta-analyses to estimate the size of any adverse effects potentially related to bisphosphonates. In order to ascertain whether adverse effects differed by individual factors such as age, or interacted with other common adjuvant breast cancer treatments, we examined individual-level patient data for one large trial, AZURE. FINDINGS: We identified 56 trials that reported adverse data, which included a total of 29,248 patients (18,301 receiving bisphosphonate drugs versus 10,947 not). 24 out of the 103 different adverse outcomes analysed showed a statistically and practically significant increase in patients receiving a bisphosphonate drug compared with those not (2 additional outcomes that appeared statistically significant came only from small studies with low event counts and no clinical suspicion so are likely artifacts). Most of these 24 are already clinically recognised: 'flu-like symptoms, fever, headache and chills; increased bone pain, arthralgia, myalgia, back pain; cardiac events, thromboembolic events; hypocalcaemia and osteonecrosis of the jaw; as well as possibly stiffness and nausea. Oral clodronate appeared to increase the risk of vomiting and diarrhoea (which may also be increased by other bisphosphonates), and there may be some hepatotoxicity. Four additional potential adverse effects emerged for bisphosphonate drugs in this analysis which have not classically be recognised: fatigue, neurosensory problems, hypertonia/muscle spasms and possibly dysgeusia. Several symptoms previously reported as potential side effects in the literature were not significantly increased in this analysis: constipation, insomnia, respiratory problems, oedema or thirst/dry mouth. Individual patient-level data and subgroup analysis revealed little variation in side effects between women of different ages or menopausal status, those with metastatic versus non-metastatic cancer, or between women receiving different concurrent breast cancer therapies. CONCLUSIONS: This meta-analysis has produced estimates for the absolute frequencies of a range of side effects significantly associated with bisphosphonate drugs when used by breast cancer patients. These results show good agreement with previous literature on the subject but are the first systematic quantification of side effects and their severities. However, the analysis is limited by the availability and quality of data on adverse events, and the potential for bias introduced by a lack of standards for reporting of such events. We therefore present a table of adverse effects for bisphosphonates, identified and quantified to the best of our ability from a large number of trials, which we hope can be used to improve the communication of the potential harms of these drugs to patients and their healthcare providers

The effects of quality of evidence communication on perception of public health information about COVID-19: Two randomised controlled trials.

BACKGROUND: The quality of evidence about the effectiveness of non-pharmaceutical health interventions is often low, but little is known about the effects of communicating indications of evidence quality to the public. METHODS: In two blinded, randomised, controlled, online experiments, US participants (total n = 2140) were shown one of several versions of an infographic illustrating the effectiveness of eye protection in reducing COVID-19 transmission. Their trust in the information, understanding, feelings of effectiveness of eye protection, and the likelihood of them adopting it were measured. FINDINGS: Compared to those given no quality cues, participants who were told the quality of the evidence on eye protection was 'low', rated the evidence less trustworthy (p = .001, d = 0.25), and rated it as subjectively less effective (p = .018, d = 0.19). The same effects emerged compared to those who were told the quality of the evidence was 'high', and in one of the two studies, those shown 'low' quality of evidence said they were less likely to use eye protection (p = .005, d = 0.18). Participants who were told the quality of the evidence was 'high' showed no statistically significant differences on these measures compared to those given no information about evidence quality. CONCLUSIONS: Without quality of evidence cues, participants responded to the evidence about the public health intervention as if it was high quality and this affected their subjective perceptions of its efficacy and trust in the provided information. This raises the ethical dilemma of weighing the importance of transparently stating when the evidence base is actually low quality against evidence that providing such information can decrease trust, perception of intervention efficacy, and likelihood of adopting it

How do the UK public interpret COVID-19 test results? Comparing the impact of official information about results and reliability used in the UK, USA and New Zealand: a randomised controlled trial.

OBJECTIVES: To assess the effects of different official information on public interpretation of a personal COVID-19 PCR test result. DESIGN: A 5×2 factorial, randomised, between-subjects experiment, comparing four wordings of information about the test result and a control arm of no additional information; for both positive and negative test results. SETTING: Online experiment using recruitment platform Respondi. PARTICIPANTS: UK participants (n=1744, after a pilot of n=1657) quota-sampled to be proportional to the UK national population on age and sex. INTERVENTIONS: Participants were given a hypothetical COVID-19 PCR test result for 'John' who was presented as having a 50% chance of having COVID-19 based on symptoms alone. Participants were randomised to receive either a positive or negative result for 'John', then randomised again to receive either no more information, or text information on the interpretation of COVID-19 test results copied in September 2020 from the public websites of the UK's National Health Service, the USA's Centers for Disease Control, New Zealand's Ministry of Health or a modified version of the UK's wording. Information identifying the source of the wording was removed. MAIN OUTCOME MEASURES: Participants were asked 'What is your best guess as to the percent chance that John actually had COVID-19 at the time of his test, given his result?'; questions about their feelings of trustworthiness in the result, their perceptions of the quality of the underlying evidence and what action they felt 'John' should take in the light of his result. RESULTS: Of those presented with a positive COVID-19 test result for 'John', the mean estimate of the probability that he had the virus was 73% (71.5%-74.5%); for those presented with a negative result, 38% (36.7%-40.0%). There was no main effect of information (wording) on these means. However, those participants given the official information from the UK website, which did not mention the possibility of false negatives or false positives, were more likely to give a categorical (100% or 0%) answer (UK: 68/343, 19.8% (15.9%-24.4%); control group: 42/356, 11.8% (8.8%-15.6%)); the reverse was true for those viewing the New Zealand (NZ) wording, which highlighted the uncertainties most explicitly (20/345: 5.8% (3.7%-8.8%)). Aggregated across test result (positive/negative), there was a main effect of wording (p<0.001) on beliefs about how 'John' should behave, with those seeing the NZ wording marginally more likely to agree that 'John' should continue to self-isolate than those viewing the control or the UK wording. The proportion of participants who felt that a symptomatic individual who tests negative definitely should not self-isolate was highest among those viewing the UK wording (31/178, 17.4% (12.5%-23.7%)), and lowest among those viewing the NZ wording (6/159, 3.8% (1.6%-8.2%)). Although the NZ wording was rated harder to understand, participants reacted to the uncertainties given in the text in the expected direction: there was a small main effect of wording on trust in the result (p=0.048), with people perceiving the test result as marginally less trustworthy after having read the NZ wording compared with the UK wording. Positive results were generally viewed as more trustworthy and as having higher quality of evidence than negative results (both p<0.001). CONCLUSIONS: The public's default assessment of the face value of both the positive and negative test results (control group) indicate an awareness that test results are not perfectly accurate. Compared with other messaging tested, participants shown the UK's 2020 wording about the interpretation of the test results appeared to interpret the results as more definitive than is warranted. Wording that acknowledges uncertainty can help people to have a more nuanced and realistic understanding of what a COVID-19 test result means, which supports decision making and behavioural response. PREREGISTRATION AND DATA REPOSITORY: Preregistration of pilot at osf.io/8n62f, preregistration of main experiment at osf.io/7rcj4, data and code available online (osf.io/pvhba)

Redevelopment of the Predict: Breast Cancer website and recommendations for developing interfaces to support decision-making.

OBJECTIVES: To develop a new interface for the widely used prognostic breast cancer tool: Predict: Breast Cancer. To facilitate decision-making around post-surgery breast cancer treatments. To derive recommendations for communicating the outputs of prognostic models to patients and their clinicians. METHOD: We employed a user-centred design process comprised of background research and iterative testing of prototypes with clinicians and patients. Methods included surveys, focus groups and usability testing. RESULTS: The updated interface now caters to the needs of a wider audience through the addition of new visualisations, instantaneous updating of results, enhanced explanatory information and the addition of new predictors and outputs. A programme of future research was identified and is now underway, including the provision of quantitative data on the adverse effects of adjuvant breast cancer treatments. Based on our user-centred design process, we identify six recommendations for communicating the outputs of prognostic models including the need to contextualise statistics, identify and address gaps in knowledge, and the critical importance of engaging with prospective users when designing communications. CONCLUSIONS: For prognostic algorithms to fulfil their potential to assist with decision-making they need carefully designed interfaces. User-centred design puts patients and clinicians needs at the forefront, allowing them to derive the maximum benefit from prognostic models

How People Understand Risk Matrices, and How Matrix Design Can Improve their Use: Findings from Randomized Controlled Studies.

Risk matrices are a common way to communicate the likelihood and potential impacts of a variety of risks. Until now, there has been little empirical work on their effectiveness in supporting understanding and decision making, and on how different design choices affect these. In this pair of online experiments (total n = 2699), we show that risk matrices are not always superior to text for the presentation of risk information, and that a nonlinear/geometric labeling scheme helps matrix comprehension (when the likelihood/impact scales are nonlinear). To a lesser degree, results suggested that changing the shape of the matrix so that cells increase in size nonlinearly facilitates comprehension as compared to text alone, and that comprehension might be enhanced by integrating further details about the likelihood and impact onto the axes of the matrix rather than putting them in a separate key. These changes did not affect participants' preference for reducing impact over reducing likelihood when making decisions about risk mitigation. We recommend that designers of risk matrices consider these changes to facilitate better understanding of relationships among risks

How Accurate Are Accuracy-Nudge Interventions? A Preregistered Direct Replication of Pennycook et al. (2020).

Funder: Defense Advanced Research Projects Agency; FundRef: https://doi.org/10.13039/100000185Funder: Winton Centre for Risk & Evidence CommunicationFunder: David & Claudia Harding FoundationAs part of the Systematizing Confidence in Open Research and Evidence (SCORE) program, the present study consisted of a two-stage replication test of a central finding by Pennycook et al. (2020), namely that asking people to think about the accuracy of a single headline improves "truth discernment" of intentions to share news headlines about COVID-19. The first stage of the replication test (n = 701) was unsuccessful (p = .67). After collecting a second round of data (additional n = 882, pooled N = 1,583), we found a small but significant interaction between treatment condition and truth discernment (uncorrected p = .017; treatment: d = 0.14, control: d = 0.10). As in the target study, perceived headline accuracy correlated with treatment impact, so that treatment-group participants were less willing to share headlines that were perceived as less accurate. We discuss potential explanations for these findings and an unreported change in the hypothesis (but not the analysis plan) from the preregistration in the original study

Recommendations for designing genetic test reports to be understood by patients and non-specialists.

Funder: David and Claudia Harding FoundationPatients and non-specialist healthcare professionals are increasingly expected to understand and interpret the results of genetic or genomic testing. These results are currently reported using a variety of templates, containing different amounts, levels, and layouts of information. We set out to establish a set of recommendations for communicating genetic test results to non-expert readers. We employed a qualitative-descriptive study design with user-centred design principles, including a mixture of in-person semi-structured interviews and online questionnaires with patients, healthcare professionals and the general public. The resulting recommendations and example template include providing at-a-glance comprehension of what the test results mean for the patient; suggested next steps; and details of further information and support. Separation and inclusion of technical methodological details enhances non-specialists' understanding, while retaining important information for specialists and the patients' records. The recommendations address the high-level needs of patients and their non-specialist clinicians when receiving genetic test results. These recommendations provide a solid foundation for the major content and structure of reports, and we recommend further engagement with patients and clinicians to tailor reports to specific types of test and results