Teach people to think critically about claims and comparisons using these concepts, urge Andrew D. Oxman and an alliance of 24 researchers — they will make better decisions

A. Nsangi

A. Petrosino

Academy of Medical Sciences

Andrew D. Oxman

Andrew Pullin

Andy Haines

Anel Schoonees

Astrid Dahlgren

Brandy Maynard

Carl Heneghan

D. Renmans

Eric Barends

H. L. Bouygues

Hazel Roddam

Iain Chalmers

Informed Health Choices group

Janet Stott

Jeffrey K. Aronson

Joe Chislett

Jonathan Sharples

Luke Vale

Marie Gaarder

Marnie Brennan

Matt Oxman

N. Haber

Nerys Thomas

Nicola Randall

P. C. Abrami

P. Sumner

Peter Cunliffe-Jones

Raymond Tallis

Robert Boruch

Robert Matthews

Ruth Stewart

Nature

An alliance of researchers lays out a framework for taking decisions based on thinking critically about claims and comparisons. Everyone makes claims about what works. Politicians claim that stop and search will reduce violent crime; friends claim that vaccines cause autism; advertisers claim that natural food is healthy. One group of scientists claims that "deworming" programmes (giving deworming pills to all school children in affected areas) improve school performance and health, calling deworming one of the most potent anti-poverty interventions of our time. Another that deworming does not improve either school performance or health. Unfortunately, people often fail to think critically about the trustworthiness of claims, including policy makers weighing claims made by scientists. Schools do not do enough to prepare young people to think critically 1. So many people struggle to assess the trustworthiness of evidence. As a consequence, they may not make informed choices. To address this deficit, we present here a general tool: Key Concepts for Making Informed Choices (Table 1, with examples in Box 2). We hope scientists and professionals in all fields will use, evolve and evaluate it. The tool was adapted, drawing on the expertise of two dozen researchers, from a framework developed for healthcare 2 (Box 1). Ideally, the Key Concepts for Making Informed Choices should be embedded in education for citizens of all ages. This should be done using learning resources and teaching strategies that have been evaluated and shown to be effective. Trustworthy evidence People are flooded with information. Simply giving them more is unlikely to be helpful unless its value is understood. A recent survey in the UK showed that only about a third of the public trust evidence from medical research; about two-thirds trust the experiences of friends and family 3. Not all evidence is created equal. Yet people often don't appreciate which claims are more trustworthy than others; what sort of comparisons are needed to evaluate different proposals fairly; or what other information needs to be considered to inform good choices. For example, many people don't grasp that things can be associated without one necessarily causing the other. The media sometimes perpetuates this problem by using language suggesting that cause-and-effect has been established when it has not 4 , using statements such as "coffee can kill you", or "drinking one glass of beer a day can make you live longer". Worse, exaggerated causal claims often pepper university and journal press releases 5. Studies that make fair comparisons are vital, yet people often don't know how to assess the validity of research. Systematic reviews that synthesise well-designed studies relevant to clearly-defined questions are more trustworthy than haphazard observations; they are less susceptible to biases (systematic distortions) and the play of chance (random errors). Yet results from single studies are often reported in isolation, as facts. Hence the familiar flip-flopping headlines such as "chocolate is good for you", followed the next week by "chocolate is bad for you"

Aronson, Jeffrey K.

Barends, Eric

Boruch, Robert

Brennan, Marnie

Chalmers, Iain

Chislett, Joe

Cunliffe-Jones, Peter

Dahlgren, Astrid

Gaarder, Marie

Haines, Andy

Heneghan, Carl

Matthews, Robert

Maynard, Brandy

Oxman, Andrew D.

Oxman, Matt

Pullin, Andrew

Randall, Nicola

Roddam, Hazel

Schoonees, Anel

Sharples, Jonathan

Stewart, Ruth

Stott, Janet

Tallis, Raymond

Thomas, Nerys

Vale, Luke

Repository@Nottingham

DRAFT 25 July 2019 1  Key Concepts for making informed Choices An alliance of researchers lays out a framework for taking decisions based on thinking critically about claims and comparisons.   Everyone makes claims about what works. Politicians claim that stop and search will reduce violent crime; friends claim that vaccines cause autism; advertisers claim that natural food is healthy. One group of scientists claims that "deworming" programmes (giving deworming pills to all school children in affected areas) improve school performance and health, calling deworming one of the most potent anti-poverty interventions of our time. Another that deworming does not improve either school performance or health. Unfortunately, people often fail to think critically about the trustworthiness of claims, including policy makers weighing claims made by scientists. Schools do not do enough to prepare young people to think critically1. So many people struggle to assess the trustworthiness of evidence. As a consequence, they may not make informed choices.  To address this deficit, we present here a general tool: Key Concepts for Making Informed Choices (Table 1, with examples in Box 2). We hope scientists and professionals in all fields will use, evolve and evaluate it. The tool was adapted, drawing on the expertise of two dozen researchers, from a framework developed for healthcare 2 (Box 1).  Ideally, the Key Concepts for Making Informed Choices should be embedded in education for citizens of all ages. This should be done using learning resources and teaching strategies that have been evaluated and shown to be effective. Trustworthy evidence  People are flooded with information. Simply giving them more is unlikely to be helpful unless its value is understood. A recent survey in the UK showed that only about a third of the public trust evidence from medical research; about two-thirds trust the experiences of friends and family 3.  Not all evidence is created equal. Yet people often don’t appreciate which claims are more trustworthy than others; what sort of comparisons are needed to evaluate different proposals fairly; or what other information needs to be considered to inform good choices. For example, many people don’t grasp that things can be associated without one necessarily causing the other. The media sometimes perpetuates this problem by using language suggesting that cause-and-effect has been established when it has not 4, using statements such as “coffee can kill you”, or “drinking one glass of beer a day can make you live longer”. Worse, exaggerated causal claims often pepper university and journal press releases 5. Studies that make fair comparisons are vital, yet people often don’t know how to assess the validity of research. Systematic reviews that synthesise well-designed studies relevant to clearly-defined questions are more trustworthy than haphazard observations; they are less susceptible to biases (systematic distortions) and the play of chance (random errors). Yet results from single studies are often reported in isolation, as facts. Hence the familiar flip-flopping headlines such as “chocolate is good for you”, followed the next week by “chocolate is bad for you”.  DRAFT 25 July 2019 2  To make good choices, other types of information are needed too — for example about costs and feasibility. Judgements must also be made about the relevance of information from research (its applicability or transferability from one situation to another), and about the balance between the likely desirable and undesirable effects of a drug or therapy or regulation.  When it comes to carbon taxes, for example, policymakers need to consider evidence about their environmental and economic effects, judge how applicable that evidence is, weigh how onerous the administrative difficulties are, model how tax burdens will be distributed across socioeconomic groups, and think about whether the taxes will be accepted in their jurisdictions.    Critical thinking Individuals and organisations across many fields are working to enable people to make informed decisions. These efforts include synthesizing the best available evidence in systematic reviews; making that information more accessible, for example through plain language summaries or open access; and teaching people how to use such resources. Examples include the Cochrane Collaboration, the Campbell Collaboration, the Collaboration for Environmental Evidence, the International Society for evidence-Based Health Care, the Center for Evidence-Based Management, the Africa Centre for Evidence, the International Initiative for Impact Evaluation and the What Works Centres in the UK.  Unfortunately, academics tend to work in silos, missing opportunities to learn from others. The expertise of the authors of this article spans 14 different fields: agriculture, economics, education, environmental management, international development, healthcare, informal learning, management, nutrition, planetary health, policing, social welfare, speech and language therapy, and veterinary medicine.  We have identified many key concepts that apply across these fields (Table 1). Some additional concepts are more relevant in some fields than others. For example, it is often important to consider potential placebo effects when assessing claims about medical treatments and nutrition, but these are rarely relevant with respect to interventions in the environment. Our collaboration has already prompted many of us to develop frameworks for specific fields and to suggest improvements to the original Informed Health Choices framework 2. There is power in identifying an issue that resonates across different domains; it provides the momentum to align efforts. The Key Concepts for Informed Choices is not a checklist. It is a starting point. Although we have organised the Key Concepts in three groups (claims, comparisons and choices), it can be used to develop learning resources that include any combination of these, presented in any order. We hope it will prove useful to people helping others to think critically about what evidence to trust and what to do, including those teaching critical thinking and those responsible for communicating research findings.   Next steps Evidence-informed practice is now taught to professionals in many different fields, and these efforts must grow. It is also vital that school children learn the Key Concepts, rather than DRAFT 25 July 2019 3  delaying acquisition of these skills until adulthood. Children who have been explicitly taught critical thinking make better judgements than those who have not6. Early education sets an important foundation for teaching time-pressed adults. An important part of the work of encouraging critical thinking is learning and sharing strategies for promoting healthy scepticism while avoiding unintended adverse consequences. Possible unwanted consequences include inducing nihilism; allowing for disingenuous claims that uncertainty is a defensible argument against action (on climate change, for example); or encouraging false beliefs that competing interests among those promoting interventions renders all research untrustworthy.  Competing interests take different forms in different fields, but the challenges and remedies are similar: recognition of competing interests, transparency, and independent evaluations. Achieving these depends on improved public understanding of the need for evaluation, and public demand for investment in independent evaluations, as well as unbiased communication of evaluation findings. Further development and specialization of the Key Concepts for Informed Choices is needed, and we welcome suggestions. For example, further consideration needs to be given to how these concepts can be applied to system-wide changes, such as mitigation of the effects of climate change or adaptation to environmental change, taking into account complex, dynamic interactions and feedback loops.  To facilitate further development, we have created a website (www.thatsaclaim.org) where the Key Concepts can be adapted to different fields and target users, translated into other languages, and linked to learning resources. References 1. Bouygues, H.L. The State of Critical Thinking: A New Look at Reasoning at Home, School, and Work. (Reboot Foundation, 2018). https://reboot-foundation.org/research_/  2. Oxman AD, Chalmers I & Austvoll-Dahlgren A. Key Concepts for assessing claims about treatment effects and making well-informed treatment choices. F1000Research 7, 1784 (2018). 3. Academy of Medical Sciences. Enhancing the use of scientific evidence to judge the potential benefits and harms of medicines (Academy of Medical Sciences, 2017). https://acmedsci.ac.uk/file-download/44970096 4. Haber, N. et al. Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): a systematic review. PLoS One 13, e0196346 (2018). 5. Sumner, P. et al. Exaggerations and caveats in press releases and health-related science news. PLoS One 11, e0168217 (2016). 6. Abrami, P.C., et al. Strategies for teaching students to think critically: a meta-analysis. Rev Educ Res. 85, 275–314 (2015).   Randomised trial DRAFT 25 July 2019 4  Kids taught health tool in Uganda pass test The Informed Health Choices (IHC) Project was initially developed between 2012 and 2017 by a collaboration including some of the co-authors of this article (Andy Oxman, Astrid Dahlgren, Iain Chalmers, and Matt Oxman). It includes its own set of Key Concepts2, learning resources, and a database of multiple-choice questions to assess how well users can apply the concepts.   In 2016, a randomized trial involving 120 schools and over 10,000 school children in Uganda showed that this resource improved the ability of 10- to 12-year-old children to apply 12 of the Key Concepts 7. These concepts included, for example, recognising that personal experiences alone are an insufficient basis for claims about effects, and that small studies can be misleading. In this trial, 69% of school children who were taught the Key Concepts passed a multiple-choice test of their ability to think critically about health claims, compared to just 27% of the school children not taught the Key Concepts.  7. Nsangi, A. et al. Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects: a cluster-randomised controlled trial. Lancet 390, 374–388 (2017).    DRAFT 25 July 2019 5  Box 2. Key Concepts in Action Claims Key Concept: Beliefs alone about how interventions work are not reliable predictors of the presence or size of effects of interventions. Most people would intuitively say that it is hard to influence parents’ engagement with their children’s education. The common-sense assumption is therefore that more intensive (and more costly) interventions would be more likely to be effective. However, studies of intensive interventions have often failed to show effects on pupils’ attainment, as measured with standard tests 8. Meanwhile, a recent evaluation of the effects of simply texting parents weekly with updates about their child’s schooling had positive effects on children’s attendance, homework submission, and mathematics attainment 9. These effects were small, but the cost was very low. This illustrates that—contrary to intuitive reasoning— inexpensive interventions can be helpful, and expensive ones can fail. Comparisons Key Concept: Comparison groups (or conditions) should be as similar as possible. “Scared Straight” programmes take young offenders on prison visits on the assumption that this experience and listening to inmates’ descriptions of life in prison will deter juvenile delinquency. Before-after comparisons have found that such prison visits were followed by large reductions in delinquent behaviour. But a lot can change within a group of youngsters over time, including becoming older and more mature. How can anyone know that the prison visits caused the reduction? Fairer comparisons of prison visits were done in which youths were randomly assigned either to visit prison or not, thus creating groups of youths who were more comparable. Comparisons between these two groups showed greater subsequent delinquent behaviour in the youngsters who had been exposed to prisons than in those who had not10, 11. The before-after comparisons, lacking similar comparison groups, were misleading. Choices Key Concept: When there are important uncertainties about the effects of interventions, those uncertainties should be reduced by (further) fair comparisons. Performance-based financing schemes—where funds are released only if a specific action is taken or performance target met—have become popular in the health sector. Billions of dollars have been invested in promoting these schemes in low- and middle-income countries, with the aim of achieving international development goals12. For example, health providers have been offered financial incentives to increase the percentage of births in institutions (instead of at home), with the intention of improving maternal and newborn health and survival. However, performance-based financing schemes can have unintended adverse effects, such as encouraging health care workers to falsify records, or to neglect non-incentivized activities. In Tanzania, this scheme prompted some health facilities to threaten new mothers with fines or denial of vaccinations for their children. Where there is so much uncertainty about both the beneficial and adverse effects of an intervention, further fair comparisons should be done before or while rolling out such schemes.   DRAFT 25 July 2019 6  8. Education Endowment Foundation. Parental engagement. https://educationendowmentfoundation.org.uk/evidence-summaries/teaching-learning-toolkit/parental-engagement (2018). 9. Education Endowment Foundation. Texting parents. https://educationendowmentfoundation.org.uk/projects-and-evaluation/projects/texting-parents (2018). 10. Petrosino, A., Turpin-Petrosino, C., Hollis-Peel, M.E. & Lavenberg J.G. Well-meaning programs can have harmful effects! Lessons from experiments of programs such as scared straight. Crime. Delinq. 46, 354-379 (2000). 11. Petrosino, A. et al. Scared Straight and other juvenile awareness programs for preventing juvenile delinquency. Cochrane Database Syst. Rev. CD002796 (2013). 12. Renmans, D., Holvoet, N., Orach, C.G., & Criel, B. Opening the ‘black box’ of performance-based financing in low- and lower middle-income countries: a review of the literature. Health Policy Plan. 31, 1297-1309 (2016).  AUTHORS  Jeffrey Aronson jeffrey.aronson@phc.ox.ac.uk, Astrid Dahlgren astrid@r-bup.no, Eric Barends e.barends@cebma.org, Robert Boruch robertb@upenn.edu, Marnie Brennan marnie.brennan@nottingham.ac.uk, Iain Chalmers ichalmers@jameslindlibrary.org, Joe Chislett joe.chislett@gmail.com, Peter Cunliffe-Jones peter@africacheck.org, Andy Haines andy.haines@lshtm.ac.uk, Carl Heneghan Carl.Heneghan@phc.ox.ac.uk, Marie Gaarder mgaarder@3ieimpact.org, Robert Matthews rajm@physics.org, Brandy Maynard brandy.maynard@slu.edu, Nicola Randall nrandall@harper-adams.ac.uk, Andrew Oxman oxman@online.no, Matt Oxman matt@mattoxman.com, Andrew Pullin a.s.pullin@bangor.ac.uk, Hazel Roddam HRoddam@uclan.ac.uk, Anel Schoonees  anelschoonees@sun.ac.za, Jonathan Sharples Jonathan.Sharples@eefoundation.org.uk, Ruth Stewart ruths@uj.ac.za, Janet Stott janet.stott@oum.ox.ac.uk, Ray Tallis raymondctallis@gmail.com, Nerys Thomas Nerys.Thomas@college.pnn.police.uk, Luke Vale luke.vale@ncl.ac.uk   Correspondence: Andy Oxman oxman@online.no  Centre for Informed Health Choices Norwegian Institute of Public Health  PO Box 4404, Nydalen N-0403 Oslo, Norway Tel: (+47) 48254924   

Key concepts for making informed choices

Aronson, Jeffrey

Barends, E

Boruch, R

Brennan, M

Chalmers, I

Chislett, J

Cunliffe-Jones, P

Dahlgren, A

Gaarder, M

Haines, A

Matthews, R

Maynard, B

Oxman, AD

Oxman, M

Pullin, A

Randall, N

Roddam, H

Schoonees, A

Sharples, J

Stewart, R

Tallis, R

Thomas, N

Vale, L

Oxford University Research Archive

Aronson, Jeffrey K

Oxman, Andrew D

Bangor University Research Portal

English

Aronson, JK

Heneghan, C

Stott, J

UCL Discovery

DRAFT 13 June25 July 2019 1  Key Concepts for making informed Choices An alliance of researchers lays out a framework for taking decisions based on thinking critically about claims and comparisons.   Everyone makes claims about what works. Politicians claim that stop and search will reduce violent crime; friends claim that vaccines cause autism; advertisers claim that natural food is healthy. One group of scientists claims that "deworming" programmes (giving deworming pills to all school children in affected areas) improve school performance and health, calling dewormingxxx.  one of the most potent anti-poverty interventions of our time. Another that deworming does not improve either school performance or healthyyyy. <pls add egs, to lash to Nature reader concerns] Unfortunately, people often fail to think critically about the trustworthiness of claims, including policy makers weighing claims made by scientists <addition OK, to make it more relevant to Nature audience]. Schools do not do enough to prepare young people to think critically1. So many people struggle to assess the trustworthiness of evidence. As a consequence, they may not make informed choices.  To address this deficit, we present here a general tool: Key Concepts for Making Informed Choices (Table 1, with examples in Box 2). We hope scientists and professionals in all fields will use, evolve and evaluate it. The tool was adapted, drawing on the expertise of two dozen researchers, from a framework developed for healthcare 2 (Box 1).  Ideally, the Key Concepts for Making Informed Choices should be embedded in education for citizens of all ages. This should be done using learning resources and teaching strategies that have been evaluated and shown to be effective. Trustworthy evidence  People are flooded with information. Simply giving them more is unlikely to be helpful unless its value is understood. A recent survey in the UK showed that only about a third of the public trust evidence from medical research; about two-thirds trust the experiences of friends and family 3.  Not all evidence is created equal. Yet people often don’t appreciate which claims are more trustworthy than others; what sort of comparisons are needed to evaluate different proposals fairly; or what other information needs to be considered to inform good choices. For example, many people don’t grasp <ok ‘fail’ read as a bit judgemental] that things can be associated with without one necessarily causing the other <edits ok? Changes made because coffee/beer below are not ‘interventions’ nor is fossil fuel burning]. The media sometimes perpetuates this problem by using language suggesting that cause-and-effect has been established when it has not 4, using statements such as “coffee can kill you”, or “drinking one glass of beer a day can make you live longer”. Worse, exaggerated causal claims often pepper university and journal press releases 5. Studies that make fair comparisons are vital, yet people often don’t know how to assess the validity of research. Systematic reviews that synthesise well-designed studies relevant to clearly-defined questions are more trustworthy than haphazard observations; they are less Formatted: Font: Not BoldFormatted: Font: Not BoldFormatted: No underlineCommented [NJ1]: [Style note: Nature may want to somehow emphasize the terms claims, comparisons, and choices in this paragraph, to solidify their importance; perhaps bold or italics… I will leave that up to their copy editing team] Formatted: Font: Not BoldDRAFT 13 June25 July 2019 2  susceptible to biases (systematic distortions) and the play of chance (random errors). Yet results from single studies are often reported in isolation, as facts. Hence the familiar flip-flopping headlines such as “chocolate is good for you”, followed the next week by “chocolate is bad for you”.  To make good choices, other types of information are needed too — for example about costs and feasibility. Judgements must also be made about the relevance of information from research (its applicability or transferability from one situation to another), and about the balance between the likely desirable and undesirable effects of a drug or therapy or regulation.  When it comes to carbon taxes, for example, policymakers need to consider evidence about their environmental and economic effects, judge how applicable that evidence is, weigh how onerous the administrative difficulties are, model how tax burdens will be distributed across socioeconomic groups, and think about whether the taxes will be accepted in their jurisdictions.    Critical thinking Individuals and organisations across a many fields are working to enable people to make informed decisions. These efforts include synthesizing the best available evidence in systematic reviews; making that information more accessible, for example through plain language summaries or open access; and teaching people how to use such resources. Examples include the Cochrane Collaboration, the Campbell Collaboration, the Collaboration for Environmental Evidence, the International Society for evidence-Based Health Care, the Center for Evidence-Based Management, the Africa Centre for Evidence, the International Initiative for Impact Evaluation and the What Works Centres in the UK.  Unfortunately, academics tend to work in silos, missing opportunities to learn from others. The Our expertise of the authors of this article spans 14 different fields: agriculture, economics, education, environmental management, international development, healthcare, informal learning, management, nutrition, planetary health, policing, social welfare, speech and language therapy, and veterinary medicine.  We have identified many key concepts that apply across these fields (Table 1)., and sSome additional conceptsthat were are more relevant in some fields than others (Table 2). For example, it is often important to consider potential placebo effects when assessing claims about medical treatments and nutrition, but these are rarely relevant with respect to interventions in the environment. Our collaboration has already prompted many of us to develop frameworks for specific fields and to suggest improvements to the original Informed Health Choices framework 2. There is power in identifying an issue that resonates across different domains; it provides the momentum to align efforts. The Key Concepts for Informed Choices is not a checklist. It is a starting point. Although we have organised the Key Concepts in three groups (claims, comparisons and choices), it can be used to develop learning resources that include any combination of these, presented in any order. We hope it will prove useful to people helping others to think critically about what DRAFT 13 June25 July 2019 3  evidence to trust and what to do, including those teaching critical thinking and those responsible for communicating research findings.   Next steps Evidence-informed practice is now taught to professionals in many different fields, and these efforts must grow. It is also vital that school children learn the Key Concepts, rather than delaying acquisition of these skills until adulthood. Children who have been explicitly taught critical thinking make better judgements than those who have not6. Early education sets an important foundation for teaching time-pressed adults. An important part of the work of encouraging critical thinking is learning and sharing strategies for promoting healthy scepticism while avoiding unintended adverse consequences. Possible unwanted consequences include inducing nihilism; allowing for disingenuous claims that uncertainty is a defensible argument against action (on climate change, for example); or encouraging false beliefs that competing interests among those promoting interventions renders all research untrustworthy.  Competing interests take different forms in different fields, but the challenges and remedies are similar: recognition of competing interests, transparency, and independent evaluations. Achieving these depends on improved public understanding of the need for evaluation, and public demand for investment in independent evaluations, as well as unbiased communication of evaluation findings. Further development and specialization of the Key Concepts for Informed Choices is needed, and we welcome suggestions. For example, further consideration needs to be given to how these concepts can be applied to system-wide changes, such as mitigation of the effects of climate change or adaptation to environmental change, taking into account complex, dynamic interactions and feedback loops.  To facilitate further development, we have created a website (www.thatsaclaim.org) where the Key Concepts can be adapted to different fields and target users, translated into other languages, and linked to learning resources. References 1. Bouygues, H.L. The State of Critical Thinking: A New Look at Reasoning at Home, School, and Work. (Reboot Foundation, 2018). https://reboot-foundation.org/research_/  2. Oxman AD, Chalmers I & Austvoll-Dahlgren A. Key Concepts for assessing claims about treatment effects and making well-informed treatment choices. F1000Research 7, 1784 (2018). 3. Academy of Medical Sciences. Enhancing the use of scientific evidence to judge the potential benefits and harms of medicines (Academy of Medical Sciences, 2017). https://acmedsci.ac.uk/file-download/44970096 4. Haber, N. et al. Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): a systematic review. PLoS One 13, e0196346 (2018). 5. Sumner, P. et al. Exaggerations and caveats in press releases and health-related science news. PLoS One 11, e0168217 (2016). DRAFT 13 June25 July 2019 4  6. Abrami, P.C., et al. Strategies for teaching students to think critically: a meta-analysis. Rev Educ Res. 85, 275–314 (2015).   Randomised trial Kids taught health tool in Uganda pass test The Informed Health Choices (IHC) Project was initially developed between 2012 and 2017 by a collaboration including some of the co-authors of this article (Andy Oxman, Astrid Dahlgren, Iain Chalmers, and Matt Oxman). It includes its own set of Key Concepts2, learning resources, and a database of multiple-choice questions to assess how well users can apply the concepts.   In 2016, a randomized trial involving 120 schools and over 10,000 school children in Uganda showed that this resource improved the ability of 10- to 12-year-old children to apply 12 of the Key Concepts 7. These concepts included, for example, recognising that personal experiences alone are an insufficient basis for claims about effects, and that small studies can be misleading. In this trial, 69% of school children who were taught the Key Concepts passed a multiple-choice test of their ability to think critically about health claims, compared to just 27% of the school children not taught the Key Concepts.  7. Nsangi, A. et al. Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects: a cluster-randomised controlled trial. Lancet 390, 374–388 (2017).    DRAFT 13 June25 July 2019 5  Box 2. Key Concepts in Action Claims Key Concept: Beliefs alone about how interventions work are not reliable predictors of the presence or size of effects of interventions. Most people would intuitively say that it is hard to influence parents’ engagement with their children’s education. The common-sense assumption is therefore that more intensive (and more costly) interventions would be more likely to be effective. However, studies of intensive interventions have often failed to show effects on pupils’ attainment, as measured with standard tests 8. Meanwhile, a recent evaluation of the effects of simply texting parents weekly with updates about their child’s schooling had positive effects on children’s attendance, homework submission, and mathematics attainment 9. These effects were small, but the cost was very low. This illustrates that—contrary to intuitive reasoning— inexpensive interventions can be helpful, and expensive ones can fail. Comparisons Key Concept: Comparison groups (or conditions) should be as similar as possible. “Scared Straight” programmes take young offenders on prison visits on the assumption that this experience, and listening to inmates’ descriptions of life in prison, will deter juvenile delinquency. Before-after comparisons have found that such prison visits were followed by large reductions in delinquent behaviour. But a lot can change within a group of youngsters over time, including becoming older and more mature. How can anyone know that the prison visits caused the reduction? Fairer comparisons of prison visits were done in which youths were randomly assigned either to visit prison or not, thus creating groups of youths who were more comparable. Comparisons between these two groups showed greater subsequent delinquent behaviour in the youngsters who had been exposed to prisons than in those who had not10, 11. The before-after comparisons, lacking similar comparison groups, were misleading. Choices Key Concept: When there are important uncertainties about the effects of interventions, those uncertainties should be reduced by (further) fair comparisons. Performance-based financing schemes—where funds are released only if a specific action is taken or performance target met—have become popular in the health sector. Billions of dollars have been invested in promoting these schemes in low- and middle-income countries, with the aim of achieving international development goals12. For example, health providers have been offered financial incentives to increase the percentage of births in institutions (instead ofas opposed to at home), with the intention of improving maternal and newborn health and survival. However, performance-based financing schemes can have unintended adverse effects, such as encouraging health care workers to falsify records, or to neglect non-incentivized activities. In Tanzania, this scheme prompted some health facilities to threaten new mothers with fines or denial of vaccinations for their children. Where there is so much uncertainty about both the beneficial and adverse effects of an intervention, further fair comparisons should be done before or while rolling out such schemes.  DRAFT 13 June25 July 2019 6   8. Education Endowment Foundation. Parental engagement. https://educationendowmentfoundation.org.uk/evidence-summaries/teaching-learning-toolkit/parental-engagement (2018). 9. Education Endowment Foundation. Texting parents. https://educationendowmentfoundation.org.uk/projects-and-evaluation/projects/texting-parents (2018). 10. Petrosino, A., Turpin-Petrosino, C., Hollis-Peel, M.E. & Lavenberg J.G. Well-meaning programs can have harmful effects! Lessons from experiments of programs such as scared straight. Crime. Delinq. 46, 354-379 (2000). 11. Petrosino, A. et al. Scared Straight and other juvenile awareness programs for preventing juvenile delinquency. Cochrane Database Syst. Rev. CD002796 (2013). 12. Renmans, D., Holvoet, N., Orach, C.G., & Criel, B. Opening the ‘black box’ of performance-based financing in low- and lower middle-income countries: a review of the literature. Health Policy Plan. 31, 1297-1309 (2016).  AUTHORS  Jeffrey Aronson jeffrey.aronson@phc.ox.ac.uk, Astrid Dahlgren astrid@r-bup.no, Eric Barends e.barends@cebma.org, Robert Boruch robertb@upenn.edu, Marnie Brennan marnie.brennan@nottingham.ac.uk, Iain Chalmers ichalmers@jameslindlibrary.org, Joe Chislett joe.chislett@gmail.com, Peter Cunliffe-Jones peter@africacheck.org, Andy Haines andy.haines@lshtm.ac.uk, Carl Heneghan Carl.Heneghan@phc.ox.ac.uk, Marie Gaarder mgaarder@3ieimpact.org, Robert Matthews rajm@physics.org, Brandy Maynard brandy.maynard@slu.edu, Nicola Randall nrandall@harper-adams.ac.uk, Andrew Oxman oxman@online.no, Matt Oxman matt@mattoxman.com, Andrew Pullin a.s.pullin@bangor.ac.uk, Hazel Roddam HRoddam@uclan.ac.uk, Anel Schoonees  anelschoonees@sun.ac.za, Jonathan Sharples Jonathan.Sharples@eefoundation.org.uk, Ruth Stewart ruths@uj.ac.za, Janet Stott janet.stott@oum.ox.ac.uk, Ray Tallis raymondctallis@gmail.com, Nerys Thomas Nerys.Thomas@college.pnn.police.uk, Luke Vale luke.vale@ncl.ac.uk   Correspondence: Andy Oxman oxman@online.no  Centre for Informed Health Choices Norwegian Institute of Public Health  PO Box 4404, Nydalen N-0403 Oslo, Norway Tel: (+47) 48254924   

Oxman, Andrew

Tallis, Ray

CLoK

ArticleKey Concepts for making informed ChoicesAronson, Jeffrey, Dahlgren, Astrid, Barends, Eric, Boruch, Robert, Brennan, Marnie, Chalmers, Iain, Chislett, Joe, Cunliffe-Jones, Peter, Haines, Andy, Heneghan, Carl, Gaarder, Marie, Matthews, Robert, Maynard, Brandy, Randall, Nicola, Roddam, Hazel, Oxman, Andrew, Oxman, Matt, Pullin, Andrew, Schoonees, Anel, Sharples, Jonathan, Stewart, Ruth, Stott, Janet, Tallis, Ray, Thomas, Nerys and Vale, LukeAvailable at http://clok.uclan.ac.uk/29356/Aronson, Jeffrey, Dahlgren, Astrid, Barends, Eric, Boruch, Robert, Brennan, Marnie, Chalmers, Iain, Chislett, Joe, Cunliffe­Jones, Peter, Haines, Andy et al (2019) Key Concepts for making informed Choices. Nature, 572 . pp. 303­306. ISSN 0028­0836  It is advisable to refer to the publisher’s version if you intend to cite from the work.http://dx.doi.org/10.1038/d41586-019-02407-9For more information about UCLan’s research in this area go to http://www.uclan.ac.uk/researchgroups/ and search for <name of research Group>.For information about Research generally at UCLan please go to http://www.uclan.ac.uk/research/ All outputs in CLoK are protected by Intellectual Property Rights law, includingCopyright law.  Copyright, IPR and Moral Rights for the works on this site are retained by the individual authors and/or other copyright owners. Terms and conditions for use of this material are defined in the http://clok.uclan.ac.uk/policies/CLoKCentral Lancashire online Knowledgewww.clok.uclan.ac.ukDRAFT 13 June25 July 2019 1  Key Concepts for making informed Choices An alliance of researchers lays out a framework for taking decisions based on thinking critically about claims and comparisons.   Everyone makes claims about what works. Politicians claim that stop and search will reduce violent crime; friends claim that vaccines cause autism; advertisers claim that natural food is healthy. One group of scientists claims that "deworming" programmes (giving deworming pills to all school children in affected areas) improve school performance and health, calling dewormingxxx.  one of the most potent anti-poverty interventions of our time. Another that deworming does not improve either school performance or healthyyyy. <pls add egs, to lash to Nature reader concerns] Unfortunately, people often fail to think critically about the trustworthiness of claims, including policy makers weighing claims made by scientists <addition OK, to make it more relevant to Nature audience]. Schools do not do enough to prepare young people to think critically1. So many people struggle to assess the trustworthiness of evidence. As a consequence, they may not make informed choices.  To address this deficit, we present here a general tool: Key Concepts for Making Informed Choices (Table 1, with examples in Box 2). We hope scientists and professionals in all fields will use, evolve and evaluate it. The tool was adapted, drawing on the expertise of two dozen researchers, from a framework developed for healthcare 2 (Box 1).  Ideally, the Key Concepts for Making Informed Choices should be embedded in education for citizens of all ages. This should be done using learning resources and teaching strategies that have been evaluated and shown to be effective. Trustworthy evidence  People are flooded with information. Simply giving them more is unlikely to be helpful unless its value is understood. A recent survey in the UK showed that only about a third of the public trust evidence from medical research; about two-thirds trust the experiences of friends and family 3.  Not all evidence is created equal. Yet people often don’t appreciate which claims are more trustworthy than others; what sort of comparisons are needed to evaluate different proposals fairly; or what other information needs to be considered to inform good choices. For example, many people don’t grasp <ok ‘fail’ read as a bit judgemental] that things can be associated with without one necessarily causing the other <edits ok? Changes made because coffee/beer below are not ‘interventions’ nor is fossil fuel burning]. The media sometimes perpetuates this problem by using language suggesting that cause-and-effect has been established when it has not 4, using statements such as “coffee can kill you”, or “drinking one glass of beer a day can make you live longer”. Worse, exaggerated causal claims often pepper university and journal press releases 5. Studies that make fair comparisons are vital, yet people often don’t know how to assess the validity of research. Systematic reviews that synthesise well-designed studies relevant to clearly-defined questions are more trustworthy than haphazard observations; they are less Formatted: Font: Not BoldFormatted: Font: Not BoldFormatted: No underlineCommented [NJ1]: [Style note: Nature may want to somehow emphasize the terms claims, comparisons, and choices in this paragraph, to solidify their importance; perhaps bold or italics… I will leave that up to their copy editing team] Formatted: Font: Not BoldDRAFT 13 June25 July 2019 2  susceptible to biases (systematic distortions) and the play of chance (random errors). Yet results from single studies are often reported in isolation, as facts. Hence the familiar flip-flopping headlines such as “chocolate is good for you”, followed the next week by “chocolate is bad for you”.  To make good choices, other types of information are needed too — for example about costs and feasibility. Judgements must also be made about the relevance of information from research (its applicability or transferability from one situation to another), and about the balance between the likely desirable and undesirable effects of a drug or therapy or regulation.  When it comes to carbon taxes, for example, policymakers need to consider evidence about their environmental and economic effects, judge how applicable that evidence is, weigh how onerous the administrative difficulties are, model how tax burdens will be distributed across socioeconomic groups, and think about whether the taxes will be accepted in their jurisdictions.    Critical thinking Individuals and organisations across a many fields are working to enable people to make informed decisions. These efforts include synthesizing the best available evidence in systematic reviews; making that information more accessible, for example through plain language summaries or open access; and teaching people how to use such resources. Examples include the Cochrane Collaboration, the Campbell Collaboration, the Collaboration for Environmental Evidence, the International Society for evidence-Based Health Care, the Center for Evidence-Based Management, the Africa Centre for Evidence, the International Initiative for Impact Evaluation and the What Works Centres in the UK.  Unfortunately, academics tend to work in silos, missing opportunities to learn from others. The Our expertise of the authors of this article spans 14 different fields: agriculture, economics, education, environmental management, international development, healthcare, informal learning, management, nutrition, planetary health, policing, social welfare, speech and language therapy, and veterinary medicine.  We have identified many key concepts that apply across these fields (Table 1)., and sSome additional conceptsthat were are more relevant in some fields than others (Table 2). For example, it is often important to consider potential placebo effects when assessing claims about medical treatments and nutrition, but these are rarely relevant with respect to interventions in the environment. Our collaboration has already prompted many of us to develop frameworks for specific fields and to suggest improvements to the original Informed Health Choices framework 2. There is power in identifying an issue that resonates across different domains; it provides the momentum to align efforts. The Key Concepts for Informed Choices is not a checklist. It is a starting point. Although we have organised the Key Concepts in three groups (claims, comparisons and choices), it can be used to develop learning resources that include any combination of these, presented in any order. We hope it will prove useful to people helping others to think critically about what DRAFT 13 June25 July 2019 3  evidence to trust and what to do, including those teaching critical thinking and those responsible for communicating research findings.   Next steps Evidence-informed practice is now taught to professionals in many different fields, and these efforts must grow. It is also vital that school children learn the Key Concepts, rather than delaying acquisition of these skills until adulthood. Children who have been explicitly taught critical thinking make better judgements than those who have not6. Early education sets an important foundation for teaching time-pressed adults. An important part of the work of encouraging critical thinking is learning and sharing strategies for promoting healthy scepticism while avoiding unintended adverse consequences. Possible unwanted consequences include inducing nihilism; allowing for disingenuous claims that uncertainty is a defensible argument against action (on climate change, for example); or encouraging false beliefs that competing interests among those promoting interventions renders all research untrustworthy.  Competing interests take different forms in different fields, but the challenges and remedies are similar: recognition of competing interests, transparency, and independent evaluations. Achieving these depends on improved public understanding of the need for evaluation, and public demand for investment in independent evaluations, as well as unbiased communication of evaluation findings. Further development and specialization of the Key Concepts for Informed Choices is needed, and we welcome suggestions. For example, further consideration needs to be given to how these concepts can be applied to system-wide changes, such as mitigation of the effects of climate change or adaptation to environmental change, taking into account complex, dynamic interactions and feedback loops.  To facilitate further development, we have created a website (www.thatsaclaim.org) where the Key Concepts can be adapted to different fields and target users, translated into other languages, and linked to learning resources. References 1. Bouygues, H.L. The State of Critical Thinking: A New Look at Reasoning at Home, School, and Work. (Reboot Foundation, 2018). https://reboot-foundation.org/research_/  2. Oxman AD, Chalmers I & Austvoll-Dahlgren A. Key Concepts for assessing claims about treatment effects and making well-informed treatment choices. F1000Research 7, 1784 (2018). 3. Academy of Medical Sciences. Enhancing the use of scientific evidence to judge the potential benefits and harms of medicines (Academy of Medical Sciences, 2017). https://acmedsci.ac.uk/file-download/44970096 4. Haber, N. et al. Causal language and strength of inference in academic and media articles shared in social media (CLAIMS): a systematic review. PLoS One 13, e0196346 (2018). 5. Sumner, P. et al. Exaggerations and caveats in press releases and health-related science news. PLoS One 11, e0168217 (2016). DRAFT 13 June25 July 2019 4  6. Abrami, P.C., et al. Strategies for teaching students to think critically: a meta-analysis. Rev Educ Res. 85, 275–314 (2015).   Randomised trial Kids taught health tool in Uganda pass test The Informed Health Choices (IHC) Project was initially developed between 2012 and 2017 by a collaboration including some of the co-authors of this article (Andy Oxman, Astrid Dahlgren, Iain Chalmers, and Matt Oxman). It includes its own set of Key Concepts2, learning resources, and a database of multiple-choice questions to assess how well users can apply the concepts.   In 2016, a randomized trial involving 120 schools and over 10,000 school children in Uganda showed that this resource improved the ability of 10- to 12-year-old children to apply 12 of the Key Concepts 7. These concepts included, for example, recognising that personal experiences alone are an insufficient basis for claims about effects, and that small studies can be misleading. In this trial, 69% of school children who were taught the Key Concepts passed a multiple-choice test of their ability to think critically about health claims, compared to just 27% of the school children not taught the Key Concepts.  7. Nsangi, A. et al. Effects of the Informed Health Choices primary school intervention on the ability of children in Uganda to assess the reliability of claims about treatment effects: a cluster-randomised controlled trial. Lancet 390, 374–388 (2017).    DRAFT 13 June25 July 2019 5  Box 2. Key Concepts in Action Claims Key Concept: Beliefs alone about how interventions work are not reliable predictors of the presence or size of effects of interventions. Most people would intuitively say that it is hard to influence parents’ engagement with their children’s education. The common-sense assumption is therefore that more intensive (and more costly) interventions would be more likely to be effective. However, studies of intensive interventions have often failed to show effects on pupils’ attainment, as measured with standard tests 8. Meanwhile, a recent evaluation of the effects of simply texting parents weekly with updates about their child’s schooling had positive effects on children’s attendance, homework submission, and mathematics attainment 9. These effects were small, but the cost was very low. This illustrates that—contrary to intuitive reasoning— inexpensive interventions can be helpful, and expensive ones can fail. Comparisons Key Concept: Comparison groups (or conditions) should be as similar as possible. “Scared Straight” programmes take young offenders on prison visits on the assumption that this experience, and listening to inmates’ descriptions of life in prison, will deter juvenile delinquency. Before-after comparisons have found that such prison visits were followed by large reductions in delinquent behaviour. But a lot can change within a group of youngsters over time, including becoming older and more mature. How can anyone know that the prison visits caused the reduction? Fairer comparisons of prison visits were done in which youths were randomly assigned either to visit prison or not, thus creating groups of youths who were more comparable. Comparisons between these two groups showed greater subsequent delinquent behaviour in the youngsters who had been exposed to prisons than in those who had not10, 11. The before-after comparisons, lacking similar comparison groups, were misleading. Choices Key Concept: When there are important uncertainties about the effects of interventions, those uncertainties should be reduced by (further) fair comparisons. Performance-based financing schemes—where funds are released only if a specific action is taken or performance target met—have become popular in the health sector. Billions of dollars have been invested in promoting these schemes in low- and middle-income countries, with the aim of achieving international development goals12. For example, health providers have been offered financial incentives to increase the percentage of births in institutions (instead ofas opposed to at home), with the intention of improving maternal and newborn health and survival. However, performance-based financing schemes can have unintended adverse effects, such as encouraging health care workers to falsify records, or to neglect non-incentivized activities. In Tanzania, this scheme prompted some health facilities to threaten new mothers with fines or denial of vaccinations for their children. Where there is so much uncertainty about both the beneficial and adverse effects of an intervention, further fair comparisons should be done before or while rolling out such schemes.  DRAFT 13 June25 July 2019 6   8. Education Endowment Foundation. Parental engagement. https://educationendowmentfoundation.org.uk/evidence-summaries/teaching-learning-toolkit/parental-engagement (2018). 9. Education Endowment Foundation. Texting parents. https://educationendowmentfoundation.org.uk/projects-and-evaluation/projects/texting-parents (2018). 10. Petrosino, A., Turpin-Petrosino, C., Hollis-Peel, M.E. & Lavenberg J.G. Well-meaning programs can have harmful effects! Lessons from experiments of programs such as scared straight. Crime. Delinq. 46, 354-379 (2000). 11. Petrosino, A. et al. Scared Straight and other juvenile awareness programs for preventing juvenile delinquency. Cochrane Database Syst. Rev. CD002796 (2013). 12. Renmans, D., Holvoet, N., Orach, C.G., & Criel, B. Opening the ‘black box’ of performance-based financing in low- and lower middle-income countries: a review of the literature. Health Policy Plan. 31, 1297-1309 (2016).  AUTHORS  Jeffrey Aronson jeffrey.aronson@phc.ox.ac.uk, Astrid Dahlgren astrid@r-bup.no, Eric Barends e.barends@cebma.org, Robert Boruch robertb@upenn.edu, Marnie Brennan marnie.brennan@nottingham.ac.uk, Iain Chalmers ichalmers@jameslindlibrary.org, Joe Chislett joe.chislett@gmail.com, Peter Cunliffe-Jones peter@africacheck.org, Andy Haines andy.haines@lshtm.ac.uk, Carl Heneghan Carl.Heneghan@phc.ox.ac.uk, Marie Gaarder mgaarder@3ieimpact.org, Robert Matthews rajm@physics.org, Brandy Maynard brandy.maynard@slu.edu, Nicola Randall nrandall@harper-adams.ac.uk, Andrew Oxman oxman@online.no, Matt Oxman matt@mattoxman.com, Andrew Pullin a.s.pullin@bangor.ac.uk, Hazel Roddam HRoddam@uclan.ac.uk, Anel Schoonees  anelschoonees@sun.ac.za, Jonathan Sharples Jonathan.Sharples@eefoundation.org.uk, Ruth Stewart ruths@uj.ac.za, Janet Stott janet.stott@oum.ox.ac.uk, Ray Tallis raymondctallis@gmail.com, Nerys Thomas Nerys.Thomas@college.pnn.police.uk, Luke Vale luke.vale@ncl.ac.uk   Correspondence: Andy Oxman oxman@online.no  Centre for Informed Health Choices Norwegian Institute of Public Health  PO Box 4404, Nydalen N-0403 Oslo, Norway Tel: (+47) 48254924   

Key Concepts for making informed Choices

Teach people to think critically about claims and comparisons — they will make better decisions

Oxford University Research Archive (ORA)

Crossref

Everyone makes claims about what works. Politicians claim that stop-and-search policing will reduce violent crime; friends might assert that vaccines cause autism; advertisers declare that natural food is healthy. A group of scientists describes giving all schoolchildren deworming pills in some areas as one of the most potent anti-poverty interventions of our time. Another group counters that it does not improve children’s health or performance at school.

Unfortunately, people often fail to think critically about the trustworthiness of claims, including policymakers who weigh up those made by scientists. Schools do not do enough to prepare young people to think critically. So many people struggle to assess evidence. As a consequence, they might make poor choices.



To address this deficit, we present here a set of principles for assessing the trustworthiness of claims about what works, and for making informed choices (see ‘Key Concepts for Informed Choices’). We hope that scientists and professionals in all fields will evaluate, use and comment on it. The resources were adapted, drawing on the expertise of two dozen researchers, from a framework developed for health care2 (see ‘Randomized trial’).

Ideally, these concepts should be embedded in education for citizens of all ages. This should be done using learning resources and teaching strategies that have been evaluated and shown to be effective

Aronson, J.K.

Barends, E.

Boruch, R.

Brennan, M.

Chalmers, I.

Chislett, J.

Cunliffe-Jones, P.

Dahlgren, A.

Gaarder, M.

Haines, A.

Heneghan, C.

Matthews, R.

Maynard, B.

Oxman, A.D.

Oxman, M.

Pullin, A.

Randall, N.P.

Roddam, H.

Schoonees, A.

Sharples, J.

Stewart, R.

Stott, J.

Tallis, R.

Thomas, N.

Vale, L.

Harper Adams University Repository

https://research.bangor.ac.uk/portal/files/25149702/Key_Concepts_pre_publication_version.pdf

Key concepts for making informed choices

Abstract

Similar works

Full text

Available Versions