The ESRI’s Behavioural Research Unit undertook two controlled experiments to test whether survey estimates of public compliance with COVID-19 vary depending on how the survey questions are asked

Barjaková, Martina

Belton, Cameron

Lunn, Pete

McGinnity, Frances

Timmons, Shane

English

Archive of European Integration

HOW MUCH DO SURVEY ESTIMATES OF COMPLIANCE WITH COVID-19 ADVICE DEPEND ON HOW THE QUESTION IS ASKED? SHANE TIMMONS, FRAN MCGINNITY, CAMERON BELTON, MARTINA BARJAKOVÁ AND PETE LUNNESRI RESEARCH BULLETIN NOVEMBER 2020EVIDENCE FOR POLICYESRI Research Bulletin 202027 How much do survey estimates of compliance with COVID-19 advice depend on how the question is asked?1 Shane Timmons* (ESRI), Fran McGinnity (ESRI), Cameron Belton (ESRI), Martina Barjaková (ESRI) & Pete Lunn (ESRI) ESRI Research Bulletins provide short summaries of work published by ESRI researchers and overviews of thematic areas covered by ESRI programmes of research. Bulletins are designed to be easily accessible to a wide readership. INTRODUCTION Compliance with COVID-19 public health guidance, such as hand-washing and distancing from others, is difficult to measure objectively. Governments and public health officials rely on estimates from tracking surveys. In research funded by the Department of Health to support the Behavioural Change Subgroup of the National Public Health Emergency Team (NPHET), the ESRI’s Behavioural Research Unit undertook two controlled experiments to test whether estimates vary depending on how survey questions are asked.  The experiments tested two potential sources of bias in responses. The first was “social desirability”, which is the desire of respondents to paint themselves in a favourable light. The second was question framing. Logically, if 90% of participants report hand-washing when behaviours are framed positively (e.g. “I always wash my hands”), 10% should do so when behaviours are framed negatively (e.g. “I don’t always wash my hands”).  METHOD A representative sample of 1,800 people was recruited from an online panel held by a leading market research company. The impact of social desirability was tested using a method that increased anonymity for some participants and compared their answers to a group with the level of anonymity typically offered by a survey. The method worked as follows. Participants were randomly assigned to one of two conditions. In the anonymous “list” condition, participants were shown a list of four behaviours and were asked how many applied to them. They did not have to say which behaviours they had undertaken, only how many of the four. Contained in this list was the target behaviour relevant for compliance with health advice (e.g. 1 This Bulletin summarises the findings from: Timmons, S., McGinnity, F., Belton, C. A., Barjaková, M., & Lunn, P. D. (2020). It depends on how you ask: measuring bias in population surveys of compliance with COVID-19 public health guidance, Journal of Epidemiology and Community Health, online first, DOI: http://dx.doi.org/10.1136/jech-2020-215256 *Corresponding author: shane.timmons@esri.ie2 hand-washing, staying 2m from others). In the “control” condition, participants were shown a list containing only the three non-target behaviours2 and were also asked how many applied to them. Hence, by taking the difference between the average response in the list and control conditions, the proportion endorsing the target behaviour could be calculated, without individuals needing to confess explicitly to non-compliance with the target behaviour. Participants in the control condition were subsequently asked directly about the target behaviour, giving a standard survey estimate of compliance for comparison.  To test the effects of question framing, for some participants the behaviours were framed positively (e.g. “I wash my hands with soap and water for a full 20 seconds…”), while for others the behaviours were framed negatively (e.g. “I don’t wash my hands with soap and water for a full 20 seconds…”).  RESULTS For hand-washing, reported compliance for the standard, positively-framed question was 91%, which was in line with national tracking surveys. But in the list condition this dropped to 81%. The figure also dropped to 83% when the question was framed negatively, whether asked directly or via a list.  For maintaining social distance, reported compliance was 92% for the standard, positively framed question, which again matched national tracking surveys. It fell to 88% in the list condition – not a statistically significant difference. However, when the question was framed negatively, the figure was 79% for the direct question and 71% via a list.   While effects were mostly similar across population subgroups, some differences were significant. The gap in reported hand-washing between the list and control conditions was larger for men than women, suggesting that anonymity led more men to report not washing their hands. Similarly, reported social distancing fell further in the list condition among younger participants and those in rural areas.  CONCLUSION AND IMPLICATIONS It is important to note that reported compliance with public health advice was high in all conditions: a minimum of 71% of people reported sticking to guidelines. However, the study provides evidence that standard tracking surveys can overestimate compliance. Increasing anonymity by using this list technique to reduce social desirability bias decreased reported compliance by up to 10%-points. In addition, altering the frame of the question led estimates of compliance to vary by up to 17%-points. Thus, single survey estimates of compliance with health advice should be considered approximations. Controlled experimental tests can measure a range of likely compliance estimates, helping to identify where there is a need for targeted communications.  2 The non-target items were: I have been keeping in touch with friends and family via the internet or by phone;  I am watching less TV (or streaming TV shows less) than usual; I have a household pet that I have been spending more time with (e.g. taking the dog for a walk more often). Whitaker Square, Sir John Rogerson’s Quay, Dublin 2Telephone  +353 1 863 2000Email admin@esri.ieWeb www.esri.ieTwitter @ESRIDublin

How much do survey estimates of compliance with COVID-19 advice depend on how the question is asked? ESRI Research Bulletin 202027 November 2020.

http://aei.pitt.edu/103530/1/RB202027_0.pdf

How much do survey estimates of compliance with COVID-19 advice depend on how the question is asked? ESRI Research Bulletin 202027 November 2020.

Abstract

Similar works

Full text

Available Versions

Archive of European Integration