By analysing 121 opinion polls, Harold D. Clarke, Matthew Goodwin, and Paul Whiteley outline what happened with the EU referendum survey results. They explain why internet surveys performed substantially better than telephone ones – contrary to the post-2015 General Election ‘wisdom’ that telephone surveys should be preferred. Underlying trends showed that once methodological artefacts are controlled, Leave was almost certainly ahead of Remain over the entire last month of the campaign – and possibly throughout 2016

Clarke, Harold D.

Goodwin, Matthew

Whiteley, Paul

LSE Research Online

12/07/2016Leave was always in the lead: why the polls got the referendumresult wrongblogs.lse.ac.uk/europpblog/2016/07/12/leave-always-lead/By analysing 121 opinion polls, Harold D. Clarke, Matthew Goodwin, and PaulWhiteley outline what happened with the EU referendum survey results. They explain why internetsurveys performed substantially better than telephone ones – contrary to the post-2015 GeneralElection ‘wisdom’ that telephone surveys should be preferred. Underlying trends showed that oncemethodological artefacts are controlled, Leave was almost certainly ahead of Remain over theentire last month of the campaign – and possibly throughout 2016.After the 2015 general election, pollsters were roundly criticised for their failure to predict theConservative victory. Did they fail again in the EU referendum? If so, just how badly did they do?Did they all get it very wrong, or did some do much better than others? And, taking artefactsintroduced by polling methodology into account, what did underlying trends in Remain and Leavesupport look like in the run-up to voting day? Going forward, what lessons can we learn?We address these questions by using a technique called ‘dynamic factor analyses’ on 121 internetand telephone surveys conducted between January 11 and June 22, 2016.[i] We use a statisticalmodel originally developed in research on the performance of polls conducted in an Australiannational election campaign.[ii] A large majority of the survey data employed here was gathered bynine survey organisations (‘houses’) and we also include several additional surveys in a residual‘miscellaneous’ category for other organisations reporting one to three polls.ModesWe begin by investigating what survey researchers call ‘mode’ effects. These effects concern theimpact of differences in the way surveys are conducted on the results they produce. In the case of the 121 EUreferendum surveys of interest here, all were administered either by internet (N = 81) or telephone (N = 40).Analysing the results (percentages of Remain and Leave supporters in various surveys) using our model revealsthat both types of polls significantly overestimated the Remain vote share and underestimated the Leave share (seeFigure 1). However, there were sizable differences between the two types of polls, with the mode effect for equaling5.1 per cent for internet polls as compared to fully 9.2 per cent for telephone polls.[iii] Taken as a group, internetpolls tended to perform considerably better than their telephone-based rivals.1/5HousesWhat about the performance of individual survey companies? Our analysis of house effects indicates that surveyfirms varied markedly in their ability to gauge EU vote intentions. Figure 2 documents that, for the 9 surveyorganisations considered, TNS-BMRB, ICM and YouGov performed best — their overestimates of the Remain voteshare were +2.24 per cent, +2.71 per cent and +3.24 per cent, respectively. Similarly, these three houses did best onestimating Leave support. TNS-BMRB underestimated the Leave share by -2.35 per cent, with ICM missing by -2.81 per cent and YouGov, by -3.34 per cent. All of the surveys conducted by TNS-BMRB (5 surveys) and YouGov(28 surveys) were done by internet, as were 23 of 26 ICM surveys.2/5Note: solid colored bars are for survey houses doing all or most of their polling by internet;striped bars are for survey houses doing all or most of their polling by telephone.  Populushas 3 internet polls and 2 telephone polls.The three firms performing worst were Populus, IPSOS and ComRes, with Remain house effects equaling +6.02,+.6.34 and +6.72 per cent, respectively. Populus missed the Leave percentage by -6.10 points, whereas IPSOSmissed by -6.45 per cent and ComRes by -6.85 per cent. All 7 of IPSOS polls were done by telephone, as were 9 ofthe 11 ComRes polls, and 2 of the 5 Populus polls. Survation was another house that performed poorly (a +5.54 percent overestimate for Remain and -5.67 per cent underestimate for Leave).  All but one of Survation’s 8 polls wereconducted by telephone. ORB, with a Remain house effect of +4.76 per cent and a Leave house effect of -4.87 percent, did 9 of its 13 polls by telephone.Underlying TrendsBy controlling for mode and house effects, our analyses enable us to estimate underlying trends in the dynamics ofsupport in EU referendum vote intentions. The results (Figure 3) indicate that Leave led Remain over the entireperiod from 11 January 2016 onward. The size of the Leave lead varies widely—from a low of .39 per cent (4February) to a high of 13.2 per cent (12 June)—but Leave is always ahead.The analysis also provides insight regarding the effects of various events that occurred during the campaign. Forexample, contrary to his intentions, US President Barack Obama’s widely publicized ‘UK to the back of the queue’intervention may have boosted, rather than diminished, Leave support. In contrast, as numerous observers havespeculated, the murder of Labour MP Jo Cox on 16 June may have precipitated an ongoing erosion in Leavesupport over the final week before the vote. That downturn in Leave’s vote intention share notwithstanding, theanalysis documents that over the last month of the campaign the lower bound of the 95 per cent credible(confidence) interval for the Leave vote always exceeds the 50 per cent mark. This means that Leave was verylikely ahead throughout this entire period.3/5What lessons can we learn?Overall, the analyses suggest two general lessons for consumers of political polls. First, house effects associatedwith selected internet survey firms were quite modest, whereas firms relying heavily or exclusively on telephonesurveys found themselves considerably further off the mark. After the 2015 general election, internet surveys werewidely criticized and some commentators claimed that telephone surveys were a preferred alternative. At the timethis assertion seemed dubious given that telephone surveys’ typically dismal response rates clearly gainsays anyclaim to be true ‘probability’ surveys. As documented above, the relatively poor performance of telephone surveys inthe EU referendum reinforces doubts about their superiority.A second lesson is that the ‘topline’ results from surveys are not enough. If observers had used the polling data asinputs to statistical models of the underlying dynamics of EU referendum vote intentions rather than risking beingmisled by the vagaries introduced by mode and house effects in successive individual polls, they might not havebeen surprised by the Brexit result. High quality polls by reputable survey houses should be viewed as a keyresource—not a substitute—for informed political analysis.Please read our comments policy before commenting .Note: This article was originally published on British Politics and Policy at LSE and it gives the views of the author,and not the position of EUROPP – European Politics and Policy, nor of the London School of Economics.Shortened URL for this post: http://bit.ly/29yqtgK _________________________________About the authorsHarold D. Clarke is Ashbel Smith Professor, School of Economic, Political and Policy Sciences, University of Texasat Dallas, and Adjunct Professor, Department of Government, University of Essex.4/5Matthew Goodwin is Professor of Politics and International Relations at the School of Politics and InternationalRelations, University of Kent.Paul Whiteley is Professor at the Department of Government, University of Essex.References[i] The polling data for the UK EU referendum are available here. “Undecideds” are allocated proportionately to theRemain and Leave categories[ii] See Simon Jackman, ‘Pooling the Polls Over an Election Campaign‘. Australian Journal of Political Science 40:499-517. The analysis employs Bayesian MCMC methods. Posterior distributions for model parameters aregenerated using Winbugs 14 in conjunction with the R2Winbugs R package.The pooling the polls model has two equations—a measurement equation and a state equation for the latent level ofRemain/Leave support. The measurement equation is:yit = δi + αt + εitwhere: yit = is a poll by survey house i at time t; δi = house or mode effect parameter; αt = latent vote intention(Remain or Leave) at time t and εit is a stochastic error ~N(0,σ12).The state (latent transition) equation is a random walk: αt = 1.0* αt-1 + ζt whereζt ~ N(0, σ22). εit and ζt are assumed to be independent.[iii] Statistics reported in Figures 1, 2 and 3 are means of posterior distributions of parameters of interest.5/5

Leave was always in the lead: why the polls got the referendum result wrong

05/07/2016Leave was always in the lead: why the polls got thereferendum result wrongblogs.lse.ac.uk/politicsandpolicy/eu-referendum-polls/By analysing 121 opinion polls, Harold D. Clarke, Matthew Goodwin, and PaulWhiteley outline what happened with the EU referendum survey results. They explain whyinternet surveys performed substantially better than telephone ones – contrary to the post-2015 General Election ‘wisdom’ that telephone surveys should be preferred. Underlying trendsshowed that once methodological artefacts are controlled, Leave was almost certainly ahead ofRemain over the entire last month of the campaign – and possibly throughout 2016.After the 2015 general election, pollsters were roundly criticised for their failure to predict theConservative victory. Did they fail again in the EU referendum? If so, just how badly did they do?Did they all get it very wrong, or did some do much better than others? And, taking artefactsintroduced by polling methodology into account, what did underlying trends in Remain and Leavesupport look like in the run-up to voting day? Going forward, what lessons can we learn?We address these questions by using a technique called ‘dynamic factor analyses’ on 121 internetand telephone surveys conducted between January 11 and June 22, 2016.[i] We use astatistical model originally developed in research on the performance of polls conducted in anAustralian national election campaign.[ii] A large majority of the survey data employed herewas gathered by nine survey organisations (‘houses’) and we also include several additionalsurveys in a residual ‘miscellaneous’ category for other organisations reporting one to threepolls.ModesWe begin by investigating what survey researchers call ‘mode’ eﬀects. These eﬀects concern the impact ofdiﬀerences in the way surveys are conducted on the results they produce. In the case of the 121 EU referendumsurveys of interest here, all were administered either by internet (N = 81) or telephone (N = 40). Analysing theresults (percentages of Remain and Leave supporters in various surveys) using our model reveals that bothtypes of polls signiﬁcantly overestimated the Remain vote share and underestimated the Leave share (seeFigure 1). However, there were sizable diﬀerences between the two types of polls, with the mode eﬀect forequaling 5.1 per cent for internet polls as compared to fully 9.2 per cent for telephone polls.[iii] Taken as a group,internet polls tended to perform considerably better than their telephone-based rivals.1/4HousesWhat about the performance of individual survey companies? Our analysis of house eﬀects indicates that surveyﬁrms varied markedly in their ability to gauge EU vote intentions. Figure 2 documents that, for the 9 surveyorganisations considered, TNS-BMRB, ICM and YouGov performed best — their overestimates of the Remainvote share were +2.24 per cent, +2.71 per cent and +3.24 per cent, respectively. Similarly, these three housesdid best on estimating Leave support. TNS-BMRB underestimated the Leave share by -2.35 per cent, with ICMmissing by -2.81 per cent and YouGov, by -3.34 per cent. All of the surveys conducted by TNS-BMRB (5surveys) and YouGov (28 surveys) were done by internet, as were 23 of 26 ICM surveys.Note: solid colored bars are for survey houses doing all or most of their polling byinternet; striped bars are for survey houses doing all or most of their polling bytelephone.  Populus has 3 internet polls and 2 telephone polls.2/4The three ﬁrms performing worst were Populus, IPSOS and ComRes, with Remain house eﬀects equaling+6.02, +.6.34 and +6.72 per cent, respectively. Populus missed the Leave percentage by -6.10 points, whereasIPSOS missed by -6.45 per cent and ComRes by -6.85 per cent. All 7 of IPSOS polls were done by telephone,as were 9 of the 11 ComRes polls, and 2 of the 5 Populus polls. Survation was another house that performedpoorly (a +5.54 per cent overestimate for Remain and -5.67 per cent underestimate for Leave).  All but one ofSurvation’s 8 polls were conducted by telephone. ORB, with a Remain house eﬀect of +4.76 per cent and aLeave house eﬀect of -4.87 per cent, did 9 of its 13 polls by telephone.Underlying TrendsBy controlling for mode and house eﬀects, our analyses enable us to estimate underlying trends in the dynamicsof support in EU referendum vote intentions. The results (Figure 3) indicate that Leave led Remain over theentire period from 11 January 2016 onward. The size of the Leave lead varies widely—from a low of .39 per cent(4 February) to a high of 13.2 per cent (12 June)—but Leave is always ahead.The analysis also provides insight regarding the eﬀects of various events that occurred during the campaign. Forexample, contrary to his intentions, US President Barack Obama’s widely publicized ‘UK to the back of thequeue’ intervention may have boosted, rather than diminished, Leave support. In contrast, as numerousobservers have speculated, the murder of Labour MP Jo Cox on 16 June may have precipitated an ongoingerosion in Leave support over the ﬁnal week before the vote. That downturn in Leave’s vote intention sharenotwithstanding, the analysis documents that over the last month of the campaign the lower bound of the 95 percent credible (conﬁdence) interval for the Leave vote always exceeds the 50 per cent mark. This means thatLeave was very likely ahead throughout this entire period.What lessons can we learn?Overall, the analyses suggest two general lessons for consumers of political polls. First, house eﬀectsassociated with selected internet survey ﬁrms were quite modest, whereas ﬁrms relying heavily or exclusively ontelephone surveys found themselves considerably further oﬀ the mark. After the 2015 general election, internetsurveys were widely criticized and some commentators claimed that telephone surveys were a preferredalternative. At the time this assertion seemed dubious given that telephone surveys’ typically dismal responserates clearly gainsays any claim to be true ‘probability’ surveys. As documented above, the relatively poor3/4performance of telephone surveys in the EU referendum reinforces doubts about their superiority.A second lesson is that the ‘topline’ results from surveys are not enough. If observers had used the polling dataas inputs to statistical models of the underlying dynamics of EU referendum vote intentions rather than riskingbeing misled by the vagaries introduced by mode and house eﬀects in successive individual polls, they might nothave been surprised by the Brexit result. High quality polls by reputable survey houses should be viewed as akey resource—not a substitute—for informed political analysis.___About the AuthorsHarold D. Clarke is Ashbel Smith Professor, School of Economic, Political and PolicySciences, University of Texas at Dallas, and Adjunct Professor, Department of Government,University of Essex. Matthew Goodwin is Professor of Politics and International Relations at the School of Politicsand International Relations, University of Kent.  Paul Whiteley is Professor at the Department of Government, University of Essex.  References[i] The polling data for the UK EU referendum are available here. “Undecideds” are allocatedproportionately to the Remain and Leave categories[ii] See Simon Jackman, ‘Pooling the Polls Over an Election Campaign‘. Australian Journal of Political Science40: 499-517. The analysis employs Bayesian MCMC methods.  Posterior distributions for model parameters aregenerated using Winbugs 14 in conjunction with the R2Winbugs R package.The pooling the polls model has two equations—a measurement equation and a state equation for the latentlevel of Remain/Leave support.  The measurement equation is:yit = δi + αt + εitwhere: yit = is a poll by survey house i at time t; δ i = house or mode eﬀect parameter;  αt = latent vote intention(Remain or Leave) at time t and εit is a stochastic error ~N(0,σ12).The state (latent transition) equation is a random walk: αt  = 1.0* αt-1 + ζt  whereζt ~ N(0, σ22).  εit and ζt are assumed to be independent.[iii] Statistics reported in Figures 1, 2 and 3 are means of posterior distributions of parameters of interest.4/4

Leave was always in the lead: why the polls got the referendum result wrong

Abstract

Similar works

Full text

Available Versions

LSE Research Online

LSE Research Online