The British Polling Council recently published their report about what went wrong with the polls in the 2015 general election. The report dismissed problems associated with voter registration, question wording, postal voting and mode of interviewing and opted for unrepresentative samples as the key factor in explaining what happened. Here, Paul Whiteley and Harold Clarke make the case for examining another factor: respondents who don’t tell the truth

Clarke, Harold

Whiteley, Paul

LSE Research Online

1/25/2016Representative samples are an issue for the pollsters – but soare respondents who lieblogs.lse.ac.uk/politicsandpolicy/the-2015-polling-debacle-the-british-polling-council-is-right-to-focus-on-problematic-samples-but-what-about-respondents-who-dont-tell-the-truth/The British Polling Council recently published their report about what went wrong with the polls inthe 2015 general election. The report dismissed problems associated with voter registration,question wording, postal voting and mode of interviewing and opted for unrepresentative samplesas the key factor in explaining what happened. Here, Paul Whiteley and Harold Clarke make thecase for examining another factor: respondents who don’t tell the truth.For statisticians, random samples are the ‘gold standard’ if one wants to get an accurate picture ofhow people will vote. The theory is straightforward: if every voter has an equal chance of ending upin the sample – no matter how small – then that sample should give us an accurate picture of theelectorate. No significant group of people will be left out because they all have a chance of beinginterviewed. In contrast, quota samples require interviewers to find groups of respondents withcombinations of socio-demographic characteristics that match census profiles of population sub-groups. By doing so, quota samples attempt to mimic random samples but, as any statistician willquickly argue, they are not the same.So, how do the pollsters go about getting random samples? One method used in telephone surveys is to dialnumbers at random and in theory this should provide the desired sample, albeit of people with telephones.Unfortunately, response rates for such exercises are now very poor, in some cases falling below 10 per cent. Peopleincreasingly screen out marketing calls or merely hang up the phone regarding telemarketing as a nuisance. Soachieved samples are very different from the ideal of a random sample. It is quite heroic to assume that the smallpercentage of people who do grant a phone interview are a random subsample of those who were originallycontacted.A similar point can be made about surveys conducted by interviewers in the respondent’s home – so-called face-to-face surveys. The samples for these are selected randomly but they increasingly fail to interview many of the peopleon the list, which produces a biased sample. An example of this is the 2015 British Election Study which achieved aresponse rate of only 55.9 per cent in its face-to-face survey conducted after the election. This is well down on theresponse rates achieved by the study thirty years ago. Nor is this just a British problem. The response rate for the2012 American National Election Study was only 38 per cent. Again, it is difficult to imagine that those who doparticipate in face-to-face surveys are a completely random sub-sample of the original sample chosen for aninterview.Internet surveys take a different approach and use quota samples, obtained by signing up people to do a certainnumber of surveys each month. These internet panels are usually very large and, not surprisingly, don’t tend to befully representative of the population. Polling companies can deal with this problem in two ways. One is by selectingpeople from the panel in light of census information, data on media consumption patterns, and party support so thatan achieved sample is steered in the direction of being representative of the country as a whole. A second approachis to weight the data afterwards, so called ‘post-stratification weighting’. There are a variety of schemes for doingthis, but they all involve counting some hard-to-reach respondents as being more important than the easy-to-reachones. Both of these strategies are used to try to approximate a random sample.The 2015 British Election Study (BES) did both internet and face-to face surveys in Great Britain after the election tosee how people voted. So, direct comparisons can be made between the two, to see how accurate they were inmeasuring parties’ vote shares. The face-to-face survey reported that Labour got 32.6 per cent of the vote and the1/4internet survey reported 34.3 per cent. Since Labour actually took 31.2 per cent of the vote, the face-to-face wasmore accurate. In the case of the Conservatives, the face-to-face survey said that the party received 40.6 per cent,and the internet survey reported 34.8 per cent. Given that the Conservatives actually obtained 37.7 per cent, theerrors for the two modes of interviewing were exactly the same at 2.9 per cent. Differences for the other parties werevery small, with the BES in-person and internet surveys missing the Liberal Democrat, UKIP, Green and SNP voteshares by averages of 1.1 per cent and 1.0 per cent respectively. None of these differences for the minor parties arestatistically significant. Overall, the BES post-election face-to-face survey was better than the BES post-electioninternet survey only in the case of the Labour vote.These comparisons are for post-election surveys. What about pre-election polls commentators employ for electionforecasts? The face-to-face mode is a non-starter for these polls: they take literally months to collect. The 2015 BESin-person survey was in the field from May 8th to September 13th. Face-to-face surveys are also hugely expensive,far beyond what a survey firm could expect a client to pay, even on a one-time basis. In practice, pre-electionsurveys have to be done using telephone and internet polls. Is there evidence that one of these modes ispreferable?When making this comparison it is possible to take advantage of the large number of telephone and internet surveysconducted during the five years between the 2010 and 2015 elections. A total of 1,942 polls were conducted duringthis period and, with the assistance of some multivariate modelling, we can identify latent voting intentionsunderlying the data. These latent vote share variables separate out the ‘signal’ from random noise in a large numberof opinion polls and so capture the true levels of support for various parties better than any single poll could do.Using this approach, we can see if telephone and internet polls differ when it comes to measuring true votingintentions over time.Figure 1 shows the average ‘house’ effects or errors associated with different survey companies alongside the‘mode’ effects, or errors associated with using telephone as opposed to internet surveys. The figure shows thatsome survey houses had a better track record than others, but they were all within the standard margin of error of 3per cent associated with the average poll. The average error for telephone surveys was 2.3 per cent, and for internetpolls it was 2.1 per cent, and so the difference is negligible. The statistical evidence in Figure 1 indicates that somesurvey houses were more accurate than others, but does not demonstrate the superiority of the telephone mode tointernet or vice versa.2/4All this creates a problem for explaining why the polls got it wrong in 2015. If there are no differences between theerrors in telephone and internet surveys and face-to-face surveys would not greatly improve matters – evensupposing they could be done in a cost-effective and timely way – why did the polls get it wrong in 2015 when theygot it right in 2010 and in earlier elections? This is a puzzle given that problems of getting accurate measures ofpublic opinion are not new but have existed for many years. The answer that the Polling Council gave is that thesamples polling houses use are less than optimal and may have deteriorated in recent years. Certainly, this adviceis worth heeding. But, there may be another reason as well.There is another possible explanation which has been neglected in the analysis of what went wrong in 2015. Writingshortly after the polling debacle in 1992, Ivor Crewe suggested that Britain was becoming ‘a nation of liars’. He mayhave been onto something. Not only is it becoming more difficult to get respondents to participate in surveys, it alsomay be that those who do take part are more likely to lie to interviewers about whether they voted. This was not aproblem for the 2015 election night exit poll which got the result close to spot on. For this survey it was clear that allof the participants actually voted because they were interviewed right after leaving the polling stations. This is nottrue for individuals who respond to pre-election polls. If these polls are picking up more and more people who claimthey will vote or have voted, but in reality this is not true, this will widen the gap between reported voting intentionsand what actually happens on election night.In the 2001 and 2010 British Election Studies respondents interviewed in the ‘gold-standard’ post-election face-to-face surveys were asked if they had voted in the election. The BES team then organised voter validation exercisesto find out who actually cast a ballot. This was done by inspecting records on electoral participation which are heldby the state for legal reasons. The records make it possible to identify if someone voted, but not of course how theyvoted. Using these data it is possible to document the number of respondents who ‘misreported’ their electoralparticipation to the BES interviewers.Figure 2 shows that in 2001 62.7 per cent of the respondents truthfully claimed to have voted in the electioncompared with 70.8 per cent who did so in 2010. In addition, 20.6 per cent truthfully recorded that they did not votein 2001 and 17.4 per cent did so in 2010. The most interesting group are the non-voters who falsely claimed to havevoted in each of the elections. There were 7.6 per cent of these ‘false positives’ in 2001 and 10.4 per cent in 2010—3/4a 36.8 per cent increase. ‘False negatives’ – people who claimed not to have voted when they actually did – was amuch smaller group in both years. Again, the false nonvoters increased across the two elections, from 0.3 per centin 2001 to 1.5 per cent in 2010.These figures documenting the incidence of false reports come from two face-to-face post-election surveys, butthere is no reason to believe that misreporting is confined to people participating in in-person surveys or post-election surveys. Indeed, since the over-reporting of voting turnout is considerably greater for internet and telephonepolls than it is for face-to-face surveys, the problem of lying is potentially greater for these survey modes. Moregenerally, if the number of people lying about whether they have or will cast a ballot continues to increase, then pollsof whatever mode are going to become more inaccurate over time.The upshot is that pollsters would be well-advised to pay more attention to developing accurate ‘likely voter’ models.Working to improve the overall quality of survey samples as the British Polling Council suggests should help, butthere is still a problem for those wishing to use pre-election polls for forecasting purposes if a sizable number ofrespondents, however sampled, do not tell the truth about whether they actually cast a ballot. Developing effectivelikely voter filters to cull the liars promises to be a cost-effective way to improve the accuracy of pre-election polls.___About the AuthorsPaul Whiteley is Professor in the Department of Government, University of Essex  Harold D. Clarke is Professor in the School of Economic, Political and Policy Sciences, University ofTexas 4/4

Representative samples are an issue for the pollsters – but so are respondents who lie

Abstract

Similar works

Full text

Available Versions

LSE Research Online