The result of the 2015 General Election came as a surprise for most people, but particularly those in the academic and polling community. But what is the appropriate role for academics in an electoral setting, particularly when it comes to complicated issues like the integrity of electoral contests. Ferran Martinez i Coma and Carolien Van Ham seek to answer this question, and conclude that expert surveys are useful even when treating complex and multi-faceted issues, such as electoral integrity; and even when carried out in institutional settings as different as liberal democracies and electoral autocracies

Martinez i Coma, Ferran

Van Ham, Carolien

LSE Research Online

By Democratic Audit UK 2015-6-9How seriously should we take the opinions of academicsand experts when it comes to complicated issues likeelectoral integrity?democraticaudit.com /2015/06/09/how-seriously-should-we-take-the-opinions-of-academics-and-experts-when-it-comes-to-complicated-issues-like-electoral-integrity/The result of the 2015 General Election came as a surprise for most people, but particularly those in the academicand polling community. But what is the appropriate role for academics in an electoral setting, particularly when itcomes to complicated issues like the integrity of electoral contests. Ferran Martinez i Coma and Carolien VanHam seek to answer this question, and conclude that expert surveys are useful even when treating complex andmulti-faceted issues, such as electoral integrity; and even when carried out in institutional settings as different asliberal democracies and electoral autocracies.Senate House, University of London (Credit: Steve Cadman, CC BY SA 2.0 )For many years, social scientists have been using different databases that measure and classify complex,multidimensional and contested concepts such as democracy, freedom or corruption.The utility of such data is evident not only for academics but also for the policy and the advocacy community. At aglance, such data summaries the state of democracy or corruption in a country, and position it relative to others witha score or ranking. Yet, such scores are not created in a vacuum. On the contrary, data on multi-dimensional andcomplex concepts such as democracy, freedom or corruption are normally generated through a process ofmeasurement of multiple indicators and subsequent aggregation of those indicators.In order to measure complex concepts, we need to gather information about their different elements or components.To do so, researchers could rely on public opinion surveys or information contained in media and other secondary1/4sources. However, the level of complexity and the necessary knowledge to address some issues may not be aneasy task. Consider, for example, the electoral management body’s autonomy. While probably the general public isable to have an overall assessment of its performance and also the public may know whether such body is formallydependent of the government or not, it is unlikely that the public know the details of the implications of changes inthe autonomy of the electoral management body is violated.An alternative approach is to measure complex and multidimensional concepts with expert surveys. There are goodreasons for the use of expert surveys. First, experts are aware of the specificities of the concrete matter, since theyhave the knowledge and capacity to grasp the fine details. Moreover, experts may have access to information thatcitizens do not have access to, potentially providing better data on covert practices such as corruption. Third, theyconsiderably diminish the costs of other polling alternatives.  Finally, they have been widely used to study, to mentiona few, corruption; democracy and its components; party and policy positioning,  the power of prime ministers,evaluations of electoral systems, or policy constraints horizons.However, expert surveys are not risk free. There are several limitations. The first question has to do with the objectof evaluation: do experts judge the same aspects of the concept under study? The second is on the criteria thatexperts use when judging: do they rely on their expertise, or do they also provide personal views? Third, as expertsurveys become more comprehensive and encompassing with more and more diverse experts around the world, itis fair to ask whether they share the same criteria when evaluating concepts or whether their judgments aredependent on the context in which the election take place. Finally, it is also the case that, in contrast to masssurveys, there is still no common methodology to construct expert surveys, nor agreed technical standards andcodes of good practice. This is very relevant not only for research but also for policy-makers and practitioners usingindices and rankings based on expert surveys.Given the potential advantages and limitations of expert surveys, in our paper in the European Journal of PoliticalResearch (EJPR), we assess the validity of experts’ judgments when judging the integrity of elections. We analysethree sources of bias that may arise in expert evaluations: the object, the experts and the context. These sources ofbias are applicable to almost all expert surveys. First, the object  of evaluation may be defined and perceiveddifferently by different experts. Election integrity is a complex, multifaceted concept, and different experts mayemphasise different aspects, ranging from media bias to election violence. Second, experts  may differ, both in theirlevel of expertise as well as in their degree of political neutrality. Third, contexts may differ – that is, expertevaluations may be context-bound, limiting the capacity of both concepts and data to ‘travel’.We test these three sources of bias and evaluate expert judgment validity using a new dataset on expertperceptions of election integrity, the Perceptions on Electoral Integrity (PEI) that asks experts to evaluate 49 specificindicators of election integrity. This database includes 49 variables measuring 11 dimensions of electoral integrityover the electoral cycle. The survey encompasses the full electoral cycle, ranging from the pre-electoral period, thecampaign, to polling day and its aftermath, as outlined by the United Nations. The PEI data currently has responsesof over 800 experts on 66 parliamentary and presidential elections that took place in 2012 and 2013, coveringcountries as diverse as Angola, Kuwait, Malaysia and Norway.An expert is defined as a political scientist (or social scientist in a related discipline) who has published or who hasother demonstrated knowledge of the electoral process in a particular country. By ‘demonstrated knowledge’ PEIunderstands one of the following criteria:(1) membership of a relevant research group, professional network, or organised section of such agroup;(2) existing publications on electoral or other country-specific topics in books, academic journals, orconference papers;2/4(3) employment at a university or college as a researcher or professor.For each election, the PEI survey identified and contacted around forty experts, seeking balance between domesticand international experts. When the number of available domestic experts was limited, as was the case in somedeveloping countries, PEI relies more on international experts.There are three main findings of our research. First, considering the object of evaluation, we find that questions of afactual nature generate lower deviation in expert judgments than more evaluative questions. We also find evidencethat questions that are more difficult to answer, either because the issues are technical or because the informationmight not be publicly available (i.e. voter registration, campaign finance) generate higher deviation in expertjudgments.Second, when we analyse the heterogeneity of the experts, we argue that they may differ both in their level ofexpertise as well as in their degree of neutrality. We find that having a high level of knowledge about the election (asindicated by the number of questions answered and age) is not significant in predicting expert variance. However,having strong ideological preferences appears to affect variance between experts. This result underscores theimportance of careful selection of experts as well as the consideration of their partisan background.Third, we also study whether the context –the election they assess and the country in which they are living- impactsexperts’ judgments. Among all the factors we include to capture context, almost none seems to impact the variationof expert judgments. The only element that seems to matter is the ideological polarisation between experts,increasing the variability of expert judgments.Concluding, our overall results demonstrate that expert surveys are useful even when treating complex and multi-faceted issues, such as electoral integrity; and even when carried out in institutional settings as different as liberaldemocracies and electoral autocracies.There are several implications from our research both for policy as well as for future research. First, our findingsdemonstrate the importance of testing the validity of expert surveys prior to using these data for substantiveanalyses, so that validity problems can be identified and dealt with. Second, our findings underscore the importanceof careful selection of experts and taking into account their partisan background when collecting expert survey data.Third, the widespread use of indices based on expert survey data, such as indices on corruption and democracy, bypolicy-makers and practitioners, underscores the need for developing technical standards and codes of goodpractice for gathering data using expert surveys.—This post represents the views of the author and not those of Democratic Audit or the LSE. Please read ourcomments policy before posting. —Dr Ferran Martínez i Coma  is a Research Associate at the Electoral Integrity Project at theUniversity of Sydney. Prior to this position, he was a Technical Adviser for elections for the GeneralDirection of Interior Policy of the Ministry of Internal Affairs, in Madrid, Spain. Carolien Van Ham is a Lecturer in Politics at University of New South Wales, Sydney, Australia3/44/4

How seriously should we take the opinions of academics and experts when it comes to complicated issues like electoral integrity?

Abstract

Similar works

Full text

Available Versions

LSE Research Online