This paper explores the link between accountability, evaluation and identity in the context of charitable funders. We argue that the notion of ‘evidence based’ action has influenced these charities to make claims for their evaluation processes that are established to account for activity rather then to generate knowledge. Although the terms evaluation and accountability are frequently used interchangeably, there is an essential difference as the site of interest in accountability is the relationship between funder and funded, whilst for evaluation, it is quality of knowledge produced. We look at the consequences of this distinction on identity for two funders in an analysis of website pamphlets, published to promote the use of evidence

Hinds, KS

Mackenzie-Davey, K

UCL Discovery

Accountability, Evaluation and the Role of Evidence for Charitable Funders This paper explores the link between accountability, evaluation and role of evidence in the context of charitable funders. Although the terms evaluation and accountability are frequently used interchangeably, there is an essential difference as the site of interest in accountability is the relationship between funder and funded, whilst for evaluation, it is quality of knowledge produced.   We argue that charities draw on the notion of ‘evidence based’ action to make claims for their evaluation processes that are established to account for activity rather than to generate knowledge. We draw on an analysis of websites and pamphlets of three funders to explore the different approaches to the use of evidence in the decision process.  BAM Track: Knowledge and Learning Word count: 2000Introduction The notions of ‘evidenced based’ policy and practice are increasingly used by charitable foundations to promote themselves as responsible citizens of political society (e.g. Educational Endowment Foundation 2014).  The ambition to learn and also to judge, rationally, brings together the tasks of evaluation and accounting.  The concept of evaluation suggests the neutral purpose of learning about the effectiveness of social interventions in order to improve impact.  Thus learning incorporates a tolerance of failure which, by contrast, damages reputation in an accountability regime.   In this paper, we discuss how organizational reality and societal discourses have shaped accountability and evaluation.  This paper goes on to examine the arguments made by two funders for evidence in pamphlets that set out their ideas and contrasts these with a third that has a more traditional, religious approach to philanthropy.  All three are endowed charities, meaning that they are free from fundraising, and thus, notionally, less scrutinised by public stakeholders.  We argue that this freedom allows them to be more experimental in their grant making. Evaluation and Accountability Much evaluation research is characterised by claims about the value and use of the ‘scientific method’, usually meaning a positivist approach, a comparison group design, and the use of professional researchers.  This positivist emphasis is particularly strong in discussions about evidence based medicine and management (Morrell 2008).  However, as Legge (1984) and others (e.g. Weiss 1973, 1975) argue, the political context of organizations, compromise the science of the method and challenge the values of the evaluator.   Methodological concerns about appropriateness and usefulness of the scientific method in complex organizational settings have been raised (Weiss 1972, Pawson and Tilley 1997).  Although other strategies for evaluating change have been tried (e.g. Guba and Lincoln 1981), the ‘evidence into policy’ lobby overwhelmingly promotes the randomised controlled trial (e.g. Goldacre 2014).   The espoused product of evaluation is greater understanding of the intervention under study, to support rational decision making.  The intention of evaluators is not to scapegoat, although the consequences for advocates of a failed intervention within an organization may be harsh.  Indeed, it would be naive to assume that evaluation in organizations can always be separated from the discipline of holding people to account. Whereas the scientific method of evaluation research is muddied by organizational context, the relational function (Bovens 2005) of accountability is situated within a societal context that values rationality.   Accounts are created by organizations to influence opinions of their activities and performance, but the narratives do not form their norms or practices (Black 2008).  Rather, accounts both shape the identity of the accounting organization and the authority to which the account is given. Accountability may thus result in a symbolic exchange of information, designed to enhance the reputation of both the giver and receiver (McNulty 2012).   Accountability practice borrows from scientific practice to increase its own legitimacy and trustworthiness.  Information management, through auditable standards, follows a pattern of abstraction and de-contextualization which mimics scientific practice.  This falls in with ideas about the management of society through data and knowledge (Tsoukas 1997), and the rationalisation of action (Locke 2001). However, information management is created for the purposes of bureaucracy rather than the development of meaning (Tsoukas 1997) The critics of audit identify it as a tool for control across many organizations and contexts, rather than as a way of revealing a meaningful and nuanced understanding of organizations and change (Strathern 2000, Tsoukas 1997). The transparency of audit strengthens our trust in other organizations but also encourages defensive and self-protective strategies as we witnessed in the recent financial crisis (Roberts 2009).  Asking charities to account for their expenditure is the central task of the management of grant contracts.  Accountability also establishes the reputation of charities by opening up the organization to democratic scrutiny (Gray et al. 1991), by increasing its moral legitimacy to act in public debates (Lister 2003) as well as making available the rational discourses of modernity (Locke 2001).   However, charities must demonstrate success to participate in public debate and to secure funding, and thus accountability processes push them into strategies to protect their reputations (Ebrahim 2005b).  Their charitable funders aid them to realise their reputational capital from their investment in the funded charity, to build their own reputations (Ebrahim 2005a).   Not only are organizations concerned to protect themselves, but this motivation is also found in individuals (Sinclair 1995).  Thus the pressures of organizational and personal accountability drive evaluative research further into the realms of political activity, and away from the ideal of neutral, scientific assessment.  Nevertheless, the organization must uphold the ideals of transparency and science to be considered successful.   The Role of Evidence in Decision Making Processes of Charitable Foundations  One arena where funders present accounts of themselves is on their websites (Silliance and Brown 2009).  In exploring how charitable funders position themselves in relation to evidence, we examined pamphlets from the websites of two UK funders who have made a public statement about the importance of evidence in their work.  We contrast these with an excerpt from a document from a Quaker funder, the Joseph Rowntree Charitable Trust (JCRT), describing their decision making process.  We have chosen these three charities because they demonstrate different stances to the inclusion of evidence in decision making.  The JCRT locates their procedures within the Quaker tradition and emphasizes emotion and contemplation, Nesta identifies itself with the entrepreneurial values of social investing, and the Paul Hamlyn Foundation stresses its historical links to disadvantage. We use short opening paragraphs from the documents to make our arguments. Firstly, as an example of an older sense of philanthropy, we examine the conduct of the JCRT.  ‘The JRCT is a Quaker Trust and it conducts its business ‘after the manner of Friends’. One of the functions of a Quaker Meeting for Worship is to try to discern the will of God and, similarly, in the conduct of business Quakers try to discern the will of God relative to particular issues. (For the non-religious, ‘the will of God’ can be translated as ‘what is right’.)  Later in the document, the writer describes the management of the meeting by the Chair.  After a discussion to clarify issues concerning each application, there is a more open debate:  ‘This is the time for speaking freely, often with passion and from the heart, before letting go of one’s personal views. Ideally there will be a short period of silence between each speaker, so that we can properly consider what we have heard.’  Here the participants seem to take a greater personal responsibility for the decision and do not rely solely, or even at all, on evidence.  This seems to go against the prevailing ideas about rational management (Tsoukas 1997).  However, their trustees avail themselves of an older non-rational and religious incentive to make arguments for or against particular applications (Billig 1991).     In contrast, Nesta sets out its method, drawing on rational arguments in the pamphlet, ‘Impact Investing’ (Puttick and Ludlow 2012): ‘…..The prospect of being able to tackle social challenges whilst making a proﬁt is an attractive proposition. But too often evidence of impact is missing. We need evidence to establish whether a product or service is beneﬁting those it sets out to serve, and then to focus investment on products and services that can make the most difference. To help us achieve this we have developed a new approach, Standards of Evidence for Impact Investing.   ….This fund will be investing in high–risk or earlier–stage innovations where we intend to create value, not only in ﬁnancial terms but also in terms impact.’   This funder claims to be concerned with profit as well as social change, and needs evidence to make the choices between the ‘products’ more straightforward.  The language is from the market and economics.   The tone is upbeat, entrepreneurial, and suggests progress.  They borrow from discourses about the superior abilities of private sector management to produce results (Eyben 2013).  The collection of evidence appears as an unproblematic task that others have neglected, but with a rational economic approach, this can now be tackled successfully.  The paragraphs also link the charity to evidence and its associated ideas of learning and judgement.  The third sentence suggests an interest in a comparison of services in relation to user benefit, and implicitly in their relative success and failure.  The extract leaves an impression of learning from evaluation in order to control the future.  This is reinforced by the final paragraph.    An accounting motivation is supported by the name of their new approach: ‘Standards of Evidence for Impact Investing’.  Here is the introduction of a measure against which all can compare themselves, in the project of continuous improvement (Roberts 2009).  Risk implies experimentation, but is linked to the idea of entrepreneurship, rather than science.  By contrast, in its foreword to ‘Assessing Impact’, the Paul Hamlyn Foundation emphasises their founder’s values in order to make a case for its future.   ‘The Paul Hamlyn Foundation’s mission is to maximise opportunities for individuals and communities to realise their potential and to experience and enjoy a better quality of life, now and in the future…  In line with our founder Paul Hamlyn’s values, we believe in finding better ways to do things. We try to pay particular attention to long-term and challenging issues and encourage participation by those with direct experience of these issues in shaping the work of the organisations we fund. We look for the development of work with the potential to influence practice beyond the scope of the organisation doing the work.   We know that to fulfil our mission and live these values, we need to understand the impact that the Foundation is making. The sorts of changes the Foundation seeks are too important to do otherwise and we rely on evidence of impact to help us to use the Foundation’s resources wisely.’  Their focus is on people in need and on the organizations which provide the help.  Taking the idea of improvement as suggested in the phrase ‘finding better ways to do things’, they argue for the importance of evidence to understand how they are doing this.   Continuity is the emphasis, from the founding sentiments to their concern for ‘long-term issues’.  Their suggestion that the social changes, they seek, are long lasting and therefore important, is emphasized by the word ‘wisely’ with its connotations of knowledge gained from experience over many years.  Evidence appears as tool that can be relied on to help the foundation make its choices.    Like Nesta, they are interested in success and promoting the idea of progress, seen in phrases like ‘maximise opportunities’, ‘realise their potential’, and ‘better quality of life’.   Evidence is used instrumentally to guide rational decision making and to persuade others to change their practice.  Additionally, they seek legitimacy by claiming solidarity with the disadvantaged, rather than aligning themselves with the organizations they fund.  Positioning themselves closer to those in need increases their legitimacy and improves their ability to speak for them (Lister 2003).    The decision makers at Nesta and the Paul Hamlyn Foundation use evidence to distance themselves from the personal responsibility of decision making unlike the trustees at the JRCT. Therefore they can feel more confident of their ability to defend their decisions when called to account by others.  Rational decisions seem easier to understand than emotional choices.     Conclusion Evaluation and accountability are tasks with different purposes, one for learning and the other for justification, that have become intertwined as one term is substituted for another in debates about organizational performance.    Charities use these terms, accountability and evaluation or research, to increase their legitimacy.  Accountability may increase transparency, and thus improve the organization’s openness to democratic scrutiny.  Research adds the impression of rationality as a guiding principle to the organization’s decision making, in a society where this idea has become such a commonplace (Billig 1987). However, the culture of accountability, with its myth of continuous improvement, impedes the discussion of failure, and thus the creation of knowledge, because of its defensive nature.    How will this paper be developed? I am developing an analysis based on rhetoric and how evidence is used to make arguments will be a part of this analysis.   This analysis will inform my field work with funders as I follow them as they produce and use evidence.   References Billig M (1987) Arguing and thinking: a rhetorical approach to social psychology.  Cambridge UK: Cambridge University Press Billig M (1991) Ideology and Opinion: Studies in Rhetorical Psychology.  London: Sage Black J (2008) ‘Constructing and contesting legitimacy and accountability in polycentric regulatory regimes.’ Regulation and Governance 2 pp 137-164 Bovens M (2005) ‘Public Accountability’ In E. Ferlie, L. Lynn and C. Pollitt (Eds), The Oxford Handbook of Public Management pp: 182-208.  New York, NY: Oxford University Press Ebrahim A (2005a) ‘Accountability myopia: losing sight of organizational learning’ Nonprofit and Voluntary Sector Quarterly 34: 1 pp 56-87 Ebrahim, A. (2005b) NGOs and Organizational Change: Discourse, Reporting and Learning. Cambridge, UK: Cambridge University Press. Education Endowment Foundation (2014) http://educationendowmentfoundation.org.uk/about/ Eyben R (2013) ‘Uncovering the politics of ‘evidence’ and ‘results’.  A framing paper for development practitioners’ http://bigpushforward.net/wp-content/uploads/2011/01/Uncovering-the-Politics-of-Evidence-and-Results-by-Rosalind-Eyben.pdf Goldacre B (2014) http://www.badscience.net/category/evidence-based-policy/ Gray, R. H., Owen, D. L., & Maunders, K. (1991) ‘Accountability, Corporate Social Reporting and the External Social Audits’. Advances in Public Interest Accounting, 4:1-21. Guba EG, Lincoln YS (1981) Effective Evaluation.  San Francisco: Jossey Bass Joseph Rowntree Charitable Trust (2013) ‘The Conduct of Business at the JRCT’ downloaded from http://www.jrct.org.uk/ on 25/09/213 Legge K (1984) Evaluating Planned Organizational Change.  London: Academic Press Lister S (2003) ‘NGO legitimacy: technical issue or social construct?’ Critique of Anthropology 23 pp 175-192 Locke S (2001) ‘Sociology and the public understanding of science: from rationalization to rhetoric’  British Journal of Sociology 52 pp 1-18 McNulty J (2012) ‘Symbolic uses of evaluation in the international aid sector: arguments for critical reflection’ Evidence and Policy 8: 4 pp 495-509 Morrell K (2008) ‘The narrative of evidence-based management: a polemic’ Journal of Management Studies 45 pp 613-635 Paul Hamlyn Foundation (2013) http://issuu.com/paulhamlynfoundation/docs/phftrusteesreport2011_12?mode=window&viewMode=doublePage  (accessed 19/04/13) Pawson R, Tilley N (1997) Realistic Evaluation.  London: Sage Puttick R, Ludlow J (2012) ‘Standards of Evidence for Impact Investing’ London: NESTA http://www.nesta.org.uk/library/documents/StandardsofEvidenceforImpactInvesting.pdf Roberts J (2009) ‘No one is perfect: the limits of transparency and an ethic for ‘intelligent’ accountability.’ Accounting, Organizations and Society 34 pp 957-970 Sillance JAA, Brown A (2009) ‘Multiple organizational identities and legitimacy: The rhetoric of police websites’ Human Relations 62 pp 1829-1856 Sinclair A. (1995) ‘The chameleon of accountability: forms and discourses’ Accounting, Organizations and Society 20 pp. 219-237 Strathern M (2000) ‘The tyranny of transparency.’ British Educational Research Journal 26 pp 309-321 Tsoukas H (1997) ‘The tyranny of light’ Futures 29 pp. 827-843 Weiss CH (1972) Evaluation Research.  Methods of Assessing Program Effectiveness. Englewood Cliffs, NJ: Prentice Hall Weiss, C. H. (1973) Organizational constraints on evaluation research. Evaluation 1 pp 49-55 Weiss, CH (1975). Evaluation research in the political context. In EL Struening and M Guttentag (eds), Handbook of Evaluation Research, vol. 1.  Beverley Hills: Sage 13-26   

Accountability, evaluation and the role of evidence for charitable funders

http://discovery.ucl.ac.uk/1473786/3/Hinds_BAM2.pdf

Accountability, evaluation and the role of evidence for charitable funders

Abstract

Similar works

Full text

Available Versions

UCL Discovery