Research impact evaluation, a wider context: Findings from a research impact pilot

In the face of increasing pressure to demonstrate the socio-economic impact of funded research, whether it is funded directly by research councils or indirectly by governmental research block grants, institutions have to tackle the complexity of understanding, tracking, collecting, and analysing the impact of all their research activities. This paper attempts to encapsulate the wider context of research impact by delineating a broad definition of what might be classified as impact. It also suggests a number of different dimensions that can help in the development of a systematic research impact assessment framework. The paper then proceeds to indicate how boundaries and criteria around the definition of impact and these dimensions can be used to refine the impact assessment framework in order to focus on the objectives of the assessor. A pilot project, run at Brunel University, was used to test the validity of the approach and possible consequences. A tool specifically developed for the pilot, the Brunel Research Impact Device for Evaluation (BRIDE), is used for the analysis of research impact collected during the pilot. The paper reports on the findings of the analysis produced by BRIDE and confirms how a number of areas might be greatly affected by the boundaries set on definition and dimensions of research impact. The pilot project shows that useful information on impacts can be generated and it also provides a way to identify areas of work from each unit of assessment for which it would be worth developing narrative case studies. The pilot project has illustrated that it is feasible to make progress in terms of assessing impact, but that there are many difficulties to be addressed before impact assessment can be incorporated into a system of assessing the impact from the university sector as a whole. The paper concludes with an institutional perspective of the value of the approach and highlights possible applications. It also confirms the intention to expand the pilot and introduce new lines of investigation

Measuring the impact of higher education libraries: the LIRG/SCONUL Impact Implementation Initiative

The Library and Information Research Group (LIRG) and the Society of College, National, and University Libraries (SCONUL) are working together on a joint initiative looking at the impact of higher education libraries on learning, teaching, and research. Twenty-two libraries have been involved in assessing the impact of particular services or new innovations in their institutions. This article considers why measuring impact is important, describes the background to the Impact Initiative and what led to it being set up, identifies the institutions that are involved and their chosen topics, outlines what is involved for the participating institutions, and reviews progress to date

Comparing alignment factors in SMEs and large organizations: a planning integration perspective

Measurement of the alignment between business strategies and information systems (IS) has demonstrated positive impact for the organizational performance. The factors that have proved relevant when assessing the maturity level of alignment are: communication, competency/value measurement, governance, partnership, architecture & scope, and skills. Existing research, however, has focused on the assessment of these factors on large organizations and has barely explored their impact on Small and Medium Enterprises (SME). This paper uses the data provided by 127 participants from large and SMEs in order to identify whether the aforementioned factors are also relevant for assessing the level of alignment maturity in SMEs. The results from this research suggest that there are not significant differences between large organizations and SMEs when assessing those factors. In addition, this research also explored the relation between different planning integration of alignment (independent, sequential and simultaneous) in order to measure the perceived relevance of the factors. The results suggest that the planning integration identified on SMEs and large organizations has a positive correlation on how these factors are ranked. For both SMEs and large organizations where the formulation is simultaneous, the relevance of the factors is higher perceived than it is for those where the formulation is independent or sequential

Quality criteria in educational research: is beauty more important than popularity?

Presentation Contents: HISTORICAL CONTEXT: The poor quality of educational research THE DISCUSSION FRAMEWORK: Assessing Quality in Educational Research CRITIQUES: Is assessing the quality of educational research possible or necessary? During the past decade, as national governments more particularly in the USA and the UK have scrutinised more closely the cost effectiveness and impact of research funding within higher education, they have become critical of the overall quality of educational research, in terms of its scientific rigour, its utility for practitioners and the manner in which it is assessed. This paper addresses the reasons why the quality of educational research was questioned and then examines the discussion framework for assessing quality which emerged. Following from this, the merits of possible internal and external criteria for the worth of educational research are considered. The links between these criterial sets and the function and purpose of qualitative and quantitative approaches to educational research are considered. Finally the question of whether educational research is an art or a science is addressed. RESEARCH: into education – are medical and economic models appropriate? EDUCATION: Is it an art or a science (or what)

The LIRG/SCONUL Impact Initiative: assessing the impact of HE libraries on learning, teaching, and research

The LIRG/SCONUL Impact Initiative took place between July 2003 and December 2005. Twenty-two higher education institutions in the UK attempted to measure the impact of their services on learning, teaching, and research. Within the context of the programme, each institution investigated the impact of a new innovation. This paper provides a final overview of the two phases of the Impact Initiative and highlights some of the findings. Measuring impact is not easy but there are significant benefits for the profile and development of academic libraries in trying to do so. It provides guidance for libraries on assessing impact drawing upon the experience of the Impact Initiative

Synthesis report of workshop on assessing the impact of policy-oriented social science research in Scheveningen, The Netherlands, November 12-13 2001

Economists have engaged for some time in developing methodologies for assessing the economic impact of agricultural research and in undertaking empirical studies to measure this impact. In recent years, they have documented more than 1,800 estimates of rates of return to agricultural research. Economists have paid little attention, however, to how to evaluate the impact of social science research. A symposium conducted by IFPRI in 1997 was one of the first attempts to address this knowledge gap. In November 2001, the Netherlands Ministry of Foreign Affairs and IFPRI brought together a group of researchers to follow up on the earlier symposium. Their conclusions fell into two broad categories: how to measure or value the economic impact of policy-oriented social science research and how to enhance the effectiveness of such research in policymaking environments. A number of lessons emerged from the workshop for donors, governments, and researchers about how to enhance the effectiveness of policy-oriented social science research....Because much remains to be learned about evaluating the impact of policy-oriented social science research, the workshop participants concluded that IFPRI should take the lead in developing a consortium to help improve interdisciplinary methods of assessing impact. The consortium would consist of institutions, donors, and individuals and would work in partnership with developing countries. Institutions could learn from each other about best practices and in the process exploit synergies, thereby increasing effectiveness and reducing the costs of what is an expensive undertaking.Agricultural research Economic aspects. ,Evaluation. ,Rate of return. ,Social sciences Research Evaluation. ,Research Economic aspects Congresses. ,

Risk factors affecting the ability for earned value management to accurately assess the performance of infrastructure projects in Australia

Purpose – The purpose of this paper is to investigate a set of risk-related factors influencing the earned value management (EVM) concept as an assessment technique in evaluating the progress of modern sustainable infrastructure construction projects. Design/methodology/approach – A qualitative research approach has been adopted for identifying risk-related factors influencing EVM concept from a literature review and through interviewing industry personnel, followed by an inductive process to form sets of key factors and their measuring items. Findings – EVM is a common method for assessing project performance. A weakness of this approach is that EVM assessment in its current form does not measure the impact of a number of project performance factors that result from the complexity of modern infrastructure construction projects, and thus does not accurately assess their impact in this performance. This paper discusses and explains a range of potential risk factors to evaluating project performance such as sustainability, stakeholder requirements, communication, procurement strategy, weather, experience of staff, site condition, design issues, financial risk, subcontractor, government requirements and material. In addition, their measuring items were identified. Practical implications – This research assists projects managers to improve the evaluation process of infrastructure construction performance by incorporating a range of factors likely to impact on that performance and which are not included in current EVM calculations. Originality/value – This research addresses the need to include in the EVM calculation a range of risk factors affecting the performance of infrastructure projects in Australia and therefore makes this calculation a more reliable tool for assessing project performance

Measuring Impact: The Art, Science and Mystery of Nonprofit News

This report seeks to answer the two-pronged question, "What is 'impact,' and how can it be measured consistently across nonprofit newsrooms?" A review of recent, relevant literature and our informal conversations with experts in the field reveal growing ambitions toward the goal of developing a common framework for assessing journalism's impact, yet few definitive conclusions about how exactly to reach that framework. This is especially the case when journalism's "impact" is defined by its ultimate social outcomes -- not merely the familiar metrics of audience reach and website traffic. As with all journalism, the frame defines the story, and audience is all-important. Defining "impact" as a social outcome proves a complicated proposition that generally evolves according to the constituency attempting to define it. Because various stakeholders have their own reasons for wanting to measure the impact of news, understanding those interests is an essential step in crafting measurement tools and interpreting the metrics they produce. Limitations of impact assessment arise from several sources: the assumptions invariably made about the product and its outcome; the divergent and overlapping categories into which nonprofit journalism falls in the digital age; and the intractable problem of attempting to quantify "quality." These formidable challenges, though, don't seem to deter people from posing and attempting to find answers to the impact question. Various models for assessing impact are continually being tinkered with, and lessons from similar efforts in other fields offer useful insight for this journalistic endeavor. And past research has pointed to specific needs and suggestions for ways to advance the effort. From all of this collective wisdom, several principles emerge as the cornerstones upon which to build a common framework for impact assessment

F1000 recommendations as a new data source for research evaluation: A comparison with citations

F1000 is a post-publication peer review service for biological and medical research. F1000 aims to recommend important publications in the biomedical literature, and from this perspective F1000 could be an interesting tool for research evaluation. By linking the complete database of F1000 recommendations to the Web of Science bibliographic database, we are able to make a comprehensive comparison between F1000 recommendations and citations. We find that about 2% of the publications in the biomedical literature receive at least one F1000 recommendation. Recommended publications on average receive 1.30 recommendations, and over 90% of the recommendations are given within half a year after a publication has appeared. There turns out to be a clear correlation between F1000 recommendations and citations. However, the correlation is relatively weak, at least weaker than the correlation between journal impact and citations. More research is needed to identify the main reasons for differences between recommendations and citations in assessing the impact of publications