and detailed analysis of how resources impact learning. Simultaneously, it is inherent in the concept of OER that resources are to be used in different contexts and in varied ways. Encouraging institutions to appropriate resources to their needs is important to the effective application of OER. A major challenge is to develop research practices that can effectively evaluate this heterogeneous usage and impact.
In this paper we consider experiences of assessing impact from Bridge to Success: an initiative in which online courses in mathematics and study skills, originally developed at the Open University UK, were remixed for a US audience and made available as OER. The released materials were subsequently used in more than 16 US-based institutions including colleges, universities, high schools, and projects to help the long-term unemployed. The materials were used in a variety of ways. In some cases, units of these open online courses were used as supplementary materials, whilst elsewhere the courses were used in drop-in labs and face-to-face sessions. In several contexts, the materials were provided to a cohort of students to help them prepare for formal assessments. In others, underachieving students were specifically targeted and advised to use the resources. This variety of pilot contexts provided a challenging dimension to understanding the value of the OER intervention to the learner.
Understanding use is seen as a key component in the further development of OER (Atkins, Seely Brown & Hammond, 2007). However approaches to this where institutions have been encouraged to contextualise materials to their own needs are currently limited. We explore different approaches to evaluating this type of impact across contexts. Analysing data on frequency and type of access, learning gains, institutional enrolment and persistence of students are critical areas for this type of research, and require an understanding of institutional and learner characteristics, in addition to the varied ways in which the materials are facilitated. Openness creates challenges for researchers to collect and link data about all kinds of use. For example, institutions collect information on enrolment, assessment and retention in different formats: this data can be sensitive or difficult to share. In addition, to truly understand impact, researchers need to be able to distinguish different forms of resource use and connect this to specific user groups.
The paper evaluates our experiences by defining the types of impact data required by a range of stakeholders, and the achievements and challenges in delivering this information as part of the Bridge to Success initiative. A wide range of quantitative and qualitative methods were used in researching the impact of the initiative, but limitations exist in our current ability to understand use and to link this to the requirements of the different contexts of use. Based on both the successes and failures we had in monitoring impact we outline suggested approaches towards an effective general model for assessing OER impact
