The workshop will introduce participants to use a range of tools, methods and approaches to designing and
evaluating Open Educational Resources (OERs). The session will include an overview of the OLnet initiative
which is a new global network of support for researchers, users and producers of OERs. Participants will have a
chance to try out some of the tools that have been developed as part of OLnet and to discuss relevance and
application to their own practice

Brasher, Andrew

Conole, Grainne

Cross, Simon

Culver, Juliette

McAndrew, Patrick

Wilson, Tina

English

Open Research Online (Open)

Open Research OnlineThe Open University’s repository of research publicationsand other research outputsNew methods and approaches to the design andevaluation of open educational resourcesConference or Workshop ItemHow to cite:Conole, Grainne; Wilson, Tina; McAndrew, Patrick; Culver, Juliette; Brasher, Andrew and Cross, Simon (2009). Newmethods and approaches to the design and evaluation of open educational resources. In: EDEN Annual Conference2009, 10-13 Jun 2009, Gdansk, Poland.For guidance on citations see FAQs.c© 2009 The AuthorsVersion: Accepted ManuscriptLink(s) to article on publisher’s website:http://www.eden-online.org/eden.php?menuId=410Copyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyrightowners. For more information on Open Research Online’s data policy on reuse of materials please consult the policiespage.oro.open.ac.ukWORKSHOP ABSTRACT NEW METHODS AND APPROACHES TO THE DESIGN AND EVALUATION OF OERS Gråinne Conole, Tina Wilson, Patrick McAndrew, Juliette Culver, Andrew Brasher and Simon Cross  The Open University, UK, Email: g.c.conole@open.ac.uk Overview The workshop will introduce participants to use a range of tools, methods and approaches to designing and evaluating Open Educational Resources (OERs). The session will include an overview of the OLnet initiative which is a new global network of support for researchers, users and producers of OERs. Participants will have a chance to try out some of the tools that have been developed as part of OLnet and to discuss relevance and application to their own practice.   Background The aims of the Hewlett Foundation's Open Educational Resource (OER) program in 2007 were stated as to: 1. Sponsor High-Quality Open Academic Content; 2. Break Down Barriers to Open Educational Content; and, 3. Encourage People Worldwide to Use Open Educational Resources. The resulting OER movement has been successful in promoting the idea that knowledge is a public good, expanding the aspirations of organizations and individuals to publish OERs. However as yet the potential of OERs to transform practice has not being realised, there is a need for innovative forms of support on the creation and evaluation of OERs, as well as an evolving empirical evidence-base about the effectivness of OERs. To achieve this goal Hewlett have funded a new initiative the Open Learning Network (OLnet); a joint partnership between the Open University UK and the Carnegie Mellon Institute. OLnet aims to extend this cultural shift, to evaluate the impact of OERs on teaching and learning and to facilitate transformative educational practices. This is being achieved by developing the links among the design, use and evaluation of OERs. OLnet provides the infrastructure and build capacity for a community concerned with improving OER design and applying methods for assessing robustness of OERs. Through its activities OLnet encourages adoption of evaluative techniques to provide evidence on the effectiveness of OERs in use. The very openness of OERs means that we cannot be sure how they are being used, what ways people are getting the best out of them, or if they are meeting the original goals of the developers. OER authors are often not in the best position to address these questions, partly because their position inhibits neutrality and partly because of the need for expertise and resources to do the research. The approach of OLnet is to build new structures and activities to nurture the growing pool of OERs and associated services, developing a complementary infrastructure to investigate and report on the issues around OER deployment and evaluation. OLnet takes the opportunity to go beyond isolated, individual views of an OER's effectiveness, by aggregating data, sharing evaluation know-how, and mediating dialogue and debate within the community. Workshop outcomes At the end of the workshop, participants will have: • a good overview of the different approaches to and issues in creating and evaluating OERs • an understanding of the OLnet initiative and how it can be used to support the design and evaluation of OERs • an awareness of the range of resources, tools and methods which are available to support the design and evaluation of OERs – including case studies of good practice, learning object repositories and learning design tools/methods • experience of thinking about the OER effectiveness cycle from different perspectives • had hands on experience of some of the OLnet tools including CompendiumLD for visualising designs, Cloudworks for sharing and discussing the creation and use of OERs and Cohere for the development of complex collaborative co-argumentation and ideas • an understanding of how the OLnet can be applied in their own teaching context.   Who should attend? The workshop will be of interest to teachers and designers wanting to create and use OERs or those with an interest in OERs as an approach. The session will need to be located in a PC lab with Internet connected computers – one computer for every two/three participants is needed. In addition the room should have a data projector facility for the overview sections of the workshop.  Workshop format The session will be highly participative. Delegates will be given plenty of opportunity to discuss the relevance of the workshop material to their own context. The hands on session will give them a chance to use the OLnet tools to create and evaluate their own OERs. Ideally it would be useful if participants could come to the workshop with an idea of a OER or set of OERs they would like to work on, however this is not essential as there will be an opportunity during the workshop to decide what to work on. We will be able to accommodate up to 20 delegates Further information and the vision behind OLnet The OLnet initiative started in January 2009, but builds on two substantive areas of research – the Openlearn initiative (openlearn.open.ac.uk) a repository of OERs developed from the Open University UK’s course materials and the OU Learning Design Initiative (ouldi.open.ac.uk). The tools that participants will use include CompendiumLD (compendiulmld.open.ac.uk), Cloudworks (Cloudworks.ac.uk) and Cohere (cohere.open.ac.uk). Researching Open Educational Resources raises issues in how to address global connections, reuse, design and evaluation of world wide efforts to work with learning resources that are available for free use and alteration. The initial investment from the Hewlett Foundation to seed OERs through its Education Program has raised interest and activity in Open Education and there are now many indications that the continued production and publication of open content is self-sustaining. Open provision of course materials has become a more extended movement with many universities adopting the approach: the OpenCourseWare Consortium now has more than 150 institutional members; in the Multilingual Open Resources for Independent Learning (MORIL) project the open and distance universities are acting together to share resources through open release; the UK's JORUM repository (http://www.jorum.ac.uk) announced in April 2008 that it would offer open educational resources to facilitate sharing and reuse and to “provide a showcase for UK universities and colleges on the international stage”; and, the Open University’s own investment in a continuous process for open content production. These are indications of adoption of an open approach, however the diverse OER projects have not received much research attention to establish how best to move from existing provision to better structures for open operation.  OLnet provides a basis for gathering evidence, developing methodologies, supporting involvement and gain value by aggregating and sharing information through appropriate infrastructure. It offers a community-based solution: a researcher/practitioner network, supported by Web tools, for aggregating, sharing, debating and improving OERs.. In line with research on Community of Practice approaches we aim to build on existing communities and the needs that they have for evidence and investigation, rather than attempt either to create a new community or provide catalogues of existing efforts.  Science depends on community and the chance to discuss, contest and identify the soundness of new ideas. Community approaches can lead to increases from a small base of activity by connecting with enthusiasm and interest. We believe that enthusiasm is evident in the community around open educational resources from researchers, individuals and institutions demonstrated through participation in conferences, contributions and emerging strategies.  Two analogies may help illustrate the need we see for OLnet:  International development: helping the helpers. One way to help developing countries is to get out there in the front lines. That is great — clearly someone has to be doing that. But fragmented, individual efforts can be made more effective by agencies that connect frontline groups with each other to share know-how and build collective intelligence about a region or problem of common concern. The OER movement lacks the feedback loops and socio-technical fabric to help this happen, and this will be one of the key outcomes of OLnet.  A research roadmap: mapping the territory. In more established fields, for example medical research, there is a consensus map of the structure of the field, the major research questions, and the different sub-communities and associated methodologies. It is possible to place oneself on the map, and to contribute and coordinate effort in a well-understood way. In contrast OER research is a relatively young field, which is still being articulated and defined. OLnet will help to facilitate researchers in the area to articulate the scope of the field and typical methodological stances. At various points OLnet will trigger a series of questions and provide mechanisms to enable those in the field to progress answers to those questions. What is the OER research map? What is the OER design process? What does it mean to validate an OER? What are the central challenges that all agree on? OLnet seeks to create a structured 'place' where questions such as these can be debated, , enabling more effective coordination of action around issues and OERs of common interest. Research Questions The driving research question behind OLnet pinpoints what we see as the next evolutionary step in the OER movement, namely: • How can we build a robust evidence base to support and enhance the design, evaluation and use of OERs?  This high level question is refined into three sub-issues: 1. How to improve the process of OER reuse/design, delivery, evaluation and data analysis? 2. How to make the associated design processes and products more easily shared? 3. How to build a socio-technical infrastructure to serve as a collective evolving intelligence for the  community? The project aims to define a framework for answering these questions on an iterative basis by research on the nature of information sharing about OER research and by developing research studies of diverse kinds on improving OERs.  The research studies are threads that feed information into the network of OER developers and users.  Likewise, the discussions in the human network of OER developers and users will feed information back about the kind of research that still needs to be designed and performed.   To initiate these cycles of information exchange we will use technology to support the development of a collective intelligence in a social network designed to become self-sustaining. We see capacity building, practice and research as essentially parallel and interacting strands that build on the base of technology and activities shared among the participants. The identification and sharing of methodologies is a function of OLnet, however these can only be achieved by supporting real activities to ground the research and increase the value of participation. The illustrations later in the proposal show one way in which these aspects can work together. One of our hypotheses is that capacity building and research need to happen at the same time so that they can interact: the sustainability, effectiveness and transfer of OERs require both so that they can bootstrap each other.  This is what has been missing in many other educational capacity building efforts that separate research from practice.  A distinctive aspect of this proposal is to situate our research in what was termed by Donald Stokes as  "Pasteur’s quadrant" where basic and applied science intersect and where the choice of problems and the conduct of research is inspired by societal needs.    A core concept underpinning OLnet is the notion of an OER effectiveness lifecycle. For each stage of the cycle there will be an associated set of tools, resources, etc. available within the OLnet community. Each stage can also generate specific outputs such as a design representation or a new evaluation instrument, which can be put back into the OLnet community for others to use. So for example a user might query an existing OER repository such as OpenLearn as a means of selecting an OER for use. Another user might develop a new survey instrument for evaluating the use of say, science-focused OERs which they then make available to the OLnet community, and yet another user then  use that instrument to evaluate their use of Science OERs. Our central argument is that all too often, the feedback loop that links from evaluation, to data collection, to cumulative design improvement is broken, and that those links should be forged and nurtured. This design cycle is focused on OERs as the objects of interest, with other tools facilitating its transition at different stages, hence the label "OER Effectiveness Cycle" at the component, or sub-system, level.  We want to emphasis that we see the cycle as reflexive: OERs are not the only objects of interest in an epistemic community of practice. Any of the design representations or other artifacts generated, or used to analyze, OER design can themselves become "social objects", that is, artifacts shared, deployed, evaluated and improved on by the community. Thus, the very infrastructure that we use to accomplish this process – OLnet – becomes the object of reflection, hence the same effectiveness cycle applies to OLnet itself at the system level.  The second diagram shows the emergence of these specialist communities, whose focus is a particular node or arc in the OER or OLnet cycle. Five OER effectiveness cycles are shown. It shows how OLnet provides a facilitative infrastructure and network to enable connections to be made across different sub-project or activities, so that outputs from one activity can be taken up and reused by another activity. So for example a design representation in one cycle can be picked up and used as a starting point for a different OER cycle, or evaluation findings on the use of one OER can be used to inform and shape the design of a different OER. Examples of these different connections are shown by the arrows between OER cycles in the diagram. In addition to this transfer of knowledge and outputs between OER cycles, OLnet also aggregates outputs in different ways, for example by providing a means to group people and sub-communities in different ways, aggregating tools and resources, collating designs, evaluations and case studies and performing a range of relevant meta-synthesis studies. This will ensure that the sum is greater than the parts and that maximum benefit is derived at the level of individual OER cycles and the overall OLnet. Presenters Grainne Conole is professor of e-learning, she has extensive experience across the e-learning domain and has published over 300 articles and presented widely at conferences in the field. More information can be found at www.e4innovation.com. Patrick McAndrew was research and evaluation for the Hewlett funded openlearn initiative and is the principle investigator for OLnet. He has extensive expertise in the field and has published and presented extensively. The remaining authors are members of the OLnet team who have a range of skills on the research and development of OERs.  

New methods and approaches to the design and evaluation of open educational resources

Abstract

Similar works

Full text

Available Versions

Open Research Online (Open)

Leicester Research Archive