This paper describes work in progress in the development of a short online
course for university teaching staff to encourage and assist them in their use
of e-assessment. Two important criteria for the design of the course were that
it should be felt to be enjoyable and that its content should include a particular
focus on giving feedback

Christina Mainka (7193852)

Gill Harrison (7193855)

Mark de Groot (5517959)

English

Loughborough University Institutional Repository

   This item was submitted to Loughborough’s Institutional Repository (https://dspace.lboro.ac.uk/) by the author and is made available under the following Creative Commons Licence conditions.      For the full text of this licence, please go to: http://creativecommons.org/licenses/by-nc-nd/2.5/  ENJOYABLE ONLINE STAFF DEVELOPMENT IN E-ASSESSMENT WITH FEEDBACK   Christina Mainka, Gill Harrison and Mark de Groot Enjoyable Online Staff Development in E-Assessment with Feedback Christina Mainka Educational Development Napier University Craighouse Campus Edinburgh EH10 5GL c.mainka@napier.ac.uk  Gill Harrison Technology Enhanced Learning Team  Leeds Metropolitan University Old Broadcasting House  Leeds LS2 9EN  g.harrison@leedsmet.ac.uk  Mark de Groot Innovation North Leeds Metropolitan University Civic Quarter Leeds LS1 3HE m.degroot@leedsmet.ac.uk Abstract This paper describes work in progress in the development of a short online course for university teaching staff to encourage and assist them in their use of e-assessment. Two important criteria for the design of the course were that it should be felt to be enjoyable and that its content should include a particular focus on giving feedback. Introduction In a collaboration between Napier University and Leeds Metropolitan University, an online course has been developed to introduce university teaching staff at Leeds Metropolitan to the effective and meaningful use of e-assessment. The aim of the course is both to teach the practical aspects of creating and delivering e-assessment within the university’s Virtual Learning Environment and to explore and discuss the pedagogic aspects. It builds on an existing online staff development course at Napier but places a much stronger emphasis on the use of feedback. In addition the Leeds Metropolitan approach pays particular attention to a more engaging instructional design together with the introduction of an appealing course theme that is reflected throughout in activities thought to capture and maintain participants’ interest more effectively. Recognising the difficulties that staff in Higher Education face in attending staff development activities at a particular time and place, the course runs online over five days with the aim of taking only a moderate amount of time each day. While closely aligned to the learning outcomes, the activities and support materials are chosen to be as engaging and enjoyable as possible. The need for staff development – online  As the development of scholarship in education is increasingly seen as important at Higher and Further Education institutions, staff development plays a key role in helping to attain higher teaching standards. Introducing staff to the use of technology to enhance student learning, including the use of e-assessment, is for most institutions a major consideration in this area. Technology improves access to learning not only for students but also for staff; it increases access to and availability of staff development opportunities and can transform the way educators learn. In business and industry technology-supported training has continued to replace traditional face to face provision ever since the appropriate technologies became available (Berge, 2001). The benefits of online staff development, especially in an area such as e-assessment that is inherently technology-related, include:  • more flexible professional learning provision • increased access to professional development opportunities • an embedding of staff development into technology enhanced teaching practice • potential for cross-institutional collaborative professional development programmes Original online course in e-assessment at Napier An online course to teach university staff about e-assessment from both the practical and pedagogic points of view has been running at Napier University for some time.  Pedagogic underpinning Dialogue is recognised to be central to all learning (Mayes, 1997). Until there is a two-way dialogue, either between peers or tutor and peers, learning may not take place; this includes learning within a professional development context. This social dimension of learning is not always sufficiently considered in online course design in general. While self-study online staff developmental resources are widely available (see for example JISC, 2005; Chico, 2006), it is only where tutor-supported online training allows opportunities for academics to engage with one another and not merely with online tools and content that staff development provision can be said to have progressed to adopt the socio-constructivist theory increasingly accepted in education today (Vygotsky, 1986). Didactic teacher-centred delivery of information tends to fail for academics as much as for their students.  For these reasons, the course was designed to have intensive interactive support from its online tutors and considerable peer interaction, by way of discussion forums and other collaborative activities as shown below. Structure The course includes as important features  • the taking of a test that illustrates different available question formats • the taking of a test containing badly designed questions • discussions, involving both tutors and participants, to explore pedagogy including the design of questions • training materials on video (primarily screen captures with audio commentary) • the creation of a test by each participant • the taking of one another’s tests and subsequent discussion about them  A theme of policing, crime and punishment provides a context for the tests to illustrate question format and bad question design. Resource implications This is undeniably a time intensive approach. Vries et al (2005) report a surge in process-related support compared to content-related support in the transition to online delivery of courses, and this is confirmed by JISC reports (Brown, 2002). At Napier, the staff development team have found the five day online assessment course highly time intensive, and have opted to run as an occasional alternative a three-hour face to face hands-on workshop in order to save facilitator time.   This would seem a reminder that if universities are to be successful in their move towards course provision that is more flexible and student-centred (or in this case staff-as-student-centred) and that goes beyond the electronic availability of course content, then the appropriate resourcing of its staff development support units is paramount. Adaptation of the original course Pedagogy The pedagogic principles followed in the design of the original course were adhered to in the adaptation. Support from all three tutors, from the two institutions, as course facilitators was planned for the first delivery in June 2008. Theme At Leeds Metropolitan University, a number of initiatives have titles related to water, for example the designation of professors as “Running Stream Professors”. A previous collaboration with Napier has resulted in the adoption and adaptation of an online staff development course (Napier, 2007), designed to “expose enrolled participants to a range of educational technologies, to engage them in online activities ...... thus empowering each to identify the potential for technology in teaching and learning, and in acquiring relevant skills” (Mainka, 2007). The Leeds Metropolitan version of this previous course has been entitled “Immersion”, and has aimed to familiarise participants with the VLE as well as to introduce them to the potential of technology enhanced learning.  To continue the watery theme, it was decided to use water sports in the new course, and questions were created to do with swimming, surfing etc. The course title “Surf’s Up” was chosen, Surf standing for Student assessment Using Rich Feedback, and suitable icons were used (see Figure 1).   Figure 1 – Course Home Page Support materials Reference sources remained largely the same though with a greater emphasis on feedback, but training in the use of the assessment tools of the Virtual Learning Environment (Blackboard Vista) was provided through materials developed by Leeds Metropolitan University itself. These were a mixture of step-by-step written guides and screen capture videos. The main video resource contained sections on creating questions, creating assessments using those questions, and setting release dates etc., with each section being viewable separately. A question and answer format was adopted, with demonstration by an expert in response to questions by a novice (who was in fact a student working on placement in the university). This format was felt to be more lively and interesting than a simple demonstration with accompanying commentary. Structure For each day, a series of steps was provided, using Blackboard Vista’s Learning Module structure that allows a Table of Contents to contain among its steps not merely files, in HTML or other formats, but direct links to any kind of activity such as a discussion or a test (see Figure 2). This was a somewhat simplified format compared to that used in the original Napier course, and followed closely the format used in the Immersion course. Immersion, though also based on one of Napier’s courses, had a simpler structure that evolved over time in response to participants’ feedback.   Figure 2 – Typical Table of Contents The new course was supposed to take no more than 45 minutes per day of the participants’ time, and the intention of providing a very clear structure was to try to achieve this target. The structure required participants to create questions on Tuesday, put them together into an assessment on Wednesday, and access one another’s assessments on Thursday. This created certain logistical problems with regard to the participant’s role as student or builder in a course within the VLE, and the impossibility of being both at the same time. It was decided to create a second course, and to give all participants the rights to build assessments within it, and then for the course facilitators to transfer the built assessments over to the original course. The first pilot run in June will show whether this approach is tenable or needs re-thinking.  Emphasis on feedback The importance of providing feedback to students is an increasingly prominent topic in Higher Education, not least because “evidence continues to emerge of pervasive student concerns about the provision of feedback in an era of mass higher education” (Hounsell, 2007). At Leeds Metropolitan University, an initiative was launched in 2007 to raise the profile of feedback and to encourage students’ awareness and use of it by supplying them with a tailor-made guide to using feedback (Race, 2007).  The role of e-assessment in feedback can be vital. In Chapter 10 of Race (2005), entitled “Putting the Learning into e-learning”, he comments on the immediacy of online feedback: “One of the advantages of e-learning is that immediate on-screen feedback can appear every time learners make a decision, or select an option, or enter a number and so on.” and the potential for individualising feedback, when students give wrong answers: “Multiple-choice question formats are particularly useful here, as they allow different learners making different mistakes each to receive individual feedback on their attempts at such questions.” Thus a particular focus of the course design was the provision of feedback, using a range of available features in the VLE, as part of the online assessment process, and examples of this were incorporated in the tests provided to the participants. The multiple-choice questions within these tests had carefully-chosen distracters with tailored feedback attached to each. For example, one multiple-choice question was: Henri La Mothe holds the record for the highest shallow dive. He dived 12 m (40 ft) from the Flatiron Building in NYC into 32 cm (12.5 in) of water.  What was his kinetic energy when he hit the water? Assume his mass to be 70 kg. A student choosing the response 28kJ would receive the response: That is not correct but you have applied the right formula. Remember to use metric units (metres not ft) when calculating potential energy in kJ. For other types of question, general feedback was shown, with references where necessary. Two examples of feedback, which give a further flavour of the types of question within the course, are: (Feedback on a “fill in the blanks” question about a medical condition that surfers may suffer from):  For more details on Surfer's Ear please consult the website:......... (Feedback on a “calculation” question about the speed with which a shark could approach a surfer)  You should have taken the shark's length x 10 then x 3600 then divided by 1000. Just a pure number, with one figure after the decimal point, must be entered, e.g. 81.4 or 98.0. There was also consideration of the use of Assertion-Reason questions (see CAA Centre, 2002) and how to give feedback in more sophisticated ways. The participants were asked to create feedback in the test that they built, to peer review one another’s tests with respect to feedback provision (as well as question design), and to engage in a substantial discussion about the provision of feedback. Enjoyability University teachers are busy people, and giving up their time to study an online course of this kind is no small commitment. A minimalist, focused and straightforward presentation of course materials was intended to make this commitment of time as immediately productive as possible, ensuring staff engagement so that they would stay with the course and achieve the learning outcomes. Just as with students, retention was vital. Thus an element of fun seemed also to be an obvious requirement for the course. The use of photos and images related to the course theme ensured a lively and stimulating web design (see Figure 1 above). Images and video were also used within questions (Figure 3 shows an example). Headings, titles and annotations were chosen appropriately and questions (as can be seen from the examples in the Feedback section above) were designed not only to demonstrate available question types and opportunities for feedback but also to ‘immerse’ the participant as fully as possible into the chosen theme, which had been carefully selected to capture an interdisciplinary group’s attention as well as possible.  Figure 3: Image used within a question Collaborative working Three staff members, two from Leeds Metropolitan and one from Napier, collaborated on this course. The project was supported by money from the Teaching Quality Enhancement Fund, a financial resource provided by the Higher Education Funding Council for England (HEFCE) to support teaching and learning strategies within Higher Education institutions. Some support was also given by the Centre for Excellence in Higher Education (Active Learning in Computing) (Durham 2006; Leeds Metropolitan University 2007). Contact was largely electronic, through email and via a Blackboard Vista course. Within this course, discussions and occasionally live chat were utilised for communication, documents and proposals were exchanged, and prototypes of the course including the assessments were developed. There were also a small number of face-to-face meetings in Leeds.  Proposed Evaluation The Immersion course currently running at Leeds Metropolitan University contains an evaluation questionnaire to be completed by participants at the end of the course. There has generally been a high response rate to this, averaging above 70%, and several changes have been made to the course as a result of suggestions and ideas arising from this evaluation. It is proposed that a similar questionnaire will be included in the Surf’s Up course, to gain the reactions of the participants. It would also be desirable to evaluate the effectiveness of this staff development exercise, in line with current thinking on this topic (see for example Northumbria, 2006) by following up a sample of the staff at a later date, to investigate the impact that the course has had on their subsequent use of e-assessment.  Conclusions and future work The current state is very much that of a work in progress, and the course is now nearing its final stage of design and implementation. Comments have been sought from some 30 staff across the university who have been shown the current design, and their feedback has been taken into account during the finalising of the course. A first pilot presentation is planned for June 2008. The Immersion course previously referred to is offered once a month, and the Surf’s Up course will, if there is sufficient demand, be offered with the same frequency. One faculty within Leeds Metropolitan has already expressed an interest in having all staff with responsibility for leading a module take the course. Future reports will discuss the results arising from the proposed evaluation described in the previous section, as well as the success or otherwise of the peer interactions, including the peer review of tests.  It is hoped that this course will make a useful contribution to the meeting of staff needs in relation to their use of e-assessment and feedback. References Berge, Z. (2001) The context of distance training: Predicting change. In Berge, Z. (ed), Sustaining distance training: Integrating learning technologies into the fabric of enterprise. San Francisco: Jossey-Bass. Brown S (2002) TechLearn Senior Management Briefing on Online Tutoring, JISC Technologies Centre  http://www.jisc.ac.uk/uploaded_documents/ebriefingshort.doc  (accessed 7th May, 2008)  CAA Centre (2002) Resources: Assertion-Reason Questions http://www.caacentre.ac.uk/resources/objective_tests/assertion.shtml (accessed 12th May 2008) California State University at Chico (2003) Rubric for Online Instruction  http://www.csuchico.edu/celt/roi/  (accessed 4 May 2008)  Durham University (2006) Active Learning in Computing http://www.dur.ac.uk/alic/ (accessed 12th May 2008) Hounsell, D. (2007) Towards more sustainable feedback to students. In Boud, D. and Falchikov, N. (eds) Rethinking Assessment in Higher Education: Learning for the longer term. New York, Routledge. JISC (2005) X4L Staff Development Resources http://www.x4l.org/exemplars.htm (accessed 17th May 2008)  Leeds Metropolitan University (2007) Active Learning in Computing http://www.leedsmet.ac.uk/inn/alic (accessed 12th May2008) Mainka, C. (2007) Putting staff first in staff development for the effective use of technology in teaching. British Journal of Educational Technology 38 (1) , pp. 158–160. Mayes, J.T. (1997) Pedagogy, Lifelong Learning and ICT http://www.ipm.ucl.ac.be/ChaireIBM/Mayes.pdf (accessed 17th May 2008)  Napier University (2007) Online Taster http://www2.napier.ac.uk/transform/Online_Taster.pdf (accessed 11th May 2008) Northumbria University (2006) Evaluation of Staff Training and Development: Guidance Notes http://northumbria.ac.uk/sd/central/hr/std/td_eval/ (accessed 12th May 2008) Race, P. (2005) Making Learning Happen. London, Sage Publications Ltd. Race, P. (2007) Feedback http://www.leedsmet.ac.uk/ALTre-source/Feedback_-_longer_version.doc (accessed 11th May 2008) Vygotsky, L. (1986) Thought and language. Cambridge Mass, MIT Press.  Vries, F.J. de, Kester, L., Sloep, P., Van Rosmalen, P. Pannekeet, K. and Koper, R. (2005) Identification of critical time-consuming student support activities in elearning. ALT-J, Research in Learning Technology, 13(3), pp.219-229.   

Enjoyable online staff development in  e-assessment with feedback

https://repository.lboro.ac.uk/articles/Enjoyable_online_staff_development_in_e-assessment_with_feedback/9488669/files/17114327.pdf

Enjoyable online staff development in e-assessment with feedback

Abstract

Similar works

Full text

Available Versions

Loughborough University Institutional Repository