This short paper addresses a number of the key themes of the 12th
International CAA conference with particular regard to evaluation, innovation
and strategic developments. It is based on the current findings and
experiences from two interrelated CAA projects underway at the University of
Bradford: “Embedded support processes for e-Assessment” and “Integrating
thin client systems and smart card technology to provide flexible, accessible
and secure e-Assessment”. These two projects, along with specific aims in the University’s Learning,
Teaching and Assessment Strategy and other projects conducted as part the
institution’s e-Strategy, aim to establish an effective and efficient system for
online summative and formative assessment at the University of Bradford that
will meet the needs of a Higher Education Institution in the 21st century. This
is very much a work in progress, and it is hoped that this work will be written
up as a long paper for a future CAA conference

John Dermo (7194257)

Sara Eyre (7194434)

English

Loughborough University Institutional Repository

   This item was submitted to Loughborough’s Institutional Repository (https://dspace.lboro.ac.uk/) by the author and is made available under the following Creative Commons Licence conditions.      For the full text of this licence, please go to: http://creativecommons.org/licenses/by-nc-nd/2.5/  SECURE, RELIABLE AND EFFECTIVE INSTITUTION-WIDE E-ASSESSMENT: PAVING THE WAY FOR NEW TECHNOLOGIES   John Dermo and Sara Eyre Secure, Reliable and Effective Institution-Wide E-Assessment: Paving the Way for New Technologies. John Dermo and Sara Eyre, Learner Support Services, University of Bradford,  Richmond Road, Bradford BD7 1DP, UK j.dermo@bradford.ac.uk Introduction This short paper addresses a number of the key themes of the 12th International CAA conference with particular regard to evaluation, innovation and strategic developments. It is based on the current findings and experiences from two interrelated CAA projects underway at the University of Bradford: “Embedded support processes for e-Assessment” and “Integrating thin client systems and smart card technology to provide flexible, accessible and secure e-Assessment”.  These two projects, along with specific aims in the University’s Learning, Teaching and Assessment Strategy and other projects conducted as part the institution’s e-Strategy, aim to establish an effective and efficient system for online summative and formative assessment at the University of Bradford that will meet the needs of a Higher Education Institution in the 21st century. This is very much a work in progress, and it is hoped that this work will be written up as a long paper for a future CAA conference. Background and Context The UK National Student Survey has suggested that significant levels of student dissatisfaction with traditional paper-based assessment methods and feedback mechanisms exist across the HE sector, and the HEFCE Strategy for e-Learning has among its objectives for pedagogy, curriculum design and development to “encourage the use of technology to enable electronic assessment [and] produce and disseminate models of good e-learning practice including assessment.” (HEFCE, 2005:11)  At the University of Bradford these issues are now part of a debate which will lead to more comprehensive policy development regarding assessment and feedback. Based on a series of pilots, we believe that innovative e-assessment in general and computer-assisted assessment in particular can make an important contribution. CAA has great potential to reduce the assessment load and to create powerful and innovative methods of assessment in HE (Brown et al., 1997; Bull and McKenna, 2004). The University of Bradford has invested in the underpinning technology for CAA (both hardware and software) which should support the expanding demand for this service over the next few years as part of its comprehensive five year e-strategy. However, we must do more than simply provide the infrastructure and expect the rest to follow; as is clear from the literature, this must be accompanied by an institutional commitment (Zakrzewski, 1999: 172) and dedicated training and support programmes (Sim et al, 2004: 224). At the institutional (top-down) level, computer assisted assessment is closely related to issues of institutional culture, change management, quality assurance, staff development, estates planning; at the individual (bottom-up) level it affects module and course structure, training needs, workload and evolving work practices (JISC, 2007). Clearly, there are some major challenges to be faced here. The Two Projects Since May 2007 the University of Bradford has been running two related projects into CAA, the focus of which is to develop the necessary technical, administrative and pedagogic support systems to ensure reliable and secure large-scale implementation of CAA.  The first project is part of the HEA e-learning Pathfinder Programme, funded by the Higher Education Funding Council for England (HEFCE). The focus of this project is to develop the necessary administrative and support systems for CAA using Questionmark Perception. This includes drawing up rules and regulations for formative and summative e-assessment, developing models of best practice, setting up an integrated programme of staff development and establishing communication links across the institution as well as with other institutions.  This is complemented and sustained by a JISC Institutional Exemplar project. A new 100+ seater e-assessment cluster is being constructed using Sunray thin client technology instead of conventional PC-based assessment systems. Linked with smart card identification, this should be able to deliver an innovative system for summative e-assessment that is scalable and economical. At the same time it should be suitable for large classes but yet easily manageable; also it needs to be responsive to academic staff needs and flexible enough to deal with a wide range of e-assessment types that can be delivered without needing elaborate installation procedures or additional software to control for security and access. To satisfy institutional policies and strategies, we also require an assessment system that is inclusive and personalisable for widening participation, accessible for our disabled students and also sustainable and environmentally friendly. It will also have secure integration with other key University systems (e.g. student records, module registration, room bookings and examination scheduling), and when not in use for summative purposes will be able to deliver aspects of formative assessment as well as standard network services.  These are clearly challenging requirements, and are issues facing any HE institution that is seriously considering the use of online assessment on a large scale.  As one would have expected, several challenges have arisen in the process of these projects. Principal among these have been issues related to practicality, security and scalability when contemplating online assessments to cohorts of possibly 300+ students. Alongside this, a number of operational and administrative support issues have arisen, especially regarding rules and regulations, the role of the examinations office, invigilators and room bookings as well as staff development. It has been necessary to start to re-think institutional policies towards assessment, which were originally drawn up with paper-based examinations in mind. As a result of these projects the institution is now in a position to consider moving away from large scale use of scanned OMR answer sheets, towards on screen assessment. The Pathfinder project is now nearing completion and the Institutional Exemplar project is underway in its pilot stage.  Factors Affecting Implementation: Student Perceptions of CAA The technical, administrative and support solutions being investigated in these projects are both exciting and innovative, and initial findings are very positive. In order to give the project team some hard data on which to proceed, and as part of the evaluation of the Pathfinder project, a survey of student perceptions of online assessment was carried out, and we are now in a position to report on current student attitudes and opinions. After consultation with teaching and learning experts within and outside the institution, six main dimensions for enquiry were agreed upon: affective factors, pedagogical benefits, validity, security, practicality and reliability. Suitable indicators were created for each of these six dimensions, and a multiple item measure drawn up, consisting of a 30-item Likert scale questionnaire. This questionnaire was delivered to a sample of 130 students who have taken online assessments (either formative, summative or both) during this academic year. A small amount of demographic data was also collected to enable us to break down the findings by gender, age and academic subject, and opportunity was given for students to give open-ended responses where appropriate. Initial analysis of the data has already provided us with information about a number of factors which do have direct impact on our implementation of CAA.  It can be demonstrated that there are significant numbers of students who are now accustomed to using technology in all aspects of their everyday lives, including their studies. This also extends to using computers in their assessed work. For example, many students feel relaxed and comfortable about using computers in exams, do not consider it to add significant stress to the exam experience (see figure 1) and are so used to using computers in their student lives that they even expect online assessment to be used.  Figure 1: Histogram showing student stress levels in online exams. At the same time, there is also a significant minority among our students who do have concerns, especially when it comes to security and practical issues. Whilst it is clear that many students consider that online assessment is an alternative to traditional paper-based methods, there are still genuine worries among our students that need to be addressed in planning and implementation of CAA across the institution. The histogram below (figure 2) suggests that the distribution of student ratings of the practicality of online assessment is quite normal. That is to say, whilst the majority of students are to be found in the mid-portion of the graph, the number of students who have a very positive attitude (ie on the right of the graph) is matched by an equivalent number who have concerns (on the left).  Figure 2: Histogram showing student perceptions of the practicality of online assessment. It should also be pointed out here that this example graph is typical of the responses received across the range of different dimensions mentioned above. There can be no doubt that instructors and e-learning advisors still have some work to do to win over the ‘hearts and minds’ of a proportion of the student body. This is also indicated by the pie chart (see figure 3 below), which shows that whilst slightly more than half the students were positively inclined to the idea of online assessment, and a small number of students took a neutral position, the proportion of students who had negative feelings or strong negative feelings towards computer-assisted assessment was only a little less than half.  Figure 3: Pie chart showing student feelings toward online assessment. Initial analysis of the independent variables in this study reveals that there is not a significant gender difference in the ratings, and perhaps surprisingly, age does not seem to be a significant factor either. Further analysis needs to be carried out to establish whether academic discipline plays a significant role in this. The open-ended responses in the student perceptions survey also revealed some fascinating insights into student attitudes to e-assessment. Here is a small selection of anonymous comments from current undergraduates that might strike a chord with many colleagues working with CAA: “It depends on the exam, not on the method of having the exam.” “All exams should be on computer, as it is easier to read and complete!” “It is a convenient and easy way to take online tests.” “Online assessments are fantastic. I don't know why the Uni doesn't do everything that way.” “Written exams for computer programmers are REALLY naff!” “Participating in them was helpful. You could get marks for filling the question and reading the learning materials, which is helpful. It doesn’t only assess knowledge, but also expands it.” “Working online is far better than paper exams. Plus the environmental benefits of reduced number of exam papers required to be printed.” “Doing exams online is great!” “It would be a lot better to make most assessment online. It’s so much easier.” “It is better to do exams online.” “If a person is smart enough to do a university exam, he/she must be smart enough to point and click.” However, not all the feedback was so positive about online assessment: “Written exams are always the best option.” “I don’t like online exams because they lead to failing of exams.” “The system can crash, that is the main disadvantage.” “Computers are not as reliable as paper based examinations.” “Both paper and online exams are fine for me, but I think I would rather prefer the old-fashioned way.” Factors Affecting Implementation: Institutional challenges and issues The Institutional exemplar project is still at the pilot stage, and much of the work to date has dealt with discussion and consideration of the main issues affecting the implementation of this client technology, from the technical, administrative and support points of view. This has involved lengthy consultation between teams of interested stakeholders as well as discussion with colleagues at other UK HE institutions. Whilst it is certainly clear that there is widespread support among staff for the potential of the technology, we are still at the stage of asking questions rather than being able to answer them. Some major issues that have been raised include: • Ensuring accessible delivery for disabled students  • Managing students who require extra time • Avoiding double booking rooms • Dealing with students who forget smart card or login password • Students whose account is banned • Students on franchised courses  • Room Bookings • Correct student module enrolment in student records • Need to hold initial grades until exam board meet before transfer into student records • Anonymous marking of word processed scripts • Keeping online exam scripts secure • Controlling the length of online exams • Archiving online exams  • Equipment failure part way through an exam • Comfort and stress affecting student exam performance online compared to paper-based Conclusion In addition to the progress made in the two current projects (as outlined above), with sustainability in mind, follow-on projects are already being set up as part of the University of Bradford’s e-strategy to ensure that the progress that has been made to date can be continued into the future.  It is envisaged that this short paper could be worked up to a full paper submission for the next event in 2009, once both main projects are complete.  It is also anticipated and hoped that that this presentation will stimulate discussion on the various challenges facing large-scale e-assessment, especially in the HE sector,  such as security, validity and reliability, and how cutting edge technology may offer possible solutions. References Brown, G., Bull, J. and Pendlebury, M. (1997) Assessing Student Learning in Higher Education. London, Routledge.  Bull, J. and McKenna, C. (2004) Blueprint for Computer-assisted Assessment. London: Routledge.  Higher Education Funding Council for England, HEFCE (2005) HEFCE strategy for e-learning. London: Joint Information Systems Committee, Higher Education Academy  JISC (2007) Effective Practice with e-assessment: an overview of technologies, policies and practice in further and higher education. Joint Information Systems Committee. Available online at: http://www.jisc.ac.uk/media/documents/themes/elearning/effpraceassess.pdf  (accessed 21 January 2008).  Sim, G., Holifield, P. and Brown, M. (2004) “Implementations of computer assisted assessment: lessons from the literature.” ALT-J Vol 12, No. 3. pp. 215-230. Zakrzewski, S. (1999) “A structured approach to mass implementation of computer-based assessment” in Brown, S. and Bull, J. (eds) Computer-assisted assessment in higher education. London: Routledge. pp 171-178. 

Secure, reliable and effective institution-wide e-assessment: paving the way for new technologies

https://repository.lboro.ac.uk/articles/Secure_reliable_and_effective_institution-wide_e-assessment_paving_the_way_for_new_technologies/9489692/files/17115350.pdf

Secure, reliable and effective institution-wide e-assessment: paving the way for new technologies

Abstract

Similar works

Full text

Available Versions

Loughborough University Institutional Repository