The practice of writing effective exam questions has attracted plenty of interest in the literature. However, the process of exam setting and moderation has received little attention. Thus, the aim of this manuscript is to describe our approach in exam setting and moderation for a transnational education (TNE) programme between the University of Glasgow and the University of Electronic Science and Technology of China (UESTC). Each year, Glasgow University staff members prepare main and resit exam papers for 32 courses in the electronic and electrical engineering subject group. To ensure a smooth workflow and a relatively quick turnover of high-quality exam papers each year, we describe the best practices in exam setting, as well as our approach in exam moderation using a new moderation form and a new system for uploading exam papers online. This new system was designed to facilitate a solid dialogue between exam setters and examiners, so that all summative assessments are fair, consistent and reliable. These best practices can be used by engineering faculty to make informed decisions regarding the process of exam setting and moderation in a transnational engineering programme

Allan, William

Ghannam, Rami

Roy, Scott

Crossref

Exam Setting and Moderation in a Transnational Engineering Programme

Enlighten

\       Ghannam, R. , Allan, W. and Roy, S. (2020) Exam Setting and Moderation in a Transnational Engineering Programme. Transnational Engineering Education Using Technology Workshop (TREET 2020), Glasgow, UK, 31 Jul 2020. ISBN 9781728188522 (doi:10.1109/TREET50959.2020.9189755)  The material cannot be used for any other purpose without further permission of the publisher and is for private use only.  There may be differences between this version and the published version. You are advised to consult the publisher’s version if you wish to cite from it.    http://eprints.gla.ac.uk/221129/           Deposited on 20 August 2020      Enlighten – Research publications by members of the University of                  Glasgow http://eprints.gla.ac.uk   XXX-X-XXXX-XXXX-X/XX/$XX.00 ©2020 IEEE Exam Setting and Moderation in a Transnational Engineering Programme  Rami Ghannam  Engineering Education Research Group, James Watt School of Engineering, University of Glasgow Glasgow, UK xxx Device Modelling Group James Watt School of Engineering, University of Glasgow Glasgow, UK William Allan James Watt School of Engineering, University of Glasgow Glasgow, UK xxx Engineering Education Research Group James Watt School of Engineering, University of Glasgow Glasgow, UK Scott Roy James Watt School of Engineering, University of Glasgow Glasgow, UK xxx xx Abstract—The practice of writing effective exam questions has attracted plenty of interest in the literature. However, the process of exam setting and moderation has received little attention. Thus, the aim of this manuscript is to describe our approach in exam setting and moderation for a transnational education (TNE) programme between the University of Glasgow and the University of Electronic Science and Technology of China (UESTC). Each year, Glasgow University staff members prepare main and resit exam papers for 32 courses in the electronic and electrical engineering subject group. To ensure a smooth workflow and a relatively quick turnover of high-quality exam papers each year, we describe the best practices in exam setting, as well as our approach in exam moderation using a new moderation form and a new system for uploading exam papers online. This new system was designed to facilitate a solid dialogue between exam setters and examiners, so that all summative assessments are fair, consistent and reliable. These best practices can be used by engineering faculty to make informed decisions regarding the process of exam setting and moderation in a transnational engineering programme. Keywords— Engineering Education, Transnational Education, Assessment Methods. I. INTRODUCTION Transnational education involves the delivery of educational services in another host country. The UK has established such programmes in almost 120 countries [1]. The practice of writing effective exam questions has received plenty of interest in the literature, since it is one of the methods used to determine how well students have grasped the learning outcomes of a course [2]. However, the process in which these questions are combined to form an examination paper to undergraduate engineering students has not received similar attention [3]. Consequently, the aim of this manuscript is to present our efforts in developing and delivering fair, valid and reliable exam papers to undergraduate engineering students. This is especially important, since multiple examiners are now involved in the preparation and delivery of a course, who are often accustomed to writing questions in their own style and format.  According to the literature, the font, question layout and exam format can strongly influence a candidate’s performance [4, 5]. Consequently, a standardised exam format and style of questions to suit a particular academic level are needed. Moreover, moderation is also essential to ensure consistent marking between multiple examiners. Therefore, in addition to a standardised template and moderations form, an online platform for uploading exam papers that facilitates a solid dialogue between setters and examiners is required. In the following sections, we describe the assessment methods commonly used in Glasgow University’s TNE programme with the University of Electronic Science and Technology of China (UESTC). We also describe the best practices communicated to staff when designing exam questions. Next, we provide screenshots from our newly developed exam database. Finally, we provide concluding remarks and areas for further investigation in the last section. II. CONTEXT A transnational education programme was established between the University of Glasgow and UESTC in 2016. This partnership resulted in the formation of the Glasgow College UESTC programme, which delivers three electronic and electrical engineering degree disciplines. These disciplines are in microelectronics, communications and information engineering. There are approximately 1200 students enrolled in these disciplines and there are 32 courses delivered by Glasgow staff in four different years of study. Courses delivered in the programme have a varied mix of assessments, as can be verified from the course specifications sheets [6]. The three forms of assessments most often used in this TNE programme are formative, summative and interim assessments, as explained below [7]: 1. Summative assessments count towards a student’s final grade. It usually takes place at the end of a particular learning objective to judge how well a student has grasped the material.  2. Formative assessments take place during a student’s learning process. Any marks given to the student are indicative and do not contribute to their final grade. Formative assessments provide feedback to students, so that they have an opportunity to improve their grades.  3. Interim assessments evaluate student performance at particular intervals within the learning process. They can be summative assessments that track student learning.  Examples of the most common types of assessments used by staff members are show in Table I. However, 22 of these courses have a final exam that weighs 75% of the course grade, as shown from the list of courses in Table II. Therefore, to prepare exam scripts, staff were encouraged to produce questions that follow Bloom’s revised taxonomy [8]. Common types of assessment used in the Glasgow College-UESTC programme. TABLE I.  TYPES OF ASSESSMENTS USED IN THE PROGRAMME. Assessment Types What is expected from the student? Laboratory Report Students are required to write a report for a laboratory exercise. Oral Presentation Students are asked to deliver either  a presentation on a particular topic for a specified length of time. Live Demonstration Students are expected to show how a piece of software or an assembled project works. Dialogue Students are engagd in a dialogue. Short Essay Students write a short extract explaining a concept or theory. Mid Term Exam Students complete a set of exam type questions to test their understadning of the course materials. Such an assessment can either be summative or formative. Final Exam Students are expected to complete questions in a specified length of time to measure their understanding of the course materials. Research Project Students are required to complete a major project about a particular topic, either individually or in a team. TABLE II.  COURSES DELIVERED IN GLASGOW COLLEGE UESTC. Course Code Course Name Exam % 1005 Programming 60 1008 Microelectronic Systems 75 1017 English 30 1018 English for Engineering A 30 1019 English for Engineering B 30 2004 Embedded Processors 75 2022 Circuit Analysis 75 2023 Engineering Career Skills 40 2029 English Communication 40 3001 Dynamics & Control 75 3002 Electronic Devices 75 3003 Electronic System Design 75 3005 Power Engineering 75 3010 Team Design & Project Skills 0 3018 Comm. Principles & Systems 75 3020 Digital Circuit Design 75 3022 Power Electronics 75 3029 Communication Circuit Design 75 3030 Engineering & Law 75 3031 Project Management & Finance 75 4003 Control 75 4004 Digital Communications 75 4007 VLSI Design 75 4014 Cyber Security 80 4015 DSP for Communications 80 4016 Microwave Circuit Design 70 4019 Real Time Computing Systems 75 4020 Wireless Sensor Networks 75 4024 Wireless & Optical Transmission 75 4028 Advanced Digital Comms. 75 4029 Advanced Devices 75 4033 Design of Integrated Circuits 75  As previously mentioned in the literature, verbs should be used as an indicator of the level of Learning Outcome (LO) being assessed [9]. For example, verbs associated with demonstrating knowledge include: ‘describe’, ‘explain’, ‘identify’ and ‘define’. These verbs relate to remembering and understanding knowledge (i.e. Bloom levels 1 and 2), as shown in figure 1. Therefore, accreditation panels expect these verbs to be used in first and second year courses. Similarly, verbs associated with evaluation (Bloom level 5) can include: ‘evaluate’, ‘assess’, ‘recommend’ and ‘select’. Such verbs ought to be used in fourth year engineering courses.   The main guidelines for exam setting and moderation involved the following important steps: 1. Staff should develop a main exam paper and a resit paper. Each paper should use the Glasgow College template, which was made available on the dedicated Moodle staff page. The paper should be marked out of 100 and each question should weigh 25 marks. To help students settle into the exam, course setters were encouraged to arrange their questions according to their level of difficulty, such that they progress from easy to hard.   2. Multiple Choice Questions (MCQs) are not allowed for fourth year courses. Otherwise, MCQs must weigh less than 25% of the paper’s marks and should have at least four answer options. 3. In addition to the exam files, staff should develop mark schemes for both the main and resit papers. All staff were reminded on the need for transparent marking criteria. These mark schemes ought to be understood by a non-expert in the subject. This was to ensure that the marking scheme is a useful resource for students, as well as external examiners.  4. To keep track of the moderation process, staff are also required to complete a Moderation Form. Staff should indicate the course code, title and setter name. Consequently, there is a need to develop an online system that enables exam setters and administrators to keep track of this logistical process, especially since many engineering courses  now include a larger percentage of   Figure 1. Staff were encouraged to develop questions that use Bloom’s revised taxonomy.   assessments that include the final exam [10]. An examination database therefore needs to take into account such assessments, which need to be moderated in the same way as exams. This database ought to benefit all staff in the school of engineering and should enable moderators and examiners to monitor the process of exam setting and moderation.    III. METHOD To help staff with exam paper preparation, a template was developed to facilitate developing four questions, each weighing 25 marks. This standardized template was used in all courses. The current typeface used was in the template was Times New Roman and the font size was set to 12 to ensure that the text is readable. However, we will shift to a sans serif font such as Arial or Helvetica in the next semester to comply with the UK’s Equality Act 2010, to ensure that the text is readable by visually impaired candidates [11].  Next, the moderation form shown in figure 2 was developed to help staff, moderators and external moderators to keep track of the process of exam preparation. The template’s rubrics, layout and structure, as well as the moderation form were discussed with other members from the college’s learning and teaching committee to test them for clarity and accessibility. Staff members maintaining this moderation form will likely ensure that the concepts of exam setting, which were previously mentioned in Section II, are sustained.  Subsequently, a new exam database was developed for staff to upload and moderate exam papers. The process of exam setting and moderation is described in the flow chart shown in figure 3. The process starts with the setter submitting their main and resit papers, as well as their mark schemes. A checklist is used to determine whether the files have been uploaded. Once the material is available, internal moderators are informed that papers are ready for moderation. If the papers are approved without comments, they are then passed to the external examiner for approval. Otherwise, the moderator comments are appended to the moderation form and the exam setter is required to update the papers subject to these comments. Once the internal and external moderator comments are addressed, the papers are ready for registry.  During this process, a series of alerts and to do items are made. For example, an email is sent to administrators one day before and one day after the deadlines to catchup with late staff submissions.   IV. RESULTS & DISCUSSIONS Screenshots from new online exam platform are shown in Figure 4. The new platform allows staff to upload main and     Figure 2. Moderation form used to keep track of assessment progress, which is divided into four sections.    Figure 3. Flow chart describing the exam setting and moderation process  resit papers, as well as their corresponding mark schemes. Figure 4a shows the first screenshot after logging in the system. Depending on their role, exam setters can be responsible for both moderating and developing exam papers. By clicking on one of the courses, exam setters will be presented with the screenshot shown in figure 4b. Here, exam setters can keep track and upload their exam materials. By clicking on the “Add Paper” button, a dropdown menu appears, which enables staff to select what type of file is being uploaded. This is to help moderators understand what type is uploaded.   V. CONCLUSION In this manuscript, we have briefly described our approach in exam setting and moderation in the Glasgow College UESTC transnational engineering programme. There are 32 courses delivered by Glasgow University staff members to approximately 1200 students in Chengdu. The majority of these course (69%) contain a final examination component that weighs 75% of the course grade. Therefore, to ensure an efficient workflow of exam setting and moderation, we have developed a standardised exam template, exam handbook, moderation form and a new online system for staff to upload their exam papers.  To further extend this work, we intend to develop online surveys to obtain feedback from staff members and administrators. This will help us identify areas for further improvement.  References  [1] H. B. G. Siora, P. Dickinson, E. Hooper, R. Brown, C. Hill, N. Healey and P. McNamara, "The Scale and Scope of UK Higher Education Transnational Education," 2016. [Online]. Available: https://www.britishcouncil.org/sites/default/files/scale-and-scope-of-uk-he-tne-report.pdf [2] N. O’Donovan, "There are no wrong answers: an investigation into the assessment of candidates’ responses to essay‐based examinations," Oxford Review of Education, vol. 31, no. 3, pp. 395-422, 2005. [3] V. Spalding, "Is an exam paper greater than the sum of its parts? A Literature Review of Question Paper Structure and presentation," 2009. [4] M. L. Bernard, B. S. Chaparro, M. M. Mills, and C. G. Halcomb, "Comparing the effects of text size and format on the readibility of computer-displayed Times New Roman and Arial text," International Journal of Human-Computer Studies, vol. 59, no. 6, pp. 823-835, 2003. [5] V. Crisp and E. Sweiry, "Can a picture ruin a thousand words? The effects of visual resources in exam questions," Educational Research, vol. 48, no. 2, pp. 139-154, 2006. [6] "Courses In University Of Electronic Science And Technology Of China For 2019-2020." https://www.gla.ac.uk/coursecatalogue/courselist/?code=UESTC&name=University+of+Electronic+Science+and+Technology+of+China (accessed 23 June, 2020). [7] W. Harlen and M. James, "Assessment and learning: differences and relationships between formative and summative assessment," Assessment in Education: Principles, Policy & Practice, vol. 4, no. 3, pp. 365-379, 1997. [8] D. R. Krathwohl and L. W. Anderson, A taxonomy for learning, teaching, and assessing: A revision of Bloom's taxonomy of educational objectives. Longman, 2009. [9] C. Rust, M. Price, and B. O'DONOVAN, "Improving students' learning by developing their understanding of assessment criteria and processes," Assessment & Evaluation in Higher Education, vol. 28, no. 2, pp. 147-164, 2003. [10]  R. Ghannam and W. Ahmad, "New team based learning course for Chinese students in engineering and technology," in Compass, 2019: University of Greenwich.  [11] "Accessible communication formats." https://www.gov.uk/government/publications/inclusive-communication/accessible-communication-formats (accessed 23 June, 2020). (a)  (b)  (c)  Figure 4. Snapshots from the new exam database. In (a) the setter is presented with their list of courses. By clicking on one the courses, the screen in (b) appears, which lists the main and resit exam papers. By clicking on the “Add Paper” button, the dropdown menu shown in (c) appears. 

http://eprints.gla.ac.uk/221129/8/221129.pdf

Exam Setting and Moderation in a Transnational Engineering Programme

Abstract

Similar works

Full text

Available Versions

Crossref

Enlighten