The authors would like to thank Dr Craig brown, University of Aberdeen for assistance with data analysis.Peer reviewedPublisher PD

Cleland, J.

Currie, G. P.

Denison, A. R.

Sinha, S.

Thomson, F.

The Journal of the Royal College of Physicians of Edinburgh

English

The authors would like to thank Dr Craig brown, University of Aberdeen for assistance with data analysis.Peer reviewe

Aberdeen University Research

PAPERJ r Coll Physicians edinb 2017; 47: 164–7  |  doi: 10.4997/JrCPe.2017.215EducationTablet computers in assessing performance in a high stakes exam: opinion mattersgP Currie1, s sinha2, f thomson3, J Cleland4, ar denison5Background Tablet computers have emerged as a tool to capture, process and store data in examinations, yet evidence relating to their acceptability and usefulness in assessment is limited. Methods We performed an observational study to explore opinions and attitudes relating to tablet computer use in recording performance in a  nal year objective structured clinical examination at a single UK medical school. Examiners completed a short questionnaire encompassing background, forced-choice and open questions. Forced choice questions were analysed using descriptive statistics and open questions by framework analysis. Results Ninety-two (97% response rate) examiners completed the questionnaire of whom 85% had previous use of tablet computers. Ninety per cent felt checklist mark allocation was ‘very/quite easy’, while approximately half considered recording ‘free-type’ comments was ‘easy/very easy’. Greater overall ef ciency of marking and resource savings were considered the main advantages of tablet computers, while concerns relating to technological failure and ability to record free type comments were raised. Discussion In a context where examiners were familiar with tablet computers, they were preferred to paper checklists, although concerns were raised. This study adds to the limited literature underpinning the use of electronic devices as acceptable tools in objective structured clinical examinations. Keywords: assessment, technologyDeclaration of interests: no conﬂ ict of interests declaredAbstractIntroduction the objective structured clinical examination (osCe) is a well-established tool in under- and postgraduate assessments and allows a variety of skills and practical aptitudes to be assessed in a relatively short period of time. however, this method of assessment can be associated with signiﬁ cant drawbacks such as cost, time required to assess large numbers of students, and need for signiﬁ cant resource and experienced examiners.1–3 in a traditional osCe, examiners are often required to record handwritten marks on ‘machine-readable’ paper according to pre-deﬁ ned checklists/criteria and global performance indicators, and frequently have the option to add ‘free text’ comments relating to candidates’ performance. nevertheless, issues can arise due to missing marks, lost papers, poor handwriting (especially when providing written feedback), difﬁ culties in optical software correctly recognising allocated marks, accuracy of data transfer, security, time taken to process, and storage. interest in and use of mobile electronic technologies has developed rapidly in the past decade and many opportunities exist whereby they can enhance teaching, learning and assessment in medical education.4 one study5 compared the use of personal digital assistants (Pdas) and traditional paper marking in assessment, and found no signiﬁ cant differences in students’ marks. moreover, their use resulted in substantially less time needed to process results, while assessor feedback on Pda assessment was ‘overwhelmingly positive’. in further small studies, hochlenhert et al.6 found that both examiners and examinees considered recording results in an osCe using a tablet computer was acceptable and satisfactory, while schmitz et al.7 found electronic checklists were ‘more usable’ and ‘preferred’ compared to ‘paper and pencil’ checklists. despite these positive reports, 1Consultant Chest Physician, aberdeen royal inﬁ rmary and honorary senior Clinical lecturer, institute of education for medical and dental sciences, university of aberdeen; 2,3Clinical teaching fellows, institute of education for medical and dental sciences, university of aberdeen; 4John simpson Chair of medical education research, institute of education for medical and dental sciences, university of aberdeen; 5Consultant radiologist and deputy director, institute of education for medical and dental sciences, university of aberdeen, uKCorrespondence to: gP CurrieChest Clinic Caberdeen royal inﬁ rmaryforesterhillaberdeen ab25 2ZnuK Email: graeme.currie@nhs.net164    Journal of the royal College of PhysiCians of edinburgh  volume 47  issue 2  June 2017further exploration into the acceptability, incorporating larger numbers of examiners – especially in high-stakes assessments – using more technological advanced devices in widespread use such as iPads™ is lacking. furthermore, since provision of general feedback is often of paramount importance, it is pertinent to determine the perceived ease (or otherwise) by which electronic devices can permit this.8 our aim therefore was to explore views and attitudes of examiners towards the use of tablet computers as a tool to facilitate mark allocation and record comments in a high-stakes osCe.MethodsWe conducted an observational study to explore the acceptability of tablet computers (in this case the iPad) as a tool to record osCe performance in the ﬁ nal year of the medical degree programme at the university of aberdeen, uK. the osCe comprised a 15-station assessment across two days and ﬁ ve sites in 2016, and involved 193 students and 95 examiners. the iPad interface was based on a bespoke electronic osCe app.8 in all mbChb osCes held at the university of aberdeen, invited examiners are advised and expected to undergo online training prior to being ofﬁ cially allocated an osCe station. speciﬁ c training in electronic marking was offered several weeks prior to the 2016 assessment. at the exam day brieﬁ ng, examiners were encouraged to undergo brief (lasting minutes) tablet computer training. they were also advised to type individual comments, especially in borderline or poorly performing students.  We developed an anonymous questionnaire in order to explore views and attitudes of examiners (appendix 1, available with the online version of this paper). it determined basic examiner details before presenting force choice (closed) questions, each of which had the option of adding an open comment. the ﬁ nal version of the survey was agreed following piloting with subsequent minor amendments to its design, structure and content. We sampled all ﬁ nal year osCe examiners, all of whom had experience of using traditional paper marking schedules. examiners were asked at the outset if they would complete the questionnaire at the end of their examining session. Questionnaires were collected by site co-ordinators on completion of the exam. forced-choice questions were analysed using descriptive statistics. open comments were analysed using framework analysis. this process consisted of familiarisation of data, identiﬁ cation of recurrent themes, indexing and charting. to assess the relationship between format (paper versus tablet computer) and missing data, we compared data collected by traditional, written scoring sheet from the previous year’s osCe, to that from the 2016 assessment. both osCes were equivalent in terms of number of stations, examiners and students. approval for this study was granted from the College ethics research board, College of life sciences and medicine, university of aberdeen. ResultsMissing marksin the comparative (2015) 15-station osCe, 129 (4.5%) of speedwell™ sheets contained at least one missing mark. there were no missing marks in the 2016 tablet computer-recorded osCe. Questionnaire responsesninety-three out of 95 (97% response rate) examiners completed a questionnaire of whom 16 (17%) had never previously examined in a ﬁ nal year osCe. most (87 (95%)) examiners answered all questions. demographics and other baseline data are shown in table 1. seventy-eight (85%) respondents indicated they used tablet computers for other purposes, of whom most (79%) used it ‘regularly’ or ‘often’ for recreational or professional use.‘regarding mark allocation, how did you ﬁ nd iPad marking?’• eighty-three (90%) respondents felt this was ‘very or quite easy’, 1 (1%) felt this ‘difﬁ cult or very difﬁ cult’. ‘regarding documenting ‘free-type’ comments, how do you ﬁ nd the iPad?’• forty-eight (52%) respondents considered this to be ‘very or quite easy’, 20 (22%) found this ‘quite or very difﬁ cult’. ‘do you prefer the traditional paper marking or iPads in an osCe?’• seventy-one (78%) preferred the iPad, while 7 (8%) preferred paper mark allocation. Written responses to explain their answer were given by 70 (76%) respondents. Common themes identiﬁ ed (divided into favouring iPad versus favouring paper) and frequencies were as follows:Table 1 Characteristics of examiners completing the questionnaire Gender Frequencymale 53 (58%)age (years)< 30 1 (1%)30–39 22 (24%)40–49 23 (25%)50–59 32 (35%)60–69 13 (14%)DesignationConsultant 61 (66%)specialist registrar/junior doctor 10 (11%)university faculty 6 (7%)speciality doctor 3 (3%)general practitioner 9 (10%)other 3 (3%)Previous use of iPad for OSCE markingnever 35 (38%)once 10 (11%)2–3 times 15 (16%)4 or more times 32 (35%)June 2017  volume 47  issue 2  Journal of the royal College of PhysiCians of edinburgh    165 tablet computers in assessmentfavouring iPad™ (61 comments with three main themes)1. Perception of the marking process being ‘easier’ (n = 32)2. greater accuracy of marking as unable to proceed until all questions marked (n = 21)3. Conservation of resources (e.g. paper, administrative staff time; n = 5)favouring paper (15 comments with two main themes)1. speed of typing free text comments (n = 5)2. navigation difﬁ culties within app (n = 7)‘how did you ﬁ nd the training for using the iPad?’• sixty-six (72%) respondents considered that training was at least ‘satisfactory’ or ‘very satisfactory’, only 4 (4%) found it to be ‘very unsatisfactory’ or ‘unsatisfactory’. Written comments were provided by 40 (43%). Common themes identiﬁ ed and frequencies were:1. Positive about adequacy of training (n = 32)2. negative/ambiguous statements about training (n = 8)  ‘What do you feel are the main advantages of iPad marking?’• eighty-five (92%) respondents provided comments. Common themes identiﬁ ed (131 comments across eight themes) and frequencies were:1. greater accuracy of marking (e.g. prompts against missing marks; n = 43)2. Perception of easier overall marking process among examiners versus paper marking (n = 29)3. greater efﬁ ciency (n = 17)4. beneﬁ ts to environment (n = 12)5. improved appearance of mark sheets (e.g. no handwriting (n = 5))6. improved efﬁ ciency of process post exam (n = 16)7. improvement in exam content and process (n = 2) ‘What do you feel are the main disadvantages of iPad marking?’• seventy-six (83%) respondents commented. Common themes identiﬁ ed (78 comments across four themes) and frequencies were: 1. technology failings/navigation issues (n = 37)2. too little time to type comments (n = 16)3. issues with timer function (n = 14)4. unfamiliar with technology (n = 2)Discussionthis study adds to the limited but growing body of literature describing the use of handheld electronic devices within medical education, and particularly in undergraduate assessment. it demonstrated that the majority (90%) of examiners in a high stakes undergraduate ﬁ nal year osCe found iPad data entry relatively straightforward with most (78%) preferring this method to paper checklists. only half considered that recording free-type comments was sufﬁ ciently easy. most (72%) examiners felt that training was at least satisfactory. the main advantages were removal of the potential for missing marks (the tablet computer had a built in app so that examiners were unable to proceed to the next student without completing all checklist items), greater efﬁ ciency of the overall marking process (pre and post exam) and perceived environmental beneﬁ ts. main disadvantages included concerns regarding device failure, insufﬁ cient time to document free text comments and issues relating to the timer function on the app. our ﬁ ndings are in agreement with those from smaller studies of lower-stakes exams which explored the acceptability of electronic methods of recording exam marks in medical education.5–7 unlike participants in previous studies,7 in this study the majority of examiners used tablet computers for recreational and/or work-related purposes on a regular basis, which may have contributed to their preference over paper. examiners preferred tablet computers because of savings in paper and staff time; they may also be cost-effective as the initial investment in devices and software may be quickly ‘paid back’ by such resource savings.9 however, whether this is indeed the case requires further longitudinal assessment, especially as tablet computers need upkeep and maintenance and their lifespan is ﬁ nite. in contrast to previous studies, our study reveals perceived disadvantages. Concerns were raised regarding device failure, although no tablet malfunctioned during the assessment. nevertheless, as a safeguard, traditional paper checklists should be available in the event of technological failure. although some examiners felt insufﬁ cient time was available to document free text comments, one study found that tablet computer use in a lower-stakes osCe led to greater and higher quality written feedback compared to paper-based checklists.8 Whether or not this ﬁ nding also applies to examiners in a higher stakes exam requires further exploration. some issues with the app timer were also reported, and these highlight the need to ensure all ‘glitches’ are anticipated when using such devices in assessment. it is perhaps surprising that concerns were not repeatedly expressed relating to either the potential for technological ‘glitches’ arising during the downloading and transferring of data or electronic storage/security. in this respect, the university of aberdeen employs a dedicated information technology team with the responsibility of ensuring a high level and professional standard of electronic data handling, processing and storage. our research has limitations. this was a single-site study and ﬁ ndings may not be reﬂ ective of those in other institutions, countries, postgraduate exams and healthcare disciplines. We did not capture opinions of students undergoing assessment; one previous small study did demonstrate that students were satisﬁ ed when assessors used electronic devices to record data.10 unlike previous studies6,7 we did not determine how examiners perceived the extent of mental effort required. however, as most assessors in our study used tablet computers for other purposes, it is unlikely they would have considered using electronic devices to be difﬁ cult, while major issues relating to difﬁ culty in their use were not raised in the questionnaire. 166    Journal of the royal College of PhysiCians of edinburgh  volume 47  issue 2  June 2017gP Currie, s sinha, f thomson et al.in conclusion, our study suggests that in a context in which examiners are familiar with tablet computers they are an acceptable tool by which to record marks, and should obviate problems of missing marks. it is important that checklist mark recording apps are continually reﬁ ned and examiner feedback is taken into account. although navigational concerns were not highlighted to be a major concern, developers need to be mindful that such apps should not become too complex and remain user-friendly, especially as only a ﬁ nite time is available for examiners to record marks. it is also imperative that information technology departments ensure that sufﬁ ciently secure processes and systems are in place with regards to safety of data handling and transfer, with no possibility for data to be accessed or modiﬁ ed by non-authorised or external parties. We urge others to add to the discourse on this topic by carrying out similar studies in different contexts. Acknowledgmentsthe authors would like to thank dr Craig brown, university of aberdeen for assistance with data analysis.ContributionsgPC and ard had the original idea for the study. gPC wrote the protocol, applied for ethics permission and facilitated access to the examiners. gPC, ard, JC and ss designed and piloted the questionnaire. ft and ss collected the data, and carried out the data analysis under the supervision of gC and ad. ss prepared a preliminary draft report for feedback from the other authors. gPC and JC rewrote subsequent drafts to form this paper. References1 Patricio m, Juliao m, fareleira f et al. is the osCe a feasible tool to assess competencies in undergraduate medical education? Med Teach 2013; 35: 503–14.2 brown C, ross s, Cleland J et al. money makes the (medical assessment) world go round: the cost of components of a summative final year objective structured Clinical examination (osCe). Med Teach 2015; 37: 653–9.3 Currie gP, sivasubramaniam s, Cleland J. sequential testing in a high stakes osCe: determining number of screening tests. Med Teach 2016; 38: 708–14.4 masters K, ellaway rh, topps d et al. mobile technologies in medical education: amee guide no. 105. Med Teach 2016, 38: 537–49.5 treadwell i. the usability of personal digital assistants (Pdas) for assessment of practical performance. Med Educ 2006; 40: 855–61.6 hochlehnert a, schultz Jh, moltner a et al. electronic acquisition of osCe performance using tablets. GMS Z Med Ausbild 2015; 32: doc41.7 schmitz fm, Zimmermann Pg, gaunt K et al. electronic rating of objective structured clinical examinations: mobile digital forms beat paper and pencil checklists in a comparative study. Information Quality in eHealth 2011; 7058: 501–12.8 denison a, bate e, thompson J. tablet versus paper marking in assessment: feedback matters. Perspect Med Edu 2016; 5: 108–13.9 brown C. tablet-or iPad-based marking of osCes and mmis: an imaginative cost-saving approach. Med Teach 2015; 38: 211–2.10 van schoor an, navsa n, meiring Jh et al. Perspectives on the use of Pdas as assessment tools. Clin Teach 2006; 3: 170–4.June 2017  volume 47  issue 2  Journal of the royal College of PhysiCians of edinburgh    167 tablet computers in assessment

Tablet computers in assessing performance in a high stakes exam : opinion matters

GP Currie

S Sinha

F Thomson

J Cleland

AR Denison

Crossref

Tablet computers in assessing performance in a high stakes exam: opinion matters

https://aura.abdn.ac.uk/bitstream/2164/8950/1/jrcpe_47_2_currie.pdf

Tablet computers in assessing performance in a high stakes exam : opinion matters

Abstract

Similar works

Full text

Available Versions

Aberdeen University Research

Crossref