Marking complex assignments using peer assessment with an electronic voting system and an automated feedback tool

The work described in this paper relates to the development and use of a range of initiatives in order to mark complex masters' level assignments related to the development of computer web applications. In the past such assignments have proven difficult to mark since they assess a range of skills including programming, human computer interaction and design. Based on the experience of several years marking such assignments, the module delivery team decided to adopt an approach whereby the students marked each other's practical work using an electronic voting system (EVS). The results of this are presented in the paper along with statistical comparison with the tutors' marking, providing evidence for the efficacy of the approach. The second part of the assignment related to theory and documentation. This was marked by the tutors using an automated feedback tool. It was found that the time to mark the work was reduced by more than 30% in all cases compared to previous years. More importantly it was possible to provide good quality individual feedback to learners rapidly. Feedback was delivered to all within three weeks of the test submission datePeer reviewe

The use of electronic voting and peer assessment to encourage the development of higher order thinking skills in learners

In the research reported here, electronic voting and peer assessment were used with 215 first year computer science and information technology undergraduates undertaking an electronic media design module. In previous years it proved difficult for learners following this module to develop good quality practical skills resulting in poor results in practical tests. In this study, the module delivery team used electronic voting and a form of peer assessment in order to motivate learners and to engage them more deeply in learning. It was hypothesised that this would improve higher order thinking skills and lead to improved performance in practical work. A significant improvement in performance of 6% was achieved (p<0.001) as compared to previous years. We were able to show that the most likely reason for this was the use of electronic voting systems and peer assessment. In the concluding section of the paper we discuss some issues related to this finding and our teaching approachPeer reviewe

A Study into the Usability and Security Implications of Text and Image Based Challenge Questions in the Context of Online Examination

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.Online examinations are an integral component of online learning environments and research studies have identified academic dishonesty as a critical threat to the credibility of such examinations. Academic dishonesty exists in many forms. Collusion is seen as a major security threat, wherein a student invites a third party for help or to impersonate him or her in an online examination. This work aims to investigate the authentication of students using text-based and image-based challenge questions. The study reported in this paper involved 70 online participants from nine countries completing a five week online course and simulating an abuse case scenario. The results of a usability analysis suggested that i) image-based questions are more usable than text-based questions (p < 0.01) and ii) using a more flexible data entry method increased the usability of text-based questions (p < 0.01). An impersonation abuse scenario was simulated to test the influence of sharing with different database sizes. The findings revealed that iii) an increase in the number of questions shared for impersonation increased the success of an impersonation attack and the results showed a significant linear trend (p < 0.01). However, the number of correct answers decreased when the attacker had to memorize and answer the questions in an invigilated online examination or their response to questions was timed. The study also revealed that iv) an increase in the size of challenge question database decreased the success of an impersonation attack (p < 0.01).Peer reviewe

A Dynamic Profile Questions Approach to Mitigate Impersonation in Online Examinations

© The Author(s) 2018 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.Online examinations are an integral component of many online learning environments, which face many security challenges. Collusion is seen as a major security threat to such examinations, when a student invites a third party to impersonate or abet in a test. This work aims to strengthen the authentication of students via the use of dynamic profile questions. The study reported in this paper involved 31 online participants from five countries over a five-week period. The results of usability and security analysis are reported. The dynamic profile questions were more usable than both the text-based and image-based questions (p < 0.01). An impersonation abuse scenario was simulated using email and mobile phone. The impersonation attack via email was not successful, however, students were able to share answers to dynamic profile questions with a third party impersonator in real time, which resulted in 93% correct answers. The sharing of information via phone took place in real time during an online test and the response time of an impersonator was significantly different (p < 0.01) than a student. The study also revealed that a response time factor may be implemented to identify and report impersonation attacks.Peer reviewe

Mobile Application Usability: Heuristic Evaluation and Evaluation of Heuristics

Ger Joyce, Mariana Lilley, Trevor Barker, and Amanda Jefferies, 'Mobile Application Usability: Heuristic Evaluation and Evaluation of Heuristics', paper presented at AHFE 2016 International Conference on Human Factors, Software, and Systems Engineering. Walt Disney World, Florida USA, 27-31 July 2016Many traditional usability evaluation methods do not consider mobile-specific issues. This can result in mobile applications that abound in usability issues. We empirically evaluate three sets of usability heuristics for use with mobile applications, including a set defined by the authors. While the set of heuristics defined by the authors surface more usability issues in a mobile application than other sets of heuristics, improvements to the set can be made

Evaluating Security and Usability of Profile Based Challenge Questions Authentication in Online Examinations

© 2014 Ullah et al.; licensee Springer. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited.Student authentication in online learning environments is an increasingly challenging issue due to the inherent absence of physical interaction with online users and potential security threats to online examinations. This study is part of ongoing research on student authentication in online examinations evaluating the potential benefits of using challenge questions. The authors developed a Profile Based Authentication Framework (PBAF), which utilises challenge questions for students’ authentication in online examinations. This paper examines the findings of an empirical study in which 23 participants used the PBAF including an abuse case security analysis of the PBAF approach. The overall usability analysis suggests that the PBAF is efficient, effective and usable. However, specific questions need replacement with suitable alternatives due to usability challenges. The results of the current research study suggest that memorability, clarity of questions, syntactic variation and question relevance can cause usability issues leading to authentication failure. A configurable traffic light system was designed and implemented to improve the usability of challenge questions. The security analysis indicates that the PBAF is resistant to informed guessing in general, however, specific questions were identified with security issues. The security analysis identifies challenge questions with potential risks of informed guessing by friends and colleagues. The study was performed with a small number of participants in a simulation online course and the results need to be verified in a real educational context on a larger sample sizePeer reviewedFinal Published versio

Privacy and Usability of Image and Text Based Challenge Questions Authentication in Online Examination

In many online examinations, physical invigilation is often replaced with traditional authentication approaches for student identification. Secure and usable authentication approaches are important for high stake online examinations. A Profile Based Authentication Framework (PBAF) was developed and implemented in a real online learning course embedded with summative online examination. Based on users’ experience of using the PBAF in an online course, online questionnaires were used to collect participants' feedback on effectiveness, layout and appearance, user satisfaction, distraction and privacy concerns. Based on overall findings of the quantitative analysis, there was a positive feedback on the use of a hybrid approach utilizing image and text based challenge questions for better usability. However, the number of questions presented during learning and examination processes were reported to be too many and caused distraction. Participants expressed a degree of concern on sharing personal and academic information with little or no privacy concern on using favorite question

Design, Privacy and Authentication of Challenge Questions in Online Examinations

Online examination is an essential part of the online learning and secure authentication is considered vital for the success of online learning. This study is part of an ongoing research on student authentication approaches and the use of challenge questions in online examination authentication. This paper presents the results of an empirical study based on “Profile Based Authentication Framework” (PBAF), which uses challenge questions for student authentication in online examination. The PBAF uses challenge questions related to personal, academic and professional information. These questions inform the usability, security and privacy of PBAF authentication approach. The results presented here summarizes the impact of questions design on the usability based on data collected from challenge questions authentication and a post-experiment survey on the data privac