INQUIRE: a case study in evaluating the potential of online MCQ tests in a discursive subject

There has been a wealth of investigation into the use of online multiple-choice questions as a means of summative assessment, however the research into the use of formative MCQs by the same mode of delivery still remains patchy. Similarly, research and implementation has been largely concentrated within the Sciences and Medicine rather than the more discursive subjects within the Humanities and Social Sciences. The INQUIRE (Interactive Questions Reinforcing Education) Evaluation Project was jointly conducted by two groups at the University of Oxford-the Said Business School and the Academic Computing Development Team to evaluate the use of online MCQs as a mechanism to reinforce and extend student learning. This initial study used a small set of highly focused MCQ tests that were designed to complement an introductory series of first-year undergraduate management lectures. MCQ is a simple and well-established technology, and hence the emphasis was very much on situating the tests within the student experience. The paper will cover how the online MCQs are intended to fit into the Oxford Undergraduate study agenda, and how a simple evaluation was executed and planned to investigate their usage and impact. The chosen method of evaluation was to combine focus groups with automated online methods of tracking, and the paper discusses the findings of both of these

Introducing a framework to assess newly created questions with Natural Language Processing

Statistical models such as those derived from Item Response Theory (IRT) enable the assessment of students on a specific subject, which can be useful for several purposes (e.g., learning path customization, drop-out prediction). However, the questions have to be assessed as well and, although it is possible to estimate with IRT the characteristics of questions that have already been answered by several students, this technique cannot be used on newly generated questions. In this paper, we propose a framework to train and evaluate models for estimating the difficulty and discrimination of newly created Multiple Choice Questions by extracting meaningful features from the text of the question and of the possible choices. We implement one model using this framework and test it on a real-world dataset provided by CloudAcademy, showing that it outperforms previously proposed models, reducing by 6.7% the RMSE for difficulty estimation and by 10.8% the RMSE for discrimination estimation. We also present the results of an ablation study performed to support our features choice and to show the effects of different characteristics of the questions' text on difficulty and discrimination.Comment: Accepted at the International Conference of Artificial Intelligence in Educatio

Assessment in anatomy

From an educational perspective, a very important problem is that of assessment, for establishing competency and as selection criterion for different professional purposes. Among the issues to be addressed are the methods of assessment and/or the type of tests, the range of scores, or the definition of honour degrees. The methods of assessment comprise such different forms such as the spotter examination, short or long essay questions, short answer questions, true-false questions, single best answer questions, multiple choice questions, extended match questions, or several forms of oral approaches such as viva voce examinations.Knowledge about this is important when assessing different educational objectives; assessing educational objectives from the cognitive domain will need different assessment instruments than assessing educational objectives from the psychomotor domain or even the affective domain.There is no golden rule, which type of assessment instrument or format will be the best in measuring certain educational objectives; but one has to respect that there is no assessment instrument, which is capable to assess educational objectives from all domains of educational objectives.Whereas the first two or three levels of progress can be assessed by well-structured written examinations such as multiple choice questions, or multiple answer questions, other and higher level progresses need other instruments, such as a thesis, or direct observation.This is no issue at all in assessment tools, where the students are required to select the appropriate answer from a given set of choices, as in true false questions, MCQ, EMQ, etc. The standard setting is done in these cases by the selection of the true answer

Arthroscopy or ultrasound in undergraduate anatomy education: a randomized cross-over controlled trial

Background: The exponential growth of image-based diagnostic and minimally invasive interventions requires a detailed three-dimensional anatomical knowledge and increases the demand towards the undergraduate anatomical curriculum. This randomized controlled trial investigates whether musculoskeletal ultrasound (MSUS) or arthroscopic methods can increase the anatomical knowledge uptake. Methods: Second-year medical students were randomly allocated to three groups. In addition to the compulsory dissection course, the ultrasound group (MSUS) was taught by eight, didactically and professionally trained, experienced student-teachers and the arthroscopy group (ASK) was taught by eight experienced physicians. The control group (CON) acquired the anatomical knowledge only via the dissection course. Exposure (MSUS and ASK) took place in two separate lessons (75 minutes each, shoulder and knee joint) and introduced standard scan planes using a 10-MHz ultrasound system as well as arthroscopy tutorials at a simulator combined with video tutorials. The theoretical anatomic learning outcomes were tested using a multiple-choice questionnaire (MCQ), and after cross-over an objective structured clinical examination (OSCE). Differences in student's perceptions were evaluated using Likert scale-based items. Results: The ASK-group (n = 70, age 23.4 (20--36) yrs.) performed moderately better in the anatomical MC exam in comparison to the MSUS-group (n = 84, age 24.2 (20--53) yrs.) and the CON-group (n = 88, 22.8 (20--33) yrs.; p = 0.019). After an additional arthroscopy teaching 1 % of students failed the MC exam, in contrast to 10 % in the MSUS- or CON-group, respectively. The benefit of the ASK module was limited to the shoulder area (p < 0.001). The final examination (OSCE) showed no significant differences between any of the groups with good overall performances. In the evaluation, the students certified the arthroscopic tutorial a greater advantage concerning anatomical skills with higher spatial imagination in comparison to the ultrasound tutorial (p = 0.002; p < 0.001). Conclusions: The additional implementation of arthroscopy tutorials to the dissection course during the undergraduate anatomy training is profitable and attractive to students with respect to complex joint anatomy. Simultaneous teaching of basic-skills in musculoskeletal ultrasound should be performed by medical experts, but seems to be inferior to the arthroscopic 2D-3D-transformation, and is regarded by students as more difficult to learn. Although arthroscopy and ultrasound teaching do not have a major effect on learning joint anatomy, they have the potency to raise the interest in surgery

Changing evaluation of a master programme module to improve students' overall achievements

One of the modules delivered since 2001 in the Mechanical Engineering taught Masters Programme at Dublin City University is entitled “Product Design, Development and Value Analysis”. The module is assessed by two continuous assessments, both worth 10% and by a final exam worth 80%. In 2005, the exam period of this module was increased from 120 minutes to 150 minutes. Additionally, the second continuous assessment assignment was change from a report assignment to a set of 20 Multiple Choice Questions (MCQ) covering the entire module content. This assessment was run in the final week of the module. The first assignment was and remains as a report assignment. These two modifications were introduced to give the students sufficient time to complete the exam paper, and to provide a more comprehensive evaluation by the students for themselves as to their extent of knowledge on the module content. Average results of continuous assessment, exam marks, and overall marks before and after the change were assessed from 2002 to 2015 and presented in this paper. The two above-mentioned changes brought 14.2% increase in the continuous assessment average results, and 15.7% increase in the final exam marks. Therefore the combined effect of the changes has resulted in a significant positive increase in the student overall achievement within this module. The data and reasons behind these changes are discussed

Beyond model answers: learners’ perceptions of self-assessment materials in e-learning applications

The importance of feedback as an aid to self‐assessment is widely acknowledged. A common form of feedback that is used widely in e‐learning is the use of model answers. However, model answers are deficient in many respects. In particular, the notion of a ‘model’ answer implies the existence of a single correct answer applicable across multiple contexts with no scope for permissible variation. This reductive assumption is rarely the case with complex problems that are supposed to test students’ higher‐order learning. Nevertheless, the challenge remains of how to support students as they assess their own performance using model answers and other forms of non‐verificational ‘feedback’. To explore this challenge, the research investigated a management development e‐learning application and investigated the effectiveness of model answers that followed problem‐based questions. The research was exploratory, using semi‐structured interviews with 29 adult learners employed in a global organisation. Given interviewees’ generally negative perceptions of the model‐answers, they were asked to describe their ideal form of self‐assessment materials, and to evaluate nine alternative designs. The results suggest that, as support for higher‐order learning, self‐assessment materials that merely present an idealised model answer are inadequate. As alternatives, learners preferred materials that helped them understand what behaviours to avoid (and not just ‘do’), how to think through the problem (i.e. critical thinking skills), and the key issues that provide a framework for thinking. These findings have broader relevance within higher education, particularly in postgraduate programmes for business students where the importance of prior business experience is emphasised and the profile of students is similar to that of the participants in this research

Constructed response or multiple-choice questions for assessing declarative programming knowledge? That is the question!

Aim/Purpose This paper presents a data mining approach for analyzing responses to advanced declarative programming questions. The goal of this research is to find a model that can explain the results obtained by students when they perform exams with Constructed Response questions and with equivalent Multiple-Choice Questions. Background The assessment of acquired knowledge is a fundamental role in the teachinglearning process. It helps to identify the factors that can contribute to the teacher in the developing of pedagogical methods and evaluation tools and it also contributes to the self-regulation process of learning. However, better format of questions to assess declarative programming knowledge is still a subject of ongoing debate. While some research advocates the use of constructed responses, others emphasize the potential of multiple-choice questions. Methodology A sensitivity analysis was applied to extract useful knowledge from the relevance of the characteristics (i.e., the input variables) used for the data mining process to compute the score. Contribution Such knowledge helps the teachers to decide which format they must consider with respect to the objectives and expected students results. Findings The results shown a set of factors that influence the discrepancy between answers in both formats. Recommendationsfor Practitioners Teachers can make an informed decision about whether to choose multiplechoice questions or constructed-response taking into account the results of this study. Recommendations for Researchers In this study a block of exams with CR questions is verified to complement the area of learning, returning greater performance in the evaluation of students and improving the teaching-learning process. Impact on Society The results of this research confirm the findings of several other researchers that the use of ICT and the application of MCQ is an added value in the evaluation process. In most cases the student is more likely to succeed with MCQ, however if the teacher prefers to evaluate with CR other research approaches are needed. Future Research Future research must include other question formats.info:eu-repo/semantics/publishedVersio