Marking complex assignments using peer assessment with an electronic voting system and an automated feedback tool

The work described in this paper relates to the development and use of a range of initiatives in order to mark complex masters' level assignments related to the development of computer web applications. In the past such assignments have proven difficult to mark since they assess a range of skills including programming, human computer interaction and design. Based on the experience of several years marking such assignments, the module delivery team decided to adopt an approach whereby the students marked each other's practical work using an electronic voting system (EVS). The results of this are presented in the paper along with statistical comparison with the tutors' marking, providing evidence for the efficacy of the approach. The second part of the assignment related to theory and documentation. This was marked by the tutors using an automated feedback tool. It was found that the time to mark the work was reduced by more than 30% in all cases compared to previous years. More importantly it was possible to provide good quality individual feedback to learners rapidly. Feedback was delivered to all within three weeks of the test submission datePeer reviewe

Linking factual and procedural knowledge in solving science problems: A case study in a thermodynamics course

Well-specified problems of the type presented boxed in the introduction to this article are extremely common in science courses. Unfortunately, this does not mean that students find them easy to solve, even when a teacher provides model answers to problems which differ only marginally (in the teacher's eyes) from those put before the students. The central difficulty with such courses is that they do not embody instructional principles that reflect students' need for “direction” in problem solving. In this article, we describe how the necessary heuristics and strategic knowledge were built into the remake of a conventional thermodynamics course. In contrast to mainstream American work on learning problem solving we chose to direct our curriculum reconstruction using the Gal'perin theory of stage-by-stage formation of mental actions and Landa's description of the “through” systematization of knowledge. As indicated by both, we first developed an integrated system of instructional objectives: a programme of actions and methods (PAM) to solve problems in thermodynamics. Then the plan of instruction was designed. This plan indicates which instructional procedures and materials should be used to realize the instructional functions, derived from the learning theory. The evaluation design contained two control and three experimental courses. In discussing our main findings, we consider the generalizability of the procedures we followed in constructing the PAM and the instructional plan

Training for real: starting in the University

Graduates of Engineering in Uganda face great challenges in the work place as they are expected to be fully prepared to start work with little or no further training. In view of this, training has to begin at the University, where it has been found that the quality of teaching is poor, by admission of the lecturers themselves. This paper reports on an initiative by Uganda Martyrs University to systematically analyse the nature of teaching in the Faculty of Building Design, and to design, develop, implement and evaluate an intervention in the form of training to equip lecturers with basic teaching skills. It is hoped that this will in turn increase the chances of students being trained for real

The Automatic Assessment of Multiple Artefacts: An Investigation into Design Diagrams and Their Implementations

As the Higher Education sector has moved towards student-centred learning so too has the growth in electronic support for learning. E-assessment has been a part of this growth as increasingly assessment and its feedback is seen as an integral part of the students’ learning process. Mature e-assessment systems exist, particularly where answers to questions are restricted to a prescribed list of alternatives. However, for free response artefacts, where there is a limited restriction placed on answers to questions, automated assessment systems are embryonic. This dissertation presents an investigation into the automated assessment of free response artefacts. Design diagrams and their accompanying source code implementations are examples of free response artefacts. A case study is developed that investigates how to automatically generate formative feedback for a design diagram by utilizing its accompanying implementation. The dissertation presents a two-staged solution, initially analysing the design diagram in isolation before comparing it with the implementation. A framework for this approach has been developed and tested using a tool applied to coursework submitted by undergraduate computer science students. The tool was evaluated by comparing the formative feedback comments generated by the tool with those produced by a team of computer science educators. Evaluation was undertaken via two Likert questionnaires, one completed by students and one completed by a team of computer scientists. The results presented are favourable, with the majority of comments produced by the tool being seen to be as least as good as those generated by the computer science educators

A Rubric to Evaluate and Enhance Requirements Elicitation Interviewing Skills

Eliciting effective requirements is vital for successful Information Systems development and implementation. Interviews with stakeholders and users are an important part of the requirements elicitation process. Thus, teaching students how to better perform requirements elicitation interviews is a critical task for information systems faculty. However, prior to this research, a common tool or rubric to evaluate the effectiveness of requirements elicitation interviews was not found in the literature. The purpose of this research was to develop a rubric that can be used to both evaluate (provide summative measures) and enhance (via formative training techniques) the requirements elicitation interviewing skills of information systems students. The results of this research provide both quantitative and qualitative evidence that the rubric developed and described in this paper substantially improved the ability of our students to conduct requirements elicitation interviews. Along with detailing the various methodologies we used, this paper provides practical pedagogical suggestions and lessons learned along with covering possible future avenues of research in this area

Best Practices for Evaluating Flight Deck Interfaces for Transport Category Aircraft with Particular Relevance to Issues of Attention, Awareness, and Understanding CAST SE-210 Output 2 Report 6 of 6

Attention, awareness, and understanding of the flight crew are a critical contributor to safety and the flight deck plays a critical role in supporting these cognitive functions. Changes to the flight deck need to be evaluated for whether the changed device provides adequate support for these functions. This report describes a set of diverse evaluation methods. The report recommends designing the interface-evaluation to span the phases of the device development, from early to late, and it provides methods appropriate at each phase. It describes the various ways in which an interface or interface component can fail to support awareness as potential issues to be assessed in evaluation. It summarizes appropriate methods to evaluate different issues concerning inadequate support for these functions, throughout the phases of development

Improving self- and peer assessment processes with technology

Purpose - As a way of focusing curriculum development and learning outcomes universities have introduced graduate attributes, which their students should develop during their degree course. Some of these attributes are discipline-specific, others are generic to all professions. The development of these attributes can be promoted by the careful use of self- and peer assessment. The authors have previously reported using the self- and peer assessment software tool SPARK in various contexts to facilitate opportunities to practise, develop, assess and provide feedback on these attributes. This research and that of the other developers identified the need to extend the features of SPARK, to increase its flexibility and capacity to provide feedback. This paper seeks to report the results of the initial trials to investigate the potential of these new features to improve learning outcomes. Design/methodology/approach - The paper reviews some of the key literature with regard to self- and peer assessment, discusses the main aspects of the original online self- and peer assessment tool SPARK and the new version SPARKPLUS, reports and analyses the results of a series of student surveys to investigate whether the new features and applications of the tool have improved the learning outcomes in a large multi-disciplinary Engineering Design subject. Findings - It was found that using self- and peer assessment in conjunction with collaborative peer learning activities increased the benefits to students and improved engagement. Furthermore it was found that the new features available in SPARKPLUS facilitated efficient implementation of additional self- and peer assessment processes (assessment of individual work and benchmarking exercises) and improved learning outcomes. The trials demonstrated that the tool assisted in improving students' engagement with and learning from peer learning exercises, the collection and distribution of feedback and helping them to identify their individual strengths and weaknesses. Practical implications: SPARKPLUS facilitates the efficient management of self- and peer assessment processes even in large classes, allowing assessments to be run multiple times a semester without an excessive burden for the coordinating academic. While SPARKPLUS has enormous potential to provide significant benefits to both students and academics, it is necessary to caution that, although a powerful tool, its successful use requires thoughtful and reflective application combined with good assessment design. Originality/value - It was found that the new features available in SPARKPLUS efficiently facilitated the development of new self- and peer assessment processes (assessment of individual work and benchmarking exercises) and improved learning outcomes. © Emerald Group Publishing Limited

Improving learning throughput in e-learning using interactive-cognitive based assessment

Journal article published in The International Journal of E-Learning and Educational Technologies in the Digital Media (IJEETDM)An e-learning website is not sufficient to fully attain the results of online education. There also is a need to align the educational objectives into the design of the assessment to improve and develop cognition, critical thinking and problem-solving skills. Previous studies have explored the potentials of the assessment models but few ventured into their implementation. Others only proposed and introduced conceptual frameworks. The implementation of these proposals, however, revealed that the question type in the assessment phase neglected to align their questionnaire formats into a cognitive schema. At present, the standard multiple-choice question is the most frequently used of the question type of e-learning assessments. However, if this type is the only format adopted by e-learning developers, then the potentially rich and embedded assessment of the computer platform will be given up. This paper focuses on the design of assessment questions, which is created and guided by the hierarchical Bloom cognitive taxonomy and by utilizing rich media formats. Results conducted for eighteen weeks show a dramatic increase in the academic performance of the students. Likewise, digital transcripts converted from the collected perceptions after training undergoes sentiment analysis have correlated with the student improved academic throughput