The Automatic Assessment of Multiple Artefacts: An Investigation into Design Diagrams and Their Implementations

As the Higher Education sector has moved towards student-centred learning so too has the growth in electronic support for learning. E-assessment has been a part of this growth as increasingly assessment and its feedback is seen as an integral part of the students’ learning process. Mature e-assessment systems exist, particularly where answers to questions are restricted to a prescribed list of alternatives. However, for free response artefacts, where there is a limited restriction placed on answers to questions, automated assessment systems are embryonic. This dissertation presents an investigation into the automated assessment of free response artefacts. Design diagrams and their accompanying source code implementations are examples of free response artefacts. A case study is developed that investigates how to automatically generate formative feedback for a design diagram by utilizing its accompanying implementation. The dissertation presents a two-staged solution, initially analysing the design diagram in isolation before comparing it with the implementation. A framework for this approach has been developed and tested using a tool applied to coursework submitted by undergraduate computer science students. The tool was evaluated by comparing the formative feedback comments generated by the tool with those produced by a team of computer science educators. Evaluation was undertaken via two Likert questionnaires, one completed by students and one completed by a team of computer scientists. The results presented are favourable, with the majority of comments produced by the tool being seen to be as least as good as those generated by the computer science educators

Exploring Feedback and Gamification in a Data Modeling Learning Tool

Data modeling is an essential part of IT studies. Learning how to design and structure a database is important when storing data in a relational database and is common practice in the IT industry. Most students need much practice and tutoring to master the skill of data modeling and database design. When a student is in a learning process, feedback is important. As class sizes grow and teaching is no longer campus based only, providing feedback to each individual student may be difficult. Our study proposes a tool to use when introducing database modeling to students. We have developed a web-based tool named LearnER to teach basic data modeling skills, in a collaborative project between the University of South-Eastern Norway (USN) and Kristiania University College (KUC). The tool has been used in six different courses over a period of four academic years. In LearnER, the student solves modeling assignments with different levels of difficulty. When they are done, or they need help, they receive automated feedback including visual cues. To increase the motivation for solving many assignments, LearnER also includes gamifying elements. Each assignment has a maximum score. When students ask for help, points are deducted from the score. When students manage to solve many assignments with little help, they may end up at a leaderboard. This paper tries to summarize how the students use and experience LearnER. We look to see if the students find the exercises interesting, useful and of reasonable difficulty. Further, we investigate if the automated feedback is valuable, and if the gamifying elements contribute to their learning. As we have made additions and refinements to LearnER over several years, we also compare student responses on surveys and interviews during these years. In addition, we analyze usage data extracted from the application to learn more about student activity. The results are promising. We find that student activity increases in newer versions of LearnER. Most students report that the received feedback helps them to correct mistakes when solving modeling assignments. The gamifying elements are also well received. Based on LearnER usage data, we find and describe typical errors the students do and what types of assignments they prefer to solve.publishedVersio

Personalised feedback with semi-automated assessment tool for conceptual database model

The increased presence of diagram-type student work in higher education has recently attracted researchers to look into the automation of diagram marking. This paper introduces web-based diagram drawing and marking tools for a new (semi-automatic) assessment approach. The approach reduces the number of diagram components marked by the human marker and provides individualised and detailed feedback to students. The tools which have been used in tutorials of a first year database module in the Computer Science department at Loughborough University are described together with findings from the usage of the tools

New method for summative evaluation of UML class diagrams based on graph similarities

This paper deals with the problem of the evaluation of the student's productions during the construction of a UML class diagram from textual speciations, which can be a tedious task for teachers. The main objective is to propose a method of summative and semi-automatic evaluation of the class diagrams produced by the students, in order to provide an educational reaction on the learning process, and to reduce the evaluation work for the teachers. To achieve this objective, we must analyze these productions and study the transformation, matching, similarity measurement and comparison of several UML graphs. From this study, we adopted a method based on the comparison and matching of the components of several UML diagrams. This proposal is applied to evaluate UML class diagrams and focuses on the structural and semantic aspects of the UML graph produced by students compared to several solutions proposed by the teacher

Automatic assessment of sequence diagrams

In previous work we showed how student-produced entity-relationship diagrams (ERDs) could be automatically marked with good accuracy when compared with human markers. In this paper we report how effective the same techniques are when applied to syntactically similar UML sequence diagrams and discuss some issues that arise which did not occur with ERDs. We have found that, on a corpus of 100 student-drawn sequence diagrams, the automatic marking technique is more reliable that human markers. In addition, an analysis of this corpus revealed significant syntax errors in student-drawn sequence diagrams. We used the information obtained from the analysis to build a tool that not only detects syntax errors but also provides feedback in diagrammatic form. The tool has been extended to incorporate the automatic marker to provide a revision tool for learning how to model with sequence diagrams

Comparison of Required Competences and Task Material in Modeling Education

The reform of the European academic landscape with the introduction of bachelor\u27s and master\u27s degree programs has brought about several profound changes for teaching and assessment in higher education. With regard to the examination system, the shift towards output-oriented teaching is still one of the most significant challenges. Assessments have to be integrated into the teaching and learning arrangements and consistently aligned towards the intended learning outcomes. In particular, assessments should provide valid evidence that learners have acquired competences that are relevant for a specific domain. However, it seems that this didactic goal has not yet been fully achieved in modeling education in computer science. The aim of this study is to investigate whether typical task material used in exercises and exams in modeling education at selected German universities covers relevant competences required for graphical modeling. For this purpose, typical tasks in the field of modeling are first identified by means of a content-analytical procedure. Subsequently, it is determined which competence facets relevant for graphical modeling are addressed by the task types. By contrasting a competence model for modeling with the competences addressed by the tasks, a gap was identified between the required competences and the task material analyzed. In particular, the gap analysis shows the neglect of transversal competence facets as well as those related to the analysis and evaluation of models. The result of this paper is a classification of task types for modeling education and a specification of the competence facets addressed by these tasks. Recommendations for developing and assessing student\u27s competences comprehensively are given

Drawing and marking graph diagrams

The marking of graph diagrams (that is to say diagrams that are composed of nodes, possibly joined by edges) is tedious if the diagrams are presented on paper. If the key content of the diagrams is available in electronic form then the marking can be much more efficient. This is achieved because the tutor only has to mark each different diagram element once and this mark is transmitted to all diagrams that contain the element. This benefit to the tutor is obtained by requiring the students to use a diagram drawing program of some kind. However using such an editor can simplify the process for the students by allowing them to concentrate more on the problem and less on its graphical representation. The students can also be rewarded for going to this extra effort by receiving a much more detailed, personalised commentary on their work than would have been possible before, given the same amount of tutor time. We present the evolution of a drag-and-drop diagram editor specialised for the area of ER diagrams and an associated marking system with a simple but effective feedback mechanism. Some results from initial trials are presented along with some ideas for improvement and extension

A Comparative Analysis of Graph Vs Relational Database For Instructional Module Development System

abstract: In today's data-driven world, every datum is connected to a large amount of data. Relational databases have been proving itself a pioneer in the field of data storage and manipulation since 1970s. But more recently they have been challenged by NoSQL graph databases in handling data models which have an inherent graphical representation. Graph databases with the ability to store physical relationships between two nodes and native graph processing technique have been doing exceptionally well in graph data storage and management for applications like recommendation engines, biological modeling, network modeling, social media applications, etc. Instructional Module Development System (IMODS) is a web-based software system that guides STEM instructors through the complex task of curriculum design, ensures tight alignment between various components of a course (i.e., learning objectives, content, assessments), and provides relevant information about research-based pedagogical and assessment strategies. The data model of IMODS is highly connected and has an inherent graphical representation between all its entities with numerous relationships between them. This thesis focuses on developing an algorithm to determine completeness of course design developed using IMODS. As part of this research objective, the study also analyzes the data model for best fit database to run these algorithms. As part of this thesis, two separate applications abstracting the data model of IMODS have been developed - one with Neo4j (graph database) and another with PostgreSQL (relational database). The research objectives of the thesis are as follows: (i) evaluate the performance of Neo4j and PostgreSQL in handling complex queries that will be fired throughout the life cycle of the course design process; (ii) devise an algorithm to determine the completeness of a course design developed using IMODS. This thesis presents the process of creating data model for PostgreSQL and converting it into a graph data model to be abstracted by Neo4j, creating SQL and CYPHER scripts for undertaking experiments on both platforms, testing and elaborate analysis of the results and evaluation of the databases in the context of IMODS.Dissertation/ThesisMasters Thesis Computer Science 201