Improving marking efficiency for longer programming solutions based on a semi-automated assessment approach

In recent years, many students in higher education have begun to learn programming languages. In doing so they will complete a variety of programming tasks of varying degrees of complexity. The students need to get consistent and personalised feedback to develop their programming skills. Human markers can provide personalised feedback using traditional manual approaches to assessment, but they may provide inconsistent feedback (especially for long programming solutions) since marking the programming solutions of multiple students can represent a significant workload for them. While full-automated assessment systems are the best to provide consistent feedback, they may not provide sufficiently personalised feedback for novice programmers. This study develops a novel semi-automated assessment approach in order to improve efficiency of human marker in the marking process and increase consistency of feedback (for both short and long programming solutions). It advocates the reuse of human marker’s comments for similar code snippets, defined as segmented marking in this study. New full and partial marking models are developed based on segmented marking and they are tested by expert markers. The findings show that the two models are similar in efficiency, but that a partial marking approach potentially offers an improved efficiency for longer programming solutions. Such a finding has significant potential to reduce time spent on marking throughout the sector, which would have significant impact on both resourcing and timeliness of feedback

Uniform Web-based learning environment for introductory programming course

Ohjelmoinnin massakursseilla käytetään usein erilaisia teknisiä työkaluja. Näitä ovat esimerkiksi oppimisenhallintajärjestelmät ja automaattista palautetta antavat työkalut. Nämä ovat kuitenkin monesti toisistaan irrallisia, mikä aiheuttaa oppilaille usein hämmennystä ja ylimääräistä työtä. Tässä diplomityössä vastaamme kysymyksiin: ''Miten pystymme rakentamaan yhtenäisen ohjelmoinnin oppimisympäristön? Mitä osia sen pitää sisältää ja miten näiden tulisi kommunikoida keskenään?''. Kysymyksiin vastataan tapaustutkimuksen kautta rakentamalla uusi oppimisympäristö Tampereen teknillisessä yliopistossa pidettävälle opintojaksolle Johdatus ohjelmointiin. Rakentamisen lähtökohtana on, että ympäristössä käytetään oppimisenhallintajärjestelmänä Aalto-yliopistossa kehitettyä A+:aa. Aluksi tutkimme vaatimuksia, joita opintojakso kohdistaa ympäristölle, ja määrittelemme komponentit, joista ympäristö rakentuu. Komponenttien tunnistamisen jälkeen hahmottelemme ympäristölle arkkitehtuurin käyttäen apuna sovellettua 4+1 -arkkitehtuurimallia ja tarkoitukseen sopivia tekniikoita, kuten REST. Tämän jälkeen tutustumme kirjallisuudesta löytyviin komponenttitoteutuksiin ja valitsemme niistä sopivimmat. Työn tuloksena on yhtenäinen oppimisympäristö, joka rakentuu A+-oppimisenhallintajärjestelmän ympärille. A+:aan on toteutettu REST-rajapinta, jonka kautta se kommunikoi oppilaan IDE:n ja ympäristön muiden komponenttien kanssa. Oppilaat pääsevät kaikkiin ominaisuuksiin käsiksi yhden kirjautumispisteen kautta ja mm. voivat palauttaa tehtävät suoraan IDE:stä. A+:aan tehdyt palautukset siirtyvät Mooc-grader--nimiselle komponentille, joka tarkistaa ne automatisoiduilla testeillä ja palauttaa tulokset A+:aan. Toteuttamamme oppimisympäristö helpottaa oppilaiden työskentelyä edelliseen kurssilla käytettyyn järjestelmään verrattuna. Enää opiskelijan ei tarvitse kirjautua järjestelmästä toiseen, eikä etsiä tehtävien palautuspaikkoja, vaan hän voi palauttaa tehtävän IDE:stä napin painalluksella

Automated assessment of programming assignments : visual feedback, assignment mobility, and assessment of students' testing skills

The main objective of this thesis is to improve the automated assessment of programming assignments from the perspective of assessment tool developers. We have developed visual feedback on functionality of students' programs and explored methods to control the level of detail in visual feedback. We have found that visual feedback does not require major changes to existing assessment platforms. Most modern platforms are web based, creating an opportunity to describe visualizations in JavaScript and HTML embedded into textual feedback. Our preliminary results on the effectiveness of automatic visual feedback indicate that students perform equally well with visual and textual feedback. However, visual feedback based on automatically extracted object graphs can take less time to prepare than textual feedback of good quality. We have also developed programming assignments that are easier to port from one server environment to another by performing assessment on the client-side. This not only makes it easier to use the same assignments in different server environments but also removes the need for sandboxing the execution of students' programs. The approach will likely become more important in the future together with interactive study materials becoming more popular. Client-side assessment is more suitable for self-studying material than for grading because assessment results sent by a client are often too easy to falsify. Testing is an important part of programming and automated assessment should also cover students' self-written tests. We have analyzed how students behave when they are rewarded for structural test coverage (e.g. line coverage) and found that this can lead students to write tests with good coverage but with poor ability to detect faulty programs. Mutation analysis, where a large number of (faulty) programs are automatically derived from the program under test, turns out to be an effective way to detect tests otherwise fooling our assessment systems. Applying mutation analysis directly for grading is problematic because some of the derived programs are equivalent with the original and some assignments or solution strategies generate more equivalent mutants than others

Núcleos de Avaliações Diagnóstica e Formativa para Regulação da Aprendizagem de Programação

NÚCLEOS DE AVALIAÇÕES DIAGNÓSTICA E FORMATIVA PARA REGULAÇÃO DA APRENDIZAGEM DE PROGRAMAÇÃ

Efficient Use of Teaching Technologies with Programming Education

Learning and teaching programming are challenging tasks that can be facilitated by using different teaching technologies. Visualization systems are software systems that can be used to help students in forming proper mental models of executed program code. They provide different visual and textual cues that help student in abstracting the meaning of a program code or an algorithm. Students also need to constantly practice the skill of programming by implementing programming assignments. These can be automatically assessed by other computer programs but parts of the evaluation need to be assessed manually by teachers or teaching assistants.There are a lot of existing tools that provide partial solutions to the practical problems of programming courses: visualizing program code, assessing student programming submissions automatically or rubrics that help keeping manual assessment consistent. Taking these tools into use is not straightforward. To succeed, the teacher needs to find the suitable tools and properly integrate them into the course infrastructure supporting the whole learning process. As many programming courses are mass courses, it is a constant struggle between providing sufficient personal guidance and feedback while retaining a reasonable workload for the teacher.This work answers to the question "How can the teaching of programming be effectively assisted using teaching technologies?" As a solution, different learning taxonomies are presented from Computer Science perspective and applied to visualization examples so the examples could be used to better support deeper knowledge and the whole learning process within a programming course. Then, different parts of the assessment process of programming assignments are studied to find the best practices in supporting the process, especially when multiple graders are being used, to maintain objectivity, consistency and reasonable workload in the grading.The results of the work show that teaching technologies can be a valuable aid for the teacher to support the learning process of the students and to help in the practical organization of the course without hindering the learning results or personalized feedback the students receive from their assignments. This thesis presents new visualization categories that allow deeper cognitive development and examples on how to integrate them efficiently into the course infrastructure. This thesis also presents a survey of computer-assisted assessment tools and assessable features for teachers to use in their programming assignments. Finally, the concept of rubric-based assessment tools is introduced to facilitate the manual assessment part of programming assignments