Tracking Students' Internet Browsing in a Machine Exam

Traditionally, introductory computer science courses have focused on teaching programming, and have not included teaching information retrieval skills. However, a large part of a programmer's time is spent looking at documentation or browsing the internet for guidance on how to solve the small subtasks that programming often consists of or which library to use for a specific need. We have developed a browser-plugin that tracks how students use online resources during a machine exam. Such a tool could be used -- for example -- to detect whether there is a difference between the browsing behavior of high- and low-performing students. To this end, we conduct a case study with the tool where we examine students' browsing in a lab-based programming exam. In the future, the tool could be used to examine students' browsing and possibly inform decisions on how to teach information retrieval skills to students.Peer reviewe

Does Creating Programming Assignments with Tests Lead to Improved Performance in Writing Unit Tests?

We have constructed a tool, CrowdSorcerer, in which students create programming assignments, their model solutions and associated test cases using a simple input-output format. We have used the tool as a part of an introductory programming course with normal course activities such as programming assignments and a final exam. In our work, we focus on whether creating programming assignments and associated tests correlate with students' performance in a testing-related exam question. We study this through an analysis of the quality of student-written tests within the tool, measured using the number of test cases, line coverage and mutation coverage, and students' performance in testing related exam question, measured using exam points. Finally, we study whether previous programming experience correlates with how students act within the tool and within the testing related exam question.Peer reviewe

Identification based on typing patterns between programming and free text

Identifying people based on their typing has been studied successfully in multiple different contexts. Previous research has shown that identification is possible based on writing predetermined texts such as typing passwords, free text such as essays, as well based on writing source code. In this work, we study typing pattern based identification when the text format and writing environment change. We replicate two earlier studies which suggested that typing profile identification works with programming data, and that it can be applied to a programming exam circumstances with decent results. Then, we examine how the identification accuracy changes when the user profiles are built using data from programming, and the identification is conducted on data from writing free text. Our results show that the identification accuracy is indeed high within the context of programming data, but drops when identifying essay typists based on typing profiles built from their programming data.Peer reviewe

Exploring the Applicability of Simple Syntax Writing Practice for Learning Programming

When learning programming, students learn the syntax of a programming language, the semantics underlying the syntax, and practice applying the language in solving programming problems. Research has suggested that simply the syntax may be hard to learn. In this article, we study difficulty of learning the syntax of a programming language. We have constructed a tool that provides students code that they write character-by-character. When writing, the tool automatically highlights each character in code that is incorrectly typed, and through the highlight-based feedback directs students into writing correct syntax. We conducted a randomized controlled trial in an introductory programming course organized in Java. One half of the population had the tool in the course material immediately before programming exercises where the practiced syntax was used, while the other half of the course population did not have the tool, thus approaching the exercises in a traditional way. Our results imply that isolated syntax writing practice may not be a meaningful addition to the arsenal used for teaching programming, at least when the programming course utilizes a large set of small programming exercises. We encourage researchers to replicate our work in contexts where syntax seems to be an issue.Peer reviewe

Analysis of Students' Peer Reviews to Crowdsourced Programming Assignments

We have used a tool called CrowdSorcerer that allows students to create programming assignments. The students are given a topic by a teacher, after which the students design a programming assignment: the assignment description, the code template, a model solution and a set of input-output -tests. The created assignments are peer reviewed by other students on the course. We study students' peer reviews to these student-generated assignments, focusing on examining the differences between novice and experienced programmers. We then analyze whether the exercises created by experienced programmers are rated better quality-wise than those created by novices. Additionally, we investigate the differences between novices and experienced programmers as peer reviewers: can novices review assignments as well as experienced programmers?Peer reviewe

Experimenting with Model Solutions as a Support Mechanism

We describe an experiment from an introductory programming course where we provided students an opportunity to access model solutions of programming assignments they have not yet completed. Access to model solutions was controlled with coins, which students collected by completing programming assignments. The quantity of coins was limited so that students could buy solutions to at most one tenth of the course assignments. When compared to the traditional approach where access to model solutions is limited to only after the assignment is completed or the assignment deadline has passed, students seemed to enjoy the opportunity more and collecting coins motivated some students to complete more assignments. Collected coins were mostly used close to deadlines and on more difficult assignments. Overall, the use of coins and model solutions may be a viable option to providing students additional support. Data from the use of coins and model solutions could also be used to identify students who could benefit from additional guidance.Peer reviewe

Crowdsourcing Programming Assignments with CrowdSorcerer

Small automatically assessed programming assignments are an often used resource for learning programming. Creating sufficiently large amounts of such assignments is, however, time consuming. As a consequence, offering large quantities of practice assignments to students is not always possible. CrowdSorcerer is an embeddable open-source system that students and teachers alike can use for creating and evaluating small automatically assessed programming assignments. While creating programming assignments, the students also write simple input-output -tests, and are gently introduced to the basics of testing. Students can also evaluate the assignments of others and provide feedback on them, which exposes them to code written by others early in their education. In this article we both describe the CrowdSorcerer system and our experiences in using the system in a large undergraduate programming course. Moreover, we discuss the motivation for crowdsourcing course assignments and present some usage statistics.Peer reviewe

Identification of Programmers from Typing Patterns

Being able to identify the user of a computer solely based on their typing patterns can lead to improvements in plagiarism detection, provide new opportunities for authentication, and enable novel guidance methods in tutoring systems. However, at the same time, if such identification is possible, new privacy and ethical concerns arise. In our work, we explore methods for identifying individuals from typing data captured by a programming environment as these individuals are learning to program. We compare the identification accuracy of automatically generated user profiles, ranging from the average amount of time that a user needs between keystrokes to the amount of time that it takes for the user to press specific pairs of keys, digraphs. We also explore the effect of data quantity and different acceptance thresholds on the identification accuracy, and analyze how the accuracy changes when identifying individuals across courses. Our results show that, while the identification accuracy varies depending on data quantity and the method, identification of users based on their programming data is possible. These results indicate that there is potential in using this method, for example, in identification of students taking exams, and that such data has privacy concerns that should be addressed.Peer reviewe