Creative Assessment in Programming: Diversity and Divergence

Negative stereotypes persist in computing, and align poorly with research into the motivations of female students. In particular, female students are more inclined to want to work creatively and have a positive impact through their work. However programming assignments are often tightly constrained and rather pointless in themselves so are doubly unattractive. Alongside this, concerns are often raised about plagiarism in programming assignments, particularly when the assessment process is automated. We attempt to address both of these issues by designing more creative programming assignments, allowing students to engage in work aligned with whatever their interests are. By providing a more divergent assessment, automated plagiarism detectors are much more effective because the likelihood of false positives is much lower than in more constrained, convergent assessments. We also show how to combine this with partial automation of assessment. To examine this approach we compare the results of two subsequent years of delivery of the same second-year undergraduate programming module, and find that, using more creative assessments, female students average scores were substantially increased so that they outperform male students. While the results are not quite statistically significant (according to 2-way ANOVA), they demonstrate potential that could be verified with a larger sample

Choosing Code Segments to Exclude from Code Similarity Detection

When student programs are compared for similarity as a step in the detection of academic misconduct, certain segments of code are always sure to be similar but are no cause for suspicion. Some of these segments are boilerplate code (e.g. public static void main String [] args) and some will be code that was provided to students as part of the assessment specification. This working group explores these and other types of code that are legitimately common in student assessments and can therefore be excluded from similarity checking. From their own institutions, working group members collected assessment submissions that together encompass a wide variety of assessment tasks in a wide variety of programming languages. The submissions were analysed to determine what sorts of code segment arose frequently in each assessment task. The group has found that common code can arise in programming assessment tasks when it is required for compilation purposes; when it reflects an intuitive way to undertake part or all of the task in question; when it can be legitimately copied from external sources; and when it has been suggested by people with whom many of the students have been in contact. A further finding is that the nature and size of the common code fragments vary with course level and with task complexity. An informal survey of programming educators confirms the group's findings and gives some reasons why various educators include code when setting programming assignments.Peer reviewe

TF-IDF Inspired Detection for Cross-Language Source Code Plagiarism and Collusion

Several computing courses allow students to choose which programming language they want to use for completing a programming task. This can lead to cross-language code plagiarism and collusion, in which the copied code file is rewritten in another programming language. In response to that, this paper proposes a detection technique which is able to accurately compare code files written in various programming languages, but with limited effort in accommodating such languages at development stage. The only language-dependent feature used in the technique is source code tokeniser and no code conversion is applied. The impact of coincidental similarity is reduced by applying a TF-IDF inspired weighting, in which rare matches are prioritised. Our evaluation shows that the technique outperforms common techniques in academia for handling language conversion disguises. Further, it is comparable to those techniques when dealing with conventional disguises

Mining Student Submission Information to Refine Plagiarism Detection

Plagiarism is becoming an increasingly important issue in introductory programming courses. There are several tools to assist with plagiarism detection, but they are not effective for more basic programming assignments, like those in introductory courses. The proliferation of auto-grading platforms creates an opportunity to capture additional information about how students develop the solutions to their programming assignments. In this research, we identify how to extract information from an online autograding platform, Mimir Classroom, that can be useful in revealing patterns in solution development. We explore how and to what extent this additional information can be used to better support instructors when identifying cases of probable plagiarism. We have developed a tool that takes the raw student assignment submissions from Mimir, analyzes them, and produces data sets and visualizations that help instructors to refine information extracted by existing plagiarism detection platforms. The instructors can then take this information to further investigate any probable cases of plagiarism that have been found by the tool. Our main goal is to give insight into student behaviors and identify signals that can be effective indicatives of plagiarism. Furthermore, the framework can enable the analysis of other aspects of students’ solution development processes that may be useful when reasoning about their learning. As an initial exploration scenario of the framework developed in this work, we have used student code submissions from the CSCE 121: Introduction to Program Design and Concepts course at Texas A&M University. We experimented with the student code submissions from the Fall 2018 and Fall 2019 offerings of the course

Source code authorship attribution

To attribute authorship means to identify the true author among many candidates for samples of work of unknown or contentious authorship. Authorship attribution is a prolific research area for natural language, but much less so for source code, with eight other research groups having published empirical results concerning the accuracy of their approaches to date. Authorship attribution of source code is the focus of this thesis. We first review, reimplement, and benchmark all existing published methods to establish a consistent set of accuracy scores. This is done using four newly constructed and significant source code collections comprising samples from academic sources, freelance sources, and multiple programming languages. The collections developed are the most comprehensive to date in the field. We then propose a novel information retrieval method for source code authorship attribution. In this method, source code features from the collection samples are tokenised, converted into n-grams, and indexed for stylistic comparison to query samples using the Okapi BM25 similarity measure. Authorship of the top ranked sample is used to classify authorship of each query, and the proportion of times that this is correct determines overall accuracy. The results show that this approach is more accurate than the best approach from the previous work for three of the four collections. The accuracy of the new method is then explored in the context of author style evolving over time, by experimenting with a collection of student programming assignments that spans three semesters with established relative timestamps. We find that it takes one full semester for individual coding styles to stabilise, which is essential knowledge for ongoing authorship attribution studies and quality control in general. We conclude the research by extending both the new information retrieval method and previous methods to provide a complete set of benchmarks for advancing the field. In the final evaluation, we show that the n-gram approaches are leading the field, with accuracy scores for some collections around 90% for a one-in-ten classification problem

Yours, mine, and ours : confronting the originality burden through remix and intertextuality.

This dissertation contributes to ongoing conversations regarding the goal of composition instructors “to empower students to take responsibility for their ideas and their texts while developing their curiosity and persistence in the pursuit of knowledge” (Carpenter, 2014, par. 4). In particular, this dissertation, a classroom ethnography, examines how the originality burden-an encumbrance wherein students feel overwhelmed by the need to write an “original” paper-operates in one second-semester first-year composition course dedicated to relieving students from feeling like they must write “original” texts. More specifically, this study examines the potential of two concepts, remix and intertextuality, to help show students that writing, and language more generally, always builds on what came before, therefore reducing the possibility that any text is truly original. This dissertation begins with an overview and literature review of what a term like originality means within the context of a first-year writing course, acknowledging the cultural history that influences how students understand originality (including the development of the solitary author, copyright law, and plagiarism) and the way that digital media has come to change what it means to author an “original text.” Chapter 2 outlines the methodology of the study, describing how the study site was selected, the data collection procedures used, and the data sources. Chapters 3 and 4 report the results of my research. In Chapter 3, I focus on how the instructor of the course used the term remix to explain to her students the ways in which language and writing are intertextual. This chapter describes how students used the term remix as a qualifier for the kind of writing they produced, rather than as what all writing could be labeled. That is, Chapter 3 discusses why students felt that their academic texts were remixed texts but that texts produced by more experienced writers, such as their professors, were original. In Chapter 4, I examine how an intertextual practice many students were familiar with before entering the composition classroom, the digital remix, helped alleviate the originality burden while at the same time creating a greater disconnect between digital remixed writing and more traditional academic remixed writing. Finally, Chapter 5 describes the theoretical and pedagogical implications of my findings, the limitations of this research project, and areas for future research

Computer-based assessment system for e-learning applied to programming education

Tese de Mestrado Integrado. Engenharia Informática e Computação. Faculdade de Engemharia. Universidade do Porto. 201

StyleCounsel: Seeing the (Random) Forest for the Trees in Adversarial Code Stylometry

Authorship attribution has piqued the interest of scholars for centuries, but had historically remained a matter of subjective opinion, based upon examination of handwriting and the physical document. Midway through the 20th Century, a technique known as stylometry was developed, in which the content of a document is analyzed to extract the author's grammar use, preferred vocabulary, and other elements of compositional style. In parallel to this, programmers, and particularly those involved in education, were writing and testing systems designed to automate the analysis of good coding style and best practice, in order to assist with grading assignments. In the aftermath of the Morris Worm incident in 1988, researchers began to consider whether this automated analysis of program style could be combined with stylometry techniques and applied to source code, to identify the author of a program. The results of recent experiments have suggested this code stylometry can successfully identify the author of short programs from among hundreds of candidates with up to 98\% precision. This potential ability to discern the programmer of a sample of code from a large group of possible authors could have concerning consequences for the open-source community at large, particularly those contributors that may wish to remain anonymous. Recent international events have suggested the developers of certain anti-censorship and anti-surveillance tools are being targeted by their governments and forced to delete their repositories or face prosecution. In light of this threat to the freedom and privacy of individual programmers around the world, and due to a dearth of published research into practical code stylometry at scale and its feasibility, we carried out a number of investigations looking into the difficulties of applying this technique in the real world, and how one might effect a robust defence against it. To this end, we devised a system to aid programmers in obfuscating their inherent style and imitating another, overt, author's style in order to protect their anonymity from this forensic technique. Our system utilizes the implicit rules encoded in the decision points of a random forest ensemble in order to derive a set of recommendations to present to the user detailing how to achieve this obfuscation and mimicry attack. In order to best test this system, and simultaneously assess the difficulties of performing practical stylometry at scale, we also gathered a large corpus of real open-source software and devised our own feature set including both novel attributes and those inspired or borrowed from other sources. Our results indicate that attempting a mass analysis of publicly available source code is fraught with difficulties in ensuring the integrity of the data. Furthermore, we found ours and most other published feature sets do not sufficiently capture an author's style independently of the content to be very effective at scale, although its accuracy is significantly greater than a random guess. Evaluations of our tool indicate it can successfully extract a set of changes that would result in a misclassification as another user if implemented. More importantly, this extraction was independent of the specifics of the feature set, and therefore would still work even with a more accurate model of style. We ran a limited user study to assess the usability of the tool, and found overall it was beneficial to our participants, and could be even more beneficial if the valuable feedback we received were implemented in future work