In this paper, we developed predictive models based on students’ reviewing behaviors in an Introductory Programming course. These patterns were captured using an educational technology that students used to review their graded paper- based assessments. Models were trained and tested with the goal of identifying students’ academic performance and those who might be in need of assistance. The results of the retrospective analysis show a reasonable accuracy. This suggests the possibility of developing interventions for students, such as providing feedback in the form of effective reviewing strategies

Azcona, David

Hsiao, I-Han

Smeaton, Alan F.

Vance Paredes, Yancy

Name not available

Predictive Modelling of Student Reviewing Behaviors in anIntroductory Programming CourseYancy Vance ParedesArizona State Universityyvmparedes@asu.eduDavid AzconaDublin City UniversityDavid.Azcona@insight-centre.orgI-Han HsiaoArizona State UniversitySharon.Hsiao@asu.eduAlan F. SmeatonDublin City UniversityAlan.Smeaton@insight-centre.orgABSTRACTIn this paper, we developed predictive models based on stu-dents’ reviewing behaviors in an Introductory Programmingcourse. These patterns were captured using an educationaltechnology that students used to review their graded paper-based assessments. Models were trained and tested withthe goal of identifying students’ academic performance andthose who might be in need of assistance. The results of theretrospective analysis show a reasonable accuracy. This sug-gests the possibility of developing interventions for students,such as providing feedback in the form of effective reviewingstrategies.KeywordsProgramming Learning, Educational Technology, EducationalData Mining, Predictive Modelling, Behavioral Analytics1. INTRODUCTIONIn Computer Science Education research, identifying stu-dents who may be having difficulties in programming coursesis important. Several studies have used students’ digitalfootprints, such as keystrokes, program edits, compilation,executions, or submissions obtained from a programminglearning environment [2, 9] to achieve this. Unfortunately,other sources, such as paper-based assessments, are oftenless explored because of their nature. Furthermore, captur-ing these data is quite challenging.In our research lab, we developed an educational technologyplatform, WebPGA1, to bridge this gap. Using this systemwe were able to collect data and conduct an initial investiga-tion. Based on the results of a subjective evaluation from a1https://cidsewpga.fulton.asu.eduprevious study [12], students were looking for a more person-alized reviewing experience. This could be achieved by usinga data-driven approach based on efficient reviewing patternsextracted from historical data of the student’s cohorts. Thispaper explores how to develop predictive models on thesescenario with two major goals. First, to identify studentswho might be in need of assistance; and, second, to predicttheir academic performance. By doing so, the suggestion ofpersonalized reviewing actions becomes feasible.2. LITERATURE REVIEW2.1 Behavioral Analytics in the ProgrammingLearningModelling of student’s programming learning is not a newtopic. These models reside in intelligent tutoring systems(ITS) or any adaptive educational systems. The learning ofthe students are normally estimated based on their interac-tions with the system (such as tutors) which then resultsto an update of the knowledge components. In program-ming learning, several parameters can be used to estimatea student’s knowledge in coding, such as sequence of pro-gramming problem solving success [5], programming assign-ments progression [13], assignment submission compilationbehavior [1], and generic Error Quotient measures [4]. Inthe learning analytics literature, Blikstein [3] has proposedan automatic analytic tool to access student’s learning inan open-ended environment, which considers a range of be-havioral analytics to predict learning, such as the amount ofcode changing, compilation behavior, and code editing.2.2 Multimodal Learning AnalyticsSeveral approaches have been adopted to integrate multi-modal learning analytics to gather comprehensive informa-tion about a class (i.e. collaborative confluence [11], mobiledevices [14], and EduAnalysis [8]). In addition to traditionalclick-stream logs derived from educational systems, thesemethods combine and present data from various sources, en-capsulating a wide range of learning activities which encodethe entire learning process. In a recent learning analyticshandbook, the state of the art of multimodal learning ana-lytics research is summarized. These involve the commonlycaptured and use data formats, such as textual, audio, spa-tial, visual and linguistics, etc.3. METHODOLOGY3.1 Research PlatformWebPGA was developed to connect the physical and dig-ital learning spaces in programming learning. A detaileddiscussion about its design can be found in [7]. Throughthis system, paper-based assessments can be digitized. Also,grading and distribution can be done online. Students canaccess their graded assessments virtually anytime and any-where. Whenever the student interacts with the system, allactions are captured. Examples of these actions include: log-ging in and out, selecting a question to review, time spentreviewing a question, bookmarking a question, navigatingthrough all the questions of an exam, and taking of notes.3.2 Data CollectionA classroom study was conducted in a Data Structures andAlgorithms course during the Fall 2016 semester. The in-structor administered a total of 3 exams and 13 quizzesthroughout the semester. Among the 13 quizzes, only 6were graded while 7 were for attendance only (full creditregardless of the correctness of answers). A total of 283 stu-dents were enrolled in the course. However, in this study,only 246 (86.93%) students were included as we excludedthose who dropped out of the course in the middle of thesemester, did not take the three exams, or did not use theresearch platform.In this study, review actions performed by students wereanalyzed. A review action is an event where a student ex-amines his or her graded answer. This includes reading thequestion, the answer, the assigned score and the feedbackprovided by the grader (see Fig. 1).3.3 Data ProcessingThe students were labelled according to their overall aca-demic performance. To achieve this, we computed for theaverage of the three exams of each student (see Fig. 2 forthe distribution). Using Jenks natural breaks classificationmethod [10], the break-point of 77.60% was obtained andused as a threshold to divide the students into two groups,namely high-performer and low-performer.4. PREDICTIVE MODELLING4.1 Student Digital FootprintA student’s digital footprint is shaped to combine the differ-ent modalities of student data and leverage that informationto analyse the behaviour of the students on these coursesusing Artificial Intelligence techniques. In this study, logscaptured by WebPGA as students browse through their dig-itized paper-based assessments have been leveraged to modeltheir reviewing behavior.First, a set of features was extracted based on the students’interaction within the system and the different actions theyperformed. For each assessment, the following were gath-ered: (1) their grades, (2) whether they reviewed the as-sessment or not, and (3) when they reviewed it for the firsttime. General engagement features were also collected. Thisincludes: (1) number of distinct days when the system wasused by each student, and (2) the number of interactionsor how many assessments were reviewed per student. TheTable 1: Feature correlations with the cumulativeexam averageFeature name Correlation coefficientNumber of assessments covered 34%*Average time to review −0.14%Distinct Days 28%*Distinct Actions 12%Number of web interactions 24%*Quiz 6 Mark 38%*Quiz 10 Mark 35%** p− value < 0.01predictive power of these features is measured by correlat-ing their values with the cumulative exam average (target).Table 1 shows a few of those features and the correspondingcorrelation coefficient. Students were more likely to obtain abetter score on the average exam grade as they (1) reviewedmore assessments; or (2) accessed the system regularly ondifferent days; or (3) obtained higher scores on their quizzes.On the other hand, the later the student reviewed the as-sessments (on average), the lesser the chance he or she hadin getting a high grade. This analysis confirmed the predic-tive power of our variables individually. In contrast, otherparameters such as whether the students reviewed specificassessments or how long it took them to review particularassessments were individually not correlated with the aver-age grade target.4.2 Classification and Regression ModellingA bag of learning algorithms were trained retrospectivelyusing the students’ digital footprint and their observed be-havioral patterns. We analyzed their power to predict thestudents’ performance and their generalizability. We usedcross-validation to train and test on the same dataset. Thetarget was to predict whether a student will score above orbelow the threshold of each cumulative exam average. Notethat the cumulative exam averages (our target for each pe-riod) include: (1) the first exam score before it took place,(2) an average between the first and second exam beforethe second exam, and (3) an average of the three exams be-fore the third exam was taken by students. The threshold,77.60%, was used to divide high and low performers, whichis derived in Section 3.3.The three exams divided the entire semester into four peri-ods, namely Before Exam1, Exam1-Exam2, Exam2-Exam3,and After Exam3. For each period, a learning function istrained to predict a student’s likelihood of scoring above orbelow the performance threshold on their cumulative aver-age grades. For instance, the first classifier was trained topredict the first exam outcome (above or below the thresh-old), the second one was to predict the average outcomebetween the first two exams, and so on. In a given period,the features mentioned above (Table 1) were extracted fromthe student’s interactions with the system along with theirreviewing patterns. A classifier was built by concatenatingall the features from previous assessments, such as scoresand reviewing times.Figure 1: Screenshot of what students see when they review a questionFigure 2: Distribution of students’ academic perfor-manceTable 2: Features per periodPeriod No. of features Students below thresholdBefore Exam1 16 86 (34.96%)Exam1-Exam2 34 93 (37.80%)Exam2-Exam3 52 89 (36.18%)After Exam3 55 89 (36.18%)Table 2 shows how more features were concatenated as stu-dents were being assessed throughout the semester. Thepercentage of students below the threshold was also checked.It shows that the two target classes (above and below) werebalanced.In terms of the features’ importance, their weights were plot-ted in Figure 3 using a heatmap. The general engagementfeatures, such as the number assessments reviewed by stu-dents or the number of distinct days students logged in, wereused individually in the model. Their weights were calcu-lated per period and plotted on the graph. In addition,features developed which were specific for each assessmentswere grouped (mark or score, whether they reviewed theseassessments and how early they reviewed them) into threesingle parameters: Mark, Reviewed and Time. Those threeparameters aggregated the importance for each of those cat-egories. For instance, the features that capture the timestudents review each assessment for the first time were clus-tered into one parameter, Time, that adds up the weightsof all of them for the importance graph. Therefore, Figure3 shows the following:Figure 3: Feature Importance Across Periods• Across all periods, these three parameters (the markof the assessment, the review patterns and the time toattend to review) consistently remained the key pre-dictors among all the classifiers.• The feature importance converged over time. Therewere more diverse predictors for the first classifier. Itcould possibly be due to the relatively fewer items thatcould be reviewed and/or students were learning howto use the system.• The parameters review patterns and the time to attendto review gradually increased their importance overtime until the third exam, which highlighted the na-ture of programming as accumulative. Students reliedon studying past assessments and attended to reviewthem sooner.• Another key parameter the mark of the assessmentgained importance from the first to the second examand loses it from the second to the third. It becomemore important how and when students had reviewedthan their previous scores in quizzes and exams.The empirical error minimization approach was employedto determine the learning algorithm with the fewest empir-ical error from a bag of classifiers C[6]. Cross-validationwas utilized to train and test the bag of classifiers using 10folds. Figure 4 shows a visual comparative analysis betweenclassifiers using the Receiver Operating Characteristic AreaUnder the Curve (ROC AUC), a well-known metric to eval-uate binary classifiers, and the number for each classifier oneach period is an average of the metric per fold. In addition,Table 3 shows the chosen learning algorithm, SVM with alinear kernel, and the results for the weighted average preci-sion and F1-metric, which combines precision and recall, foreach period. The values for the metrics shown are the meanand the standard deviation for the cross validation folds.In addition, a regression model was built to predict the pre-cise cumulative exam average grade for each students onthese periods. In a similar manner, a linear regression func-tions was constructed retrospectively per period. Cross-validation was employed using 10 folds. The performanceof the linear regression function can be found in Figure 5.Table 6 shows the means and the standard deviation of thefolds for each period using the Coefficient of DeterminationFigure 4: Classification Performance using ROCAUCTable 3: Linear SVM Classification Performancethroughout the periodsPeriod Precision F1-scoreMean (SD) Mean (SD)Before Exam1 70.65% (7.89%) 76% (2.76%)Exam1-Exam2 93.03% (4.45%) 84.66% (6.91%)Exam2-Exam3 96.55% (2.92%) 90.84% (4.58%)After Exam3 100% (0%) 95.59% (3.51%)Figure 5: Linear Regression Performance using R2Table 4: Linear Regression Performance throughoutthe periodsPeriod R2 MAEMean (SD) Mean (SD)Before Exam1 -0.1411 (0.2231) 0.0843 (0.0074)Exam1-Exam2 0.6404 (0.1221) 0.0476 (0.0099)Exam1-Exam3 0.8752 (0.0495) 0.0266 (0.0049)After Exam3 1.0 (0.0) 0.0 (0.0)Figure 6: Linear Regression Predictions vs. ActualResults before the third exam(R2) and the Mean Absolute Error (MAE). In Figure 6,the predicted cumulative target grades were plotted withrespect to the actual results for each of the students beforethe third exam period.5. CONCLUSIONSThis retrospective analysis managed to demonstrate (1) thegathering of student data, which tells us their reviewinglearning process; (2) the extraction of features; and (3) theidentification of patterns which could be used for predic-tions. These predictions reached a usable accuracy for po-tential interventions and feedback to students. Both mod-els worked well and increased their performance every weekand period as students completed and reviewed more as-sessments and more timing and engagement features wereextracted.After the last exam was finished, the three exam scores couldbe utilized to calculate the average exam score and, there-fore, the classification and regression models made no mis-take. It is now possible to leverage the patterns extractedfrom this reviewing data to predict how a new incoming co-hort of students will perform in next versions of this course,intervene and provide personalized help to those in need tofollow desired reviewing strategies.6. REFERENCES[1] A. Altadmri and N. C. Brown. 37 million compilations:Investigating novice programming mistakes inlarge-scale student data. In Proceedings of the 46thACM Technical Symposium on Computer ScienceEducation, pages 522–527, NY, US, 2015. ACM.[2] D. Azcona and A. F. Smeaton. Targeting at-riskstudents using engagement and effort predictors in anintroductory computer programming course. InEuropean Conference on Technology EnhancedLearning, pages 361–366, NY, USA, 2017. Springer.[3] P. Blikstein. Using learning analytics to assessstudents’ behavior in open-ended programming tasks.In Proceedings of the 1st international conference onlearning analytics and knowledge, pages 110–116, NY,US, 2011. ACM.[4] A. S. Carter, C. D. Hundhausen, and O. Adesope. Thenormalized programming state model: Predictingstudent performance in computing courses based onprogramming behavior. In Proceedings of the eleventhannual International Conference on InternationalComputing Education Research, pages 141–150, NY,US, 2015. ACM.[5] J. Guerra, S. Sahebi, Y.-R. Lin, and P. Brusilovsky.The problem solving genome: Analyzing sequentialpatterns of student work with parameterized exercises.In Educational Data Mining 2014, North Carolina,US, 2014. EDM.[6] L. Gyorfi, L. Devroye, and G. Lugosi. A probabilistictheory of pattern recognition, 1996.[7] I.-H. Hsiao, P.-K. Huang, and H. Murphy. Uncoveringreviewing and reflecting behaviors from paper-basedformal assessment. In Proceedings of the SeventhInternational Learning Analytics & KnowledgeConference, pages 319–328, NY, US, 2017. ACM.[8] I.-H. Hsiao and Y.-L. Lin. Enriching programmingcontent semantics: An evaluation of visual analyticsapproach. Computers in Human Behavior, 72:771–782,2017.[9] P. Ihantola, A. Vihavainen, A. Ahadi, M. Butler,J. Borstler, S. H. Edwards, E. Isohanni, A. Korhonen,A. Petersen, K. Rivers, et al. Educational data miningand learning analytics in programming: Literaturereview and case studies. In Proceedings of the 2015ITiCSE on Working Group Reports, pages 41–63, NY,USA, 2015. ACM.[10] G. F. Jenks. The data model concept in statisticalmapping. International yearbook of cartography,7:186–190, 1967.[11] R. Martinez-Maldonado, Y. Dimitriadis,A. Martinez-Mone´s, J. Kay, and K. Yacef. Capturingand analyzing verbal and physical collaborativelearning interactions at an enriched interactivetabletop. International Journal of Computer-SupportedCollaborative Learning, 8(4):455–485, 2013.[12] Y. V. Paredes, P.-K. Huang, H. Murphy, and I.-H.Hsiao. A subjective evaluation of web-basedprogramming grading assistant: Harnessing digitalfootprints from paper-based assessments. In CEURWorkshop Proceedings, volume 1828, pages 23–30,2017.[13] C. Piech, M. Sahami, D. Koller, S. Cooper, andP. Blikstein. Modeling how students learn to program.In Proceedings of the 43rd ACM technical symposiumon Computer Science Education, pages 153–160, NY,US, 2012. ACM.[14] K. VanLehn, S. Cheema, J. Wetzel, and D. Pead.Some less obvious features of classroom orchestrationsystems. In Educational Technologies: Challenges,Applications and Learning Outcomes. Nova SciencePublishers, Inc., NY, US, 2016.

Predictive modelling of student reviewing behaviors in an introductory programming course

Irish Universities

DCU Online Research Access Service

http://doras.dcu.ie/22540/1/Predictive%20Modelling%20of%20Student%20Reviewing%20Behaviors%20in%20an%20Introductory%20Programming%20Course.pdf

Predictive modelling of student reviewing behaviors in an introductory programming course

Abstract

Similar works

Full text

Available Versions

Name not available

Irish Universities

DCU Online Research Access Service