Predicting Performance in an Introductory Programming Course by Logging and Analyzing Student Programming Behavior

The high failure rates of many programming courses means there is a need to identify struggling students as early as possible. Prior research has focused upon using a set of tests to assess the use of a student's demographic, psychological and cognitive traits as predictors of performance. But these traits are static in nature, and therefore fail to encapsulate changes in a student's learning progress over the duration of a course. In this paper we present a new approach for predicting a student's performance in a programming course, based upon analyzing directly logged data, describing various aspects of their ordinary programming behavior. An evaluation using data logged from a sample of 45 programming students at our University, showed that our approach was an excellent early predictor of performance, explaining 42.49% of the variance in coursework marks - double the explanatory power when compared to the closest related technique in the literature

Early identification of novice programmers' challenges in coding using machine learning techniques

It is well known that many first year undergraduate university students struggle with learning to program. Educational Data Mining (EDM) applies machine learning and statistics to information generated from educational settings. In this PhD project, EDM is used to study first semester novice programmers, using data collected from students as they work on computers to complete their normal weekly laboratory exercises. Analysis of the generated snapshots has shown the potential for early identification of students who later struggle in the course. The aim of this study is to propose a method for early identification of "at risk" students while providing suggestions on how they can improve their coding style. This PhD project is within its final year

A Mathematical Model of a Course Performance Index to Measure Improvements in Students’ Soft Skills

In this paper, a method to assess engineering students’ performance after humanity course is introduced. Mathematical model for a student improvement index is derived to quantify a student’s improvements in some learning parameters related to the course. Input to the index is the data obtained from a survey given to students before the end of the course. Statements in the survey can be given different weights according to their importance. A numerical example on how to calculate improvement index is represented. A course performance index is introduced as well to measure how all students in the course achieved in comparison with previous or target performance. A case study in which a survey can be given to undergraduate engineering students before the end of a course about oral presentation skills is introduced as an application for the proposed models. The second index can also be used by the institution to measure the quality of the learning process through the course in certain semester. The proposed approach has the advantage of being of almost no additional cost and can be modified and applied to other courses as well. Also, this approach needs moderate use of Microsoft Excel and doesn’t need sophisticated academic analytics or learning management system to be owned by the institution

Syntax error based quantification of the learning progress of the novice programmer

© 2018 Association for Computing Machinery. Recent data-driven research has produced metrics for quantifying a novice programmer’s error profile, such as Jadud’s error quotient. However, these metrics tend to be context dependent and contain free parameters. This paper reviews the caveats of such metrics and proposes a more general approach to developing a metric. The online implementation of the proposed metric is publicly available at http://online-analysis-demo.herokuapp.com/

No Tests Required: Comparing Traditional and Dynamic Predictors of Programming Success

Research over the past fifty years into predictors of programming performance has yielded little improvement in the identification of at-risk students. This is possibly because research to date is based upon using static tests, which fail to reflect changes in a student's learning progress over time. In this paper, the effectiveness of 38 traditional predictors of programming performance are compared to 12 new data-driven predictors, that are based upon analyzing directly logged data, describing the programming behavior of students. Whilst few strong correlations were found between the traditional predictors and performance, an abundance of strong significant correlations based upon programming behavior were found. A model based upon two of these metrics (Watwin score and percentage of lab time spent resolving errors) could explain 56.3% of the variance in coursework results. The implication of this study is that a student's programming behavior is one of the strongest indicators of their performance, and future work should continue to explore such predictors in different teaching contexts

No tests required : comparing traditional and dynamic predictors of programming success.

Research over the past fifty years into predictors of programming performance has yielded little improvement in the identification of at-risk students. This is possibly because research to date is based upon using static tests, which fail to reflect changes in a student's learning progress over time. In this paper, the effectiveness of 38 traditional predictors of programming performance are compared to 12 new data-driven predictors, that are based upon analyzing directly logged data, describing the programming behavior of students. Whilst few strong correlations were found between the traditional predictors and performance, an abundance of strong significant correlations based upon programming behavior were found. A model based upon two of these metrics (Watwin score and percentage of lab time spent resolving errors) could explain 56.3% of the variance in coursework results. The implication of this study is that a student's programming behavior is one of the strongest indicators of their performance, and future work should continue to explore such predictors in different teaching contexts

Learning analytics in higher education: a review of impact scientific literature

Las analíticas de aprendizaje pueden definirse como una serie de técnicas para recopilar, analizar y otorgar datos procesables y generados por parte de los estudiantes con el objetivo de elaborar estrategias adecuadas para mejorar los procesos de aprendizaje, el rendimiento de los alumnos o el de la propia institución. Este tipo de técnicas son especialmente útiles para establecer patrones de acción que guíen y orienten el proceso educativo en educación superior. Bajo estas premisas, la presente investigación tiene por objetivo analizar la producción científica de mayor impacto sobre el empleo de analíticas de aprendizaje en educación superior. Para ello se ha seguido una metodología cuantitativa atendiendo diez variables: año de publicación, publicaciones periódicas, autores, instituciones, países, tipo de documento, formato de publicación, área de publicación, idioma y artículos más citados. Los resultados proyectan una tendencia de investigación que se encuentra totalmente en auge, especialmente por la mayor producción científica ocurrida en los últimos años (2015-2018) destacando países como Australia, Estados Unidos y Reino Unido. Las publicaciones, en su gran mayoría, proceden de conferencias y se encuentran publicadas en inglés. Destacan las áreas de conocimiento de Ciencias Computacionales y Ciencias Sociales.Learning analytics can be defined as a series of techniques for collecting, analyzing and granting processable data generated by students. Their objective is to develop appropriate strategies to improve the learning processes, the performance of the students or that of the institution itself. These types of techniques are especially useful to establish patterns of action that guide and guide the educational process in higher education. Under these premises, the present research aims to analyze the scientific production with the greatest impact on the use of learning analytics in higher education. For this purpose, a quantitative methodology has been followed, based on ten variables: year of publication, periodicals, authors, institutions, countries, type of document, publication format, publication area, language and most cited articles. The results project a research trend that is fully on the rise, especially due to the greater scientific production that has occurred in recent years (2015- 2018), highlighting countries such as Australia, the United States and the United Kingdom. The publications, in their great majority, come from conferences and are published in English. The knowledge areas of Computational Sciences and Social Sciences stand out.Ministry of Education, Culture and Sport through the Aid of the University Teaching Staff Training Program FPU14/0462

ArAl: An Online Tool for Source Code Snapshot Metadata Analysis

© 2019 Association for Computing Machinery. Several systems that collect data from students' problem solving processes exist. Within computing education research, such data has been used for multiple purposes, ranging from assessing students' problem solving strategies to detecting struggling students. To date, however, the majority of the analysis has been conducted by individual researchers or research groups using case by case methodologies. Our belief is that with increasing possibilities for data collection from students' learning process, researchers and instructors will benefit from ready-made analysis tools. In this study, we present ArAl, an online machine learning based platform for analyzing programming source code snapshot data. The benefit of ArAl is two-fold. The computing education researcher can use ArAl to analyze the source code snapshot data collected from their own institute. Also, the website provides a collection of well-documented machine learning and statistics based tools to investigate possible correlation between different variables. The presented web-portal is available at online-analysisdemo. herokuapp.com. This tool could be applied in many different subject areas given appropriate performance data

Learning analytics in higher education: a review of impact scientific literature

Las analíticas de aprendizaje pueden definirse como una serie de técnicas para recopilar, analizar y otorgar datos procesables y generados por parte de los estudiantes con el objetivo de elaborar estrategias adecuadas para mejorar los procesos de aprendizaje, el rendimiento de los alumnos o el de la propia institución. Este tipo de técnicas son especialmente útiles para establecer patrones de acción que guíen y orienten el proceso educativo en educación superior. Bajo estas premisas, la presente investigación tiene por objetivo analizar la producción científica de mayor impacto sobre el empleo de analíticas de aprendizaje en educación superior. Para ello se ha seguido una metodología cuantitativa atendiendo diez variables: año de publicación, publicaciones periódicas, autores, instituciones, países, tipo de documento, formato de publicación, área de publicación, idioma y artículos más citados. Los resultados proyectan una tendencia de investigación que se encuentra totalmente en auge, especialmente por la mayor producción científica ocurrida en los últimos años (2015-2018) destacando países como Australia, Estados Unidos y Reino Unido. Las publicaciones, en su gran mayoría, proceden de conferencias y se encuentran publicadas en inglés. Destacan las áreas de conocimiento de Ciencias Computacionales y Ciencias Sociales.Learning analytics can be defined as a series of techniques for collecting, analyzing and granting processable data generated by students. Their objective is to develop appropriate strategies to improve the learning processes, the performance of the students or that of the institution itself. These types of techniques are especially useful to establish patterns of action that guide and guide the educational process in higher education. Under these premises, the present research aims to analyze the scientific production with the greatest impact on the use of learning analytics in higher education. For this purpose, a quantitative methodology has been followed, based on ten variables: year of publication, periodicals, authors, institutions, countries, type of document, publication format, publication area, language and most cited articles. The results project a research trend that is fully on the rise, especially due to the greater scientific production that has occurred in recent years (2015-2018), highlighting countries such as Australia, the United States and the United Kingdom. The publications, in their great majority, come from conferences and are published in English. The knowledge areas of Computational Sciences and Social Sciences stand out.Universidad Pablo de Olavid

Computing education theories : what are they and how are they used?

In order to mature as a research field, computing education research (CER) seeks to build a better theoretical understanding of how students learn computing concepts and processes. Progress in this area depends on the development of computing-specific theories of learning to complement the general theoretical understanding of learning processes. In this paper we analyze the CER literature in three central publication venues -- ICER, ACM Transactions of Computing Education, and Computer Science Education -- over the period 2005--2015. Our findings identify new theoretical constructs of learning computing that have been published, and the research approaches that have been used in formulating these constructs. We identify 65 novel theoretical constructs in areas such as learning/understanding, learning behaviour/strategies, study choice/orientation, and performance/progression/retention. The most common research methods used to devise new constructs include grounded theory, phenomenography, and various statistical models. We further analyze how a number of these constructs, which arose in computing education, have been used in subsequent research, and present several examples to illustrate how theoretical constructs can guide and enrich further research. We discuss the implications for the whole field