Representational Learning Approach for Predicting Developer Expertise Using Eye Movements

The thesis analyzes an existing eye-tracking dataset collected while software developers were solving bug fixing tasks in an open-source system. The analysis is performed using a representational learning approach namely, Multi-layer Perceptron (MLP). The novel aspect of the analysis is the introduction of a new feature engineering method based on the eye-tracking data. This is then used to predict developer expertise on the data. The dataset used in this thesis is inherently more complex because it is collected in a very dynamic environment i.e., the Eclipse IDE using an eye-tracking plugin, iTrace. Previous work in this area only worked on short code snippets that do not represent how developers usually program in a realistic setting. A comparative analysis between representational learning and non-representational learning (Support Vector Machine, Naive Bayes, Decision Tree, and Random Forest) is also presented. The results are obtained from an extensive set of experiments (with an 80/20 training and testing split) which show that representational learning (MLP) works well on our dataset reporting an average higher accuracy of 30% more for all tasks. Furthermore, a state-of-the-art method for feature engineering is proposed to extract features from the eye-tracking data. The average accuracy on all the tasks is 93.4% with a recall of 78.8% and an F1 score of 81.6%. We discuss the implications of these results on the future of automated prediction of developer expertise. Adviser: Bonita Shari

Temporal pathways to learning: how learning emerges in an open-ended collaborative activity

The learning process depends on the nature of the learning environment, particularly in the case of open-ended learning environments, where the learning process is considered to be non-linear. In this paper, we report on the findings of employing a multimodal Hidden Markov Model (HMM) based methodology to investigate the temporal learning processes of two types of learners that have learning gains and a type that does not have learning gains in an open-ended collaborative learning activity. Considering log data, speech behavior, affective states and gaze patterns, we find that all learners start from a similar state of non-productivity, but once out of it they are unlikely to fall back into that state, especially in the case of the learners that have learning gains. Those who have learning gains shift between two problem solving strategies, each characterized by both exploratory and reflective actions, as well as demonstrate speech and gaze patterns associated with these strategies, that differ from those who don't have learning gains. Further, the teams that have learning gains also differ between themselves in the manner in which they employ the problem solving strategies over the interaction, as well as in the manner they express negative emotions while exhibiting a particular strategy. These outcomes contribute to understanding the multiple pathways of learning in an open-ended collaborative learning environment, and provide actionable insights for designing effective interventions

Predicting Pair Success in a Pair Programming Eye Tracking Experiment Using Cross-Recurrence Quantification Analysis

Pair programming is a model of collaborative learning. It has become a well-known pedagogical practice in teaching introductory programming courses because of its potential benefits to students. This study aims to investigate pair patterns in the context of pair program tracing and debugging to determine what characterizes collaboration and how these patterns relate to success, where success is measured in terms of performance task scores. This research used eye-tracking methodologies and techniques such as cross-recurrence quantification analysis. The potential indicators for pair success were used to create a model for predicting pair success. Findings suggest that it is possible to create a model capable of predicting pair success in the context of pair programming. The predictors for the pair success model that can obtain the best performance are the pairs\u27 proficiency level and degree of acquaintanceship. This was achieved using an ensemble algorithm such as Gradient Boosted Trees. The performance of the pairs is largely determined by the proficiency level of the individuals in the pairs; hence, it is recommended that the struggling students be paired with someone who is considered proficient in programming and with whom the struggling student is comfortable working with

Predicting student performance in an augmented reality learning environment using eye-tracking data

This paper investigates the use of eye-tracking data as a predictor of student performance in an augmented reality (AR) learning environment. 33 undergraduate students enrolled in an ergonomics course at the University of Missouri-Columbia participated in an AR biomechanics lecture consisting of 14 modules. Following each module students answered learning comprehension questions to test their understanding of the lecture material. An additional dataset was recorded for each module in which the participant perfectly follows the virtual instructor throughout the learning space. This dataset, referred to as the baseline, can be used as a comparison tool to gauge how well students follows the lecture material. Two methods are proposed to quantify the student's attention level for each module. The average difference method calculates the average distance between the student and baseline coordinates for each module. The distraction rate method expands upon the average difference method and aims to reduce the amount noise detected. This is done by incorporating a minimum distance threshold, a binary detection signal, and a moving average window. Both metrics are tested as factors in a set of logistic regression models to determine whether they can accurately predict student answer correctness. Average difference showed a correlation with student answer correctness, but with an underwhelming level of significance. Distraction rate outperformed average difference and proved to be a strong and statistically significant predictor of student answer correctness. Finally, two feedback systems are proposed which use distraction rate to detect when students have become distracted so that their attention can be regained through the use of module-based feedback or a real-time attention guidance system.Includes bibliographical references