Integrating knowledge tracing and item response theory: A tale of two frameworks

Traditionally, the assessment and learning science commu-nities rely on different paradigms to model student performance. The assessment community uses Item Response Theory which allows modeling different student abilities and problem difficulties, while the learning science community uses Knowledge Tracing, which captures skill acquisition. These two paradigms are complementary - IRT cannot be used to model student learning, while Knowledge Tracing assumes all students and problems are the same. Recently, two highly related models based on a principled synthesis of IRT and Knowledge Tracing were introduced. However, these two models were evaluated on different data sets, using different evaluation metrics and with different ways of splitting the data into training and testing sets. In this paper we reconcile the models' results by presenting a unified view of the two models, and by evaluating the models under a common evaluation metric. We find that both models are equivalent and only differ in their training procedure. Our results show that the combined IRT and Knowledge Tracing models offer the best of assessment and learning sciences - high prediction accuracy like the IRT model, and the ability to model student learning like Knowledge Tracing

Detecting Communities and Analysing Interactions with Learning Objects in Online Learning Repositories

The widespread use of online learning object repositories has raised the need of studies that assess the quality of their contents, and their user’s performance and engagement. The present research addresses two fundamental problems that are central to that need: the need to explore user interaction with these repositories and the detection of emergent communities of users. The current dissertation approaches those directions through investigating and mining the Khan Academy repository as a free, open access, popular online learning repository addressing a wide content scope. It includes large numbers of different learning objects such as instructional videos, articles, and exercises. In addition to a large number of users. Data was collected using the repository’s public application programming interfaces combined with Web scraping techniques to gather data and user interactions. Different research activities were carried out to generate useful insights out of the gathered data. We conducted descriptive analysis to investigate the learning repository and its core features such as growth rate, popularity, and geographical distribution. A number of statistical and quantitative analysis were applied to examine the relation between the users’ interactions and different metrics related to the use of learning objects in a step to assess the users’ behaviour. We also used different Social Network Analysis (SNA) techniques on a network graph built from a large number of user interactions. The resulting network consisted of more than 3 million interactions distributed across more than 300,000 users. The type of those interactions is questions and answers posted on Khan Academy’s instructional videos (more than 10,000 video). In order to analyse this graph and explore the social network structure, we studied two different community detection algorithms to identify the learning interactions communities emerged in Khan Academy then we compared between their effectiveness. After that, we applied different SNA measures including modularity, density, clustering coefficients and different centrality measures in order to assess the users’ behaviour patterns and their presence. Using descriptive analysis, we discovered many characteristics and features of the repository. We found that the number of learning objects in Khan Academy’s repository grows linearly over time, more than 50% of the users do not complete the watched videos, and we found that the average duration for video lessons 5 to 10 minutes which aligns with the recommended duration in literature. By applying community detection techniques and social network analysis, we managed to identify learning communities in Khan Academy’s network. The size distribution of those communities found to follow the power-law distribution which is the case of many real-world networks. Those learning communities are related to more than one domain which means the users are active and interacting across domains. Different centrality measures we applied to focus on the most influential players in those communities. Despite the popularity of online learning repositories and their wide use, the structure of the emerged learning communities and their social networks remain largely unexplored. Our findings could be considered initial insights that may help researchers and educators in better understanding online learning repositories, the learning process inside those repositories, and learner behaviou

An analysis of students’ behaviour in a Learning Management System through Process Mining

Dissertation presented as the partial requirement for obtaining a Master's degree in Information Management, specialization in Information Systems and Technologies ManagementThe exponential growth and transformation of the Internet and information technology in recent years led to the development of several analytical tools. As is the case with process mining, it emerged to fulfill the need to extract and analyze information from event logs by representing it in the form of process models. Process mining is an acclaimed tool and proved crucial in several areas, from healthcare to manufacturing and finance. Nevertheless, and despite the crucial role of digital systems in supporting learning activities and generating large amounts of data about learning processes, limited research focused on process mining applied to the educational context. Therefore, the aim of this dissertation is to apply a process-oriented approach and demonstrate the applicability of process mining techniques to explore and analyze students’ behavior and interaction patterns, based on data collected from Moodle, the widely used Learning Management System. We cover definitions of process mining, education, and a detailed search of the existing literature on educational process mining during this work. Furthermore, the paper analyzes and discusses the findings of the study that combines process mining techniques, specifically process discovery implanted in the Disco tool, with cluster analysis. Through the application of these two techniques, it was possible to recognize the relationship between the students’ behavior registered in the process models and the success of the students in the course, along with the general and specific information about the students’ learning paths. Besides, we obtained findings that allow us to predict the group of students at risk of failing. Finally, with the analysis of these results, we were able to provide improvement proposals and recommendations to enhance the learning experience

Share and reuse of context metadata resulting from interactions between users and heterogeneous web-based learning environments

L'intérêt pour l'observation, l'instrumentation et l'évaluation des systèmes éducatifs en ligne est devenu de plus en plus important ces dernières années au sein de la communauté des Environnements Informatique pour l'Apprentissage Humain (EIAH). La conception et le développement d'environnements d'apprentissage en ligne adaptatifs (AdWLE - Adaptive Web-based Learning Environments) représentent une préoccupation majeure aujourd'hui, et visent divers objectifs tels que l'aide au processus de réingénierie, la compréhension du comportement des utilisateurs, ou le soutient à la création de systèmes tutoriels intelligents. Ces systèmes gèrent leur processus d'adaptation sur la base d'informations détaillées reflétant le contexte dans lequel les étudiants évoluent pendant l'apprentissage : les ressour-ces consultées, les clics de souris, les messages postés dans les logiciels de messagerie instantanée ou les forums de discussion, les réponses aux questionnaires, etc. Les travaux présentés dans ce document sont destinés à surmonter certaines lacunes des systèmes actuels en fournissant un cadre dédié à la collecte, au partage et à la réutilisation du contexte représenté selon deux niveaux d'abstraction : le contexte brut (résultant des interactions directes entre utilisateurs et applications) et le contexte inféré (calculé à partir des données du contexte brut). Ce cadre de travail qui respecte la vie privée des usagers est fondé sur un standard ouvert dédié à la gestion des systèmes, réseaux et applications. Le contexte spécifique aux outils hétérogènes constituant les EIAHs est représenté par une structure unifiée et extensible, et stocké dans un référentiel central. Pour faciliter l'accès à ce référentiel, nous avons introduit une couche intermédiaire composée d'un ensemble d'outils. Certains d'entre eux permettent aux utilisateurs et applications de définir, collecter, partager et rechercher les données de contexte qui les intéressent, tandis que d'autres sont dédiés à la conception, au calcul et à la délivrance des données de contexte inférées. Pour valider notre approche, une mise en œuvre du cadre de travail proposé intègre des données contextuelles issues de trois systèmes différents : deux plates-formes d'apprentissage Moodle (celle de l'Université Paul Sabatier de Toulouse, et une autre déployée dans le cadre du projet CONTINT financé par l'Agence Nationale de la Recherche) et une instanciation locale du moteur de recherche de la fondation Ariadne. A partir des contextes collectés, des indicateurs pertinents ont été calculés pour chacun de ces environnements. En outre, deux applications qui exploitent cet ensemble de données ont été développées : un système de recommandation personnalisé d'objets pédagogiques ainsi qu'une application de visualisation fondée sur les technologies tactiles pour faciliter la navigation au sein de ces données de contexte.An interest for the observation, instrumentation, and evaluation of online educational systems has become more and more important within the Technology Enhanced Learning community in the last few years. Conception and development of Adaptive Web-based Learning Environments (AdWLE) in order to facilitate the process of re-engineering, to help understand users' behavior, or to support the creation of Intelligent Tutoring Systems represent a major concern today. These systems handle their adaptation process on the basis of detailed information reflecting the context in which students evolve while learning: consulted resources, mouse clicks, chat messages, forum discussions, visited URLs, quizzes selections, and so on. The works presented in this document are intended to overcome some issues of the actual systems by providing a privacy-enabled framework dedicated to the collect, share and reuse of context represented at two abstraction levels: raw context (resulting from direct interactions between users and applications) and inferred context (calculated on the basis of raw context). The framework is based on an open standard dedicated to system, network and application management, where the context specific to heterogeneous tools is represented as a unified and extensible structure and stored into a central repository. To facilitate access to this context repository, we introduced a middleware layer composed of a set of tools. Some of them allow users and applications to define, collect, share and search for the context data they are interested in, while others are dedicated to the design, calculation and delivery of inferred context. To validate our approach, an implementation of the suggested framework manages context data provided by three systems: two Moodle servers (one running at the Paul Sabatier University of Toulouse, and the other one hosting the CONTINT project funded by the French National Research Agency) and a local instantiation of the Ariadne Finder. Based on the collected context, relevant indicators have been calculated for each one of these environments. Furthermore, two applications which reuse the encapsulated context have been developed on top of the framework: a personalized system for recommending learning objects to students, and a visualization application which uses multi-touch technologies to facilitate the navigation among collected context entities

The impact of emotions on student participation in an assessed, online, collaborative activity

There is growing recognition of the importance of emotions in academic online learning contexts. However, there is still little known about the role of emotions in social and collaborative online learning settings, especially the relationship between emotions and student participation. To explore this relationship, this study used a prospective longitudinal research design to follow 46 distance learning students throughout a 3-week assessed, online, collaborative activity. This approach allowed the fluctuating and dynamic aspects of emotions to be explored as well as the relationship between emotions and student participation in the collaborative activity. Self-report data were gathered using a semistructured online diary at five time points throughout the task (once at the start of the collaborative activity, three times during the activity, and the final entry after the activity had finished). Findings revealed that learners generally perceived pleasant emotions (such as relief, satisfaction and enjoyment) to have positive impacts, or no impact, on participation, whereas unpleasant emotions (such as anxiety, frustration, and disappointment) were generally perceived to have negative impacts, or no impact, on participation. Interestingly, however, anxiety, and to a smaller extent frustration, were perceived by a number of students to have positive impacts during the activity. To conclude this paper, implications for educators are highlighted