Investigating Automated Student Modeling in a Java MOOC

With the advent of ubiquitous web, programming is no longer a sole prerogative of computer science schools. Scripting languages are taught to wider audiences and programming has become a flag post of any technology related program. As more and more students are exposed to coding, it is no longer a trade of the select few. As a result, students who would not opt for a coding class a decade ago are in a position of having to learn a rather difficult subject. The problem of assisting students in learning programming has been explored in several intelligent tutoring systems. The key component of such systems is a student model that keeps track of student progress. In turn, the foundation of a student model is a domain model – a vocabulary of skills (or concepts) that structures the representation of student knowledge. Building domain models for programming is known as a complicated task. In this paper we explore automated approaches for extracting domain models for learning programming languages and modeling student knowledge in the process of solving programming exercises. We evaluate the validity of this approach using large volume of student code submission data from a MOOC on introductory Java programming

Wide-Scale Automatic Analysis of 20 Years of ITS Research

The analysis of literature within a research domain can provide significant value during preliminary research. While literature reviews may provide an in-depth understanding of current studies within an area, they are limited by the number of studies which they take into account. Importantly, whilst publications in hot areas abound, it is not feasible for an individual or team to analyse a large volume of publications within a reasonable amount of time. Additionally, major publications which have gained a large number of citations are more likely to be included in a review, with recent or fringe publications receiving less inclusion. We provide thus an automatic methodology for the large-scale analysis of literature within the Intelligent Tutoring Systems (ITS) domain, with the aim of identifying trends and areas of research from a corpus of publications which is significantly larger than is typically presented in conventional literature reviews. We illustrate this by a novel analysis of 20 years of ITS research. The resulting analysis indicates a significant shift of the status quo of research in recent years with the advent of novel neural network architectures and the introduction of MOOCs

What Stays in Mind? - Retention Rates in Programming MOOCs

This work presents insights about the long-term effects and retention rates of knowledge acquired within MOOCs. In 2015 and 2017, we conducted two introductory MOOCs on object-oriented programming in Java with each over 10,000 registered participants. In this paper, we analyze course scores, quiz results and self-stated skill levels of our participants. The aim of our analysis is to uncover factors influencing the retention of acquired knowledge, such as time passed or knowledge application, in order to improve long-term success. While we know that some participants learned the programming basics within our course, we lack information on whether this knowledge was applied and fortified after the course's end. To fill this knowledge gap, we conducted a survey in 2018 among all participants of our 2015 and 2017 programming MOOCs. The first part of the survey elicits responses on whether and how MOOC knowledge was applied and gives participants opportunity to voice individual feedback. The second part of the survey contains several questions of increasing difficulty and complexity regarding course content in order to learn about the consolidation of the acquired knowledge. We distinguish three programming knowledge areas in the survey: First, understanding of concepts, such as loops and boolean algebra. Second, syntax knowledge, such as specific keywords. Third, practical skills including debugging and coding. We further analyzed the long-term effects separately per participant skill group. While answer rates were low, the collected data shows a decrease of knowledge over time, relatively unaffected by skill level. Application of the acquired knowledge improves the memory retention rates of MOOC participants across all skill levels

Integrating knowledge tracing and item response theory: A tale of two frameworks

Traditionally, the assessment and learning science commu-nities rely on different paradigms to model student performance. The assessment community uses Item Response Theory which allows modeling different student abilities and problem difficulties, while the learning science community uses Knowledge Tracing, which captures skill acquisition. These two paradigms are complementary - IRT cannot be used to model student learning, while Knowledge Tracing assumes all students and problems are the same. Recently, two highly related models based on a principled synthesis of IRT and Knowledge Tracing were introduced. However, these two models were evaluated on different data sets, using different evaluation metrics and with different ways of splitting the data into training and testing sets. In this paper we reconcile the models' results by presenting a unified view of the two models, and by evaluating the models under a common evaluation metric. We find that both models are equivalent and only differ in their training procedure. Our results show that the combined IRT and Knowledge Tracing models offer the best of assessment and learning sciences - high prediction accuracy like the IRT model, and the ability to model student learning like Knowledge Tracing

Towards Student Engagement Analytics: Applying Machine Learning to Student Posts in Online Lecture Videos

The use of online learning environments in higher education is becoming ever more prevalent with the inception of MOOCs (Massive Open Online Courses) and the increase in online and flipped courses at universities. Although the online systems used to deliver course content make education more accessible, students often express frustration with the lack of assistance during online lecture videos. Instructors express concern that students are not engaging with the course material in online environments, and rely on affordances within these systems to figure out what students are doing. With many online learning environments storing log data about students usage of these systems, research into learning analytics, the measurement, collection, analysis, and reporting data about learning and their contexts, can help inform instructors about student learning in the online context. This thesis aims to lay the groundwork for learning analytics that provide instructors high-level student engagement data in online learning environments. Recent research has shown that instructors using these systems are concerned about their lack of awareness about student engagement, and educational psychology has shown that engagement is necessary for student success. Specifically, this thesis explores the feasibility of applying machine learning to categorize student posts by their level of engagement. These engagement categories are derived from the ICAP framework, which categorizes overt student behaviors into four tiers of engagement: Interactive, Constructive, Active, and Passive. Contributions include showing what natural language features are most indicative of engagement, exploring whether this machine learning method can be generalized to many courses, and using previous research to develop mockups of what analytics using data from this machine learning method might look like

Sissejuhatava programmeerimise MOOCi õppijad: taustamuutujad, kaasatuse mustrid ja õpisooritus

Väitekirja elektrooniline versioon ei sisalda publikatsiooneÜks võimalus personaalseks ja professionaalseks arenguks on osalemine vaba juurdepääsuga e-kursustel (ingl massive open online courses, MOOCs). MOOCide osalejatel on suurem autonoomia võrreldes traditsiooniliste klassiruumides toimuvate tundidega. Samuti arvestades suurt osalejate hulka ja nende erinevat tausta, on kõikide õppijate kaasatus (ingl engagement) õppeprotsessis MOOCide korraldajatele väljakutseks. Osalejate taustamuutujate (ingl background variables) mõju kaasatusele, mis omakorda võib mõjutada õpisooritust (ingl performance), on jätkuvalt alauuritud valdkond. Doktoritöö eesmärk oli uurida MOOCide osalejate taustamuutujaid ja nende mõju kursusele registreerumisele ning lõpetamise tõenäosusele, tuvastada lõpetajate seas käitumuslikke (ingl behavioural engagement) ja kognitiivseid (ingl cognitive engagement) kaasatuse rühmasid ning uurida neid taustamuutujate ja õpisoorituse osas. Uurimuse fookuses oli MOOC “Programmeerimisest maalähedaselt”. Selle MOOCi osalejate ja lõpetajate taustamuutujad võrreldi MOOCidega „Programmeerimise alused I“ ja „Programmeerimise alused II“. MOOCil “Programmeerimisest maalähedaselt” oli rohkem naisi ja neid, kelle haridustase oli madalam. Lõpetajate osas selgus, et põhifookuses olnud MOOCil, ei olnud statistiliselt olulist erinevust nais- ja meeslõpetajate osakaalu ning erinevate tööhõive staatuste vahel. Suurem lõpetajate osakaal oli magistrikraadiga lõpetajate hulgas. Väiksem lõpetajate osakaal oli nende õppijate puhul, kes ei ole varem programmeerimist õppinud. Samad tulemused lõpetajate kohta olid ka MOOCil “Programmeerimise alused I“. Uurides MOOCi “Programmeerimisest maalähedaselt” lõpetajate ja mittelõpetajate õpisooritust, selgus, et nad vajasid testi sooritamiseks keskmiselt sama palju katseid. Mittelõpetajatel oli programmeerimisülesannete lahenduste esitamiskordade arv suurem ja neil oli testipunktid madalamad. Lõpetajate käitumusliku ja kognitiivse kaasatuse analüüs näitas, et lõpetajad ei ole homogeenne rühm. Käitumusliku kaasatuse puhul eristati lähtudes tegevuste hulgast 4 rühma. Uurimuse tulemused näitasid, et MOOCil võivad olla lõpetajad, kes teevad kõiki tegevusi, aga ka need, kes teevad vaid mõnda tegevust. Kognitiivse kaasatuse korral eristus 5 rühma, mille puhul kasutati abiallikaid erineva sagedusega. Tulemused näitasid, et lõpetajate erinevat sagedust erinevate abiallikate kasutamisel võib pidada märgiks püsivast soovist MOOC edukalt läbida. Samuti selgus, et abiallikate kasutamise võib võtta aluseks kognitiivse kaasatuse tuvastamiseks ja mõõtmiseks MOOCidel. Lõpetajate taustamuutujad ja õpisooritused varieerusid eristatud rühmade vahel. Doktoritöös esitatud tulemused aitavad uurijatel paremini aru saada MOOCi fenomenist ja kursuste korraldajatel pakkuda tulevikus kulutõhusamaid MOOCe. Uurimistulemustest võib järeldada, et MOOCide korraldajad peavad pakkuma erinevaid tegevusi ja abiallikaid, mis oleksid suunatud konkreetsetele sihtrühmadele. See võib hõlbustada personaliseeritud õppimist ja õppijate tõhusat kaasatust õppeprotsessis.One opportunity to facilitate personal and professional development is to participate in massive open online courses (MOOCs). MOOCs participants have greater autonomy compared to traditional physical classes. In addition, considering the huge number of participants and diversity of their backgrounds, it is a challenge for MOOCs instructors to engage them all in learning. The impact of background variables on engagement, which in turn may influence performance, remains understudied. The doctoral thesis aimed to study MOOCs participants’ background variables and their impact on course enrolment and completion probability, and explore different behavioural and cognitive engagement clusters among completers in terms of background variables and performance. The thesis focused on a MOOC “About Programming”. The MOOC participants’ and completers’ background variables were examined in comparison to MOOCs “Introduction to Programming I” and “Introduction to Programming II”. Females and those with a lower education level dominated in the MOOC “About Programming”. In this course, among completers there was no difference by gender and employment statuses. Master’s degree holders were more likely to complete the MOOC, while inexperienced in programming were less likely to complete it. The same results about completers were found in the MOOC “Introduction to Programming I”. With regard to performance, no difference between the MOOC “About programming” completers and non-completers in the average number of attempts per quiz was found. But non-completers made on average more attempts per programming task and received lower scores per quiz. The analysis of behavioural and cognitive engagement solely among completers indicated that they cannot be considered a homogeneous group. In terms of behavioural engagement, there were identified 4 groups based on the amount of activities a completer engaged with during the MOOC. The study results indicated that in a MOOC there can be completers who engage with all activities, as well as those who engage with only a few activities. In terms of cognitive engagement, there were identified 5 groups that were engaged with help sources at different frequency. The results indicated that the different frequency, with which completers use different help sources, can be considered as a sign of persistent desire to successfully complete the MOOC. In addition, it was revealed that the use of help sources can be applied as a basis for identifying and measuring cognitive engagement in the MOOC context. The background variables and performance of completers from different identified groups varied. The results of the thesis can prove quite beneficial to the scientific literature to understand the phenomenon of MOOC. This comprehension in terms of a variety of background variables, engagement patterns and performance can be helpful for course instructors to develop cost-effective MOOCs and provide personalised learning where different course activities and help sources can be targeted at specific groups.  https://www.ester.ee/record=b552843