In recent years, distance education has enjoyed a major boom. Much work at The Open University (OU) has focused on improving retention rates in these modules by providing timely support to students who are at risk of failing the module. In this paper we explore methods for analysing student activity in online virtual learning environment (VLE) - General Unary Hypotheses Automaton (GUHA) and Markov chain-based analysis - and we explain how this analysis can be relevant for module tutors and other student support staff. We show that both methods are a valid approach to modelling student activities. An advantage of the Markov chain-based approach is in its graphical output and in the possibility to model time dependencies of the student activities.





Drahomira Herrmannova,Lucie Vachova,Jakub Kuzilek,Zdenek Zdrahal,Annika Wolf

Herrmannova, Drahomira

Hlosta, Martin

Kuzilek, Jakub

Vachova, Lucie

Wolff, Annika

Zdrahal, Zdenek

English

Open Research Online

Open Research OnlineThe Open University’s repository of research publicationsand other research outputsModelling student online behaviour in a virtual learningenvironmentConference or Workshop ItemHow to cite:Hlosta, Martin; Herrmannova, Drahomira; Vachova, Lucie; Kuzilek, Jakub; Zdrahal, Zdenek and Wolff, Annika(2014). Modelling student online behaviour in a virtual learning environment. In: Machine Learning and LearningAnalytics workshop at The 4th International Conference on Learning Analytics and Knowledge (LAK14), 24-28 March2014, Indianapolis, Indiana, USA, 24-28 Mar 2014, Indianapolis, Indiana, USA.For guidance on citations see FAQs.c© 2014 The AuthorsVersion: Accepted ManuscriptLink(s) to article on publisher’s website:http://lak14indy.wordpress.com/Copyright and Moral Rights for the articles on this site are retained by the individual authors and/or other copyrightowners. For more information on Open Research Online’s data policy on reuse of materials please consult the policiespage.oro.open.ac.ukModelling student online behaviour in a virtual learningenvironmentMartin Hlosta† Drahomira Herrmannova† Lucie Vachova††Jakub Kuzilek† Zdenek Zdrahal† Annika Wolff†Knowledge Media Institute, The Open University†Walton HallMilton Keynes, MK7 6AA{martin.hlosta; d.herrmannova; jakub.kuzilek;z.zdrahal; annika.wolff}@open.ac.ukUniversity of Economics, Prague††∗Department of Exact Methods, Faculty of ManagementJarosovska 1117/IIJindrichuv Hradec, 377 01vachova@fm.vse.czABSTRACTIn recent years, distance education has enjoyed a majorboom. Much work at The Open University (OU) has focusedon improving retention rates in these modules by provid-ing timely support to students who are at risk of failing themodule. In this paper we explore methods for analysing stu-dent activity in online virtual learning environment (VLE) –General Unary Hypotheses Automaton (GUHA) and Markovchain-based analysis – and we explain how this analysis canbe relevant for module tutors and other student supportstaff. We show that both methods are a valid approach tomodelling student activities. An advantage of the Markovchain-based approach is in its graphical output and in thepossibility to model time dependencies of the student activ-ities.Categories and Subject DescriptorsD.4.8 [Performance]: Modelling and Prediction;H.2.8 [Database Applications]: Data MiningGeneral TermsAlgorithms, Design, Experimentation, Human FactorsKeywordsStudent Data, Distance Learning, Predictive Models, Ma-chine Learning, Information Visualisation1. INTRODUCTIONThe recent years have seen a massive growth of differ-ent possibilities of online education, such as the well known∗This work was carried out at the Knowledge Media Insti-tute.Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee. Request permissions from Permissions@acm.org.LAK ’14, March 24 – 28 2014, Indianapolis, IN, USACopyright 2014 ACM 978-1-4503-2664-3/14/03 ...$15.00.massive open online courses (MOOCs) [Cormier,2008]. Theconcept of distance education is however not new. The OpenUniversity is an institution with over forty years of experi-ence with distance education, historically based on off-linematerials and nowadays making an increasing use of the In-ternet. The great advantage of the online courses is in thefact they are accessible to virtually anyone with Internetaccess.The other side of the coin is that the retention rates inthese courses are often low. [Koller et al.,2013] mention, thatan average retention rate of a Coursera1 course is around5%. The situation at traditional universities as well as atThe Open University is significantly better, however, thereis still a room for improvement.There might be many reasons for the low retention rates,from the fact that the online courses are often offered toanybody interested to the fact that the performance of eachstudent depends almost exclusively on how much are theywilling to study on their own at home. Our work at TheOpen University aims at analysing students’ activities in theonline courses in order to gain insight into their behaviouralpatterns, which can be utilised for building prediction mod-els.1.1 Problem DescriptionThe Open University2 is the biggest university in the Unit-ed Kingdom, offering several hundred distance learning mod-ules, which can be studied both as standalone modules or aspart of a university degree. Anybody can sign up for a mod-ule provided by The OU, without any previous educationwhatsoever. The students receive their study material andsubmit their assignments through an online virtual learningenvironment.Students participating in a module are generally split intosmaller study groups of no more than few tens of students,typically according to their geographic location. Each grouphas an assigned tutor. The tutors grade the students’ as-signments and exams, answer their questions in the onlineforums, provide general advice and guidance, etc.In order to support the students who are at risk of fail-ing the module The OU also implements various interven-1https://www.coursera.org/, a well known and one of thebiggest platforms providing open online courses.2http://www.open.ac.uk/tions (such as phone calls from a specialised student supportteams) during the course of the module. Because the numberof students studying each module can reach several thousand(the modules used in our analysis have enrolment of aroundtwo thousand students) and the resources available for theinterventions are limited, the interventions have to be care-fully planned. Therefore, an important question one mightask is how to identify students at risk of failing the moduleso that the intervention is meaningful and efficient.Improving student retention through these focused inter-ventions and helping the tutors to focus on the students,who require help, provides many benefits, from improvedstudent satisfaction to financial savings for the university.In order to support the identification of students who arecurrently at risk, we utilise several statistical and machinelearning methods. The available data contain both the in-formation about the students’ activity in the VLE as wellas their demographic information. However, for modellingstudent behaviour, only the data from the VLE was used.A more detailed description of our data set can be found inSection 3.2. PREVIOUS WORKThe current work builds on previous research done atThe Open University. The initial experiments with machinelearning techniques were using the VLE and assessment data[Wolff and Zdrahal,2012]. One of the main findings of thisresearch was that decision trees generally outperform theother methods [Wolff and Zdrahal,2012]. This research alsoincluded creation of a dashboard providing the universitystaff with real-time information about student performance.Additional methods were tested in [Wolff et al.,2013b] andin [Wolff et al.,2013a]. In the latter work, demographic datawere added to the predictions, however this research didnot confirm that this data provide a significant increase inperformance.3. DATA SPECIFICATIONThe analyses we performed were done using real data ofseveral modules from The Open University. We examined anumber of subsequent presentations of each module.The available data contain two types of information:• Information about the results of student assignments(TMAs – tutor marked assignments). There are sev-eral assignments in each module, typically between fiveand seven. Generally, the module is ended by a finalexam.• Data about student activity from the virtual learningenvironment (VLE).The VLE data are aggregated by days and content type(e.g. forum, wiki, resource, ...). This means that for eachday we know how many times did the student interact withgiven content type. For our analysis we summarise the databy weeks and content types. Summarising the data by weeksseemed to be reasonable, it simplifies our analyses withoutloosing too much detail. The features generated by the sum-marisation and used in the methods are:• click counts aggregated by week,• click counts aggregated by week and content type,• binary flags indicating whether student was active inthe VLE and in various content types.4. METHODSFor analysis of student behaviour in the virtual learn-ing environments, we have used two different approaches –GUHA [Ha´jek et al.,1966] and modelling based on Markovchains [Norris,1998].4.1 Activity types analysisAs mentioned in Section 3, the VLE data contain infor-mation about the type of content the student accessed. Thecontent type can be for example forum, wiki, resource, quiz,etc. In total there are 11 different content types. Using thebinary flags, indicating whether student was active in givenweek and content type, we utilised Bayes Theorem [Bishopand Nasrabadi,2006] for determining the probability that thestudent will fail to complete the module. Moreover, we anal-ysed each of the content types in terms of mean number ofstudents succeeding in the module based on activity or inac-tivity in the given content type. Based on this investigationwe have selected a set of content types which significantlyinfluence students’ performance, these were then used in thefurther analyses.4.2 GUHAGeneral Unary Hypotheses Automaton (GUHA), originallypublished in [Ha´jek et al.,1966], is one of the oldest datamining methods for automatic discovery of new interestinghypotheses from the data. To achieve this goal, GUHA usesvarious different procedures (ASSOC, IMPL, CORREL). Thechoice of the procedure to use depends mainly on the userneeds and his experience. We have selected the ASSOC pro-cedure [Rauch and Simunek,2001], which allows to discoverinteresting associations between attributes in the data. Theinterestingness of the association is mostly based on theirco-occurrence. The ASSOC procedure allows to limit theresulting rules by specifying constraints on the attributes.This property is important for our field of interest.For our research, we used the ASSOC procedure that isimplemented in the 4ft-Miner module within the Lisp-Minersoftware tool 3. The specification of the constraints enabledus to restrict the rules only to those, which cover studentsthat fail or succeed in the TMA. We used three basic typesof features for the analysis introduced in Section 3.The search space for both types of binary flags was rea-sonable to perform analysis. For weekly aggregated counts,it was necessary to reduce the search space via intervaldiscretisation. For this purpose, we utilized LISp-minerand unsupervised equal frequency discretisation [Wong andChiu,1987].GUHA method produce large set of hypotheses. The ex-ample of such results with the categorized binary flags aredepicted in the Figure 1. Unfortunately the information con-tained in the output is difficult to interpret. Moreover, theinformation of the time dimension is lost and this is evenworse when using various content types. For us, this wasthe motivation to look for another modelling method.4.3 Markov chain-based analysis of student ac-tivity3LISp-Miner lispminer.vse.cz/ – software tool for imple-mentation of the GUHA method.Figure 1: Screenshot with discovered rules fromLISp-miner.Scenario TMA 1 TMA 1not submit success1. zero in any of 0 - 4 51.6 42.62. zero only in 1 - 4 95.7 4.33. zero only in 2 - 4 92.3 7.74. zero only in 3 - 4 100.0 0.05. zero only in 4 54.3 45.76. zero only in 0 15.4 71.87. zero only in 0 - 1 6.7 86.78. zero only in 0 - 2 15.4 69.29. zero only in 0 - 3 57.1 42.910. zero in at least one of 0 - 3,non-zero in 418.2 71.611. zero in at least one of 0 - 2,non-zero in 3 - 414.6 75.112. zero in at least one of 0 - 1,non-zero in 2 - 49.3 80.4Table 1: Summary of examined situations for stu-dents behaviour evaluation (data in % of total num-ber of course students)In this part of analysis we examined the differences in in-tensity of student activity between students who were suc-cessful in the first TMA (TMA 1) and those who did notsubmit TMA 1. Students who failed in TMA 1 are not in-cluded in this analysis, due to the fact that they representonly a small portion of students who submit TMA 1. In thisstage of research we analysed only students who at least oncehad zero VLE activity in one of the weeks under consider-ation. In VLE passive students represent the specific groupimportant while looking for potentially at-risk students. Onthis group we have studied different scenarios - the list ofthem is displayed in Table 1. Moreover, the Table 1 showspercentage of students who behaved according to given Sce-nario and were successful in TMA 1, in comparison to thosewho did not submit TMA 1, as well. Numbers in columnScenario represents particular weeks of course.Based on the data in Table 1, we can identify behaviour ofat-risk students. This is evident especially in scenarios 2, 3,4 and 7. This shows us that students who tend to reach zeroVLE activity in later weeks are more probable not tu submit(scenarios 2, 3, 4). On the other hand those who have zeroVLE activity in earlier weeks and later start to show interestFigure 2: Representation of students behaviour inscenario 3 (TMA 1 was not submitted)Figure 3: Representation of students behaviour inscenario 3 (TMA 1 was passed)represented by VLE activity, raise their chance to success inTMA 1 (scenarios 6, 7, 8).Figures 2 and 3 specify more closely the situations fromthe scenario 3 (with the TMA 1 not submitted, and TMA1 passed, respectively). Colour tones of arrows differ (fromwhite to red) depending on the percentage of students whomoved in given direction. The more red the colour the biggerthe percentage of students it represents. The rows representthe weeks in which VLE activity was measured. First rowshows activity before the beginning of the course (Week 0),the other four rows capture the VLE activity in week 1, 2,3 and 4 respectively. The columns represent the intervals ofVLE activity - different colour of the node mean differentinterval. First column is zero VLE activity, while in othercolumns the activity is divided into intervals with the cutpoints in multiples of 30.This type of analysis enables us to look for specific pat-terns in students’ behaviour. In the similar way we cananalyse the different types of VLE activities as shown inFigure 4. In this case the nodes represent not the intensityof student activity, but capture the interest of student inspecific content type. And (unlike the previous two figures)this directed acyclic graph as a whole depicts the Markovchain [Norris,1998].5. CONCLUSIONFigure 4: Markov chain for the various activity typescombinations in weeks 0-5.In this paper we have examined two methods for analysingactivity of students in the online virtual learning environ-ment before the first tutor marked assignment – GUHA andMarkov chain-based graphical models. Both methods pro-vide useful insights into the students’ behaviour during theirstudies. The benefit of the latter lies mostly in its graph-ical output, which might be easier to interpret and couldpotentially provide support in planning interventions, andin the possibility to model time dependencies of the studentactivities. We believe that the understanding of the studentbehavioural patterns will also be useful for building betterpredictive models of student performance.ReferencesChristopher M Bishop and Nasser M Nasrabadi. 2006.Pattern recognition and machine learning, volume 1.springer New York.Dave Cormier. 2008. The CCK08 MOOC–connectivismcourse, 1/4 way. Dave’s Educational Blog, 2.Petr Ha´jek, I. Havel, and Michal Chytil. 1966. The guhamethod of automatic hypotheses determination. Com-puting, 1(4):293–308.Daphne Koller, Andrew Ng, Chuong Do, and ZhenghaoChen. 2013. Retention and intention in massive openonline courses: In depth. EDUCAUSE, June.James R Norris. 1998. Markov chains. Number 2008 inCambridge series in statistical and probabilistic math-ematics. Cambridge university press.Jan Rauch and Milan Simunek. 2001. Mining for associationrules by 4ft-miner. In INAP, pages 285–295.Annika Wolff and Zdenek Zdrahal. 2012. Improving reten-tion by identifying and supporting ”at-risk” students.EDUCAUSE Review Online, July/Summer.Annika Wolff, Zdenek Zdrahal, Drahomira Herrmannova,and Petr Knoth. 2013a. Predicting student per-formance from combined data sources. In AlejandroPen˜a-Ayala, editor, Educational Data Mining: Appli-cations and Trends, number 524 in Studies in Compu-tational Intelligence, pages 175–202. Springer Interna-tional Publishing, Cham.Annika Wolff, Zdenek Zdrahal, Andriy Nikolov, and MichalPantucek. 2013b. Improving retention: predicting at-risk students by analysing clicking behaviour in a vir-tual learning environment. In Third Conference onLearning Analytics and Knowledge (LAK 2013). ISBN978-1-4503-1785-6.Andrew K. C. Wong and David K. Y. Chiu. 1987. Syn-thesizing statistical knowledge from incomplete mixed-mode data. IEEE Trans. Pattern Anal. Mach. Intell.,9(6):796–805, June.

Modelling student online behaviour in a virtual learning environment

http://oro.open.ac.uk/40670/1/LA_machinelearning_submission_4.pdf

Modelling student online behaviour in a virtual learning environment

Abstract

Similar works

Full text

Available Versions

Open Research Online