Soon Seng Thah,

English

Open University Malaysia Knowledge Repository

OUMLearning Outcomes in ODL: A Study of Learners’ ExperienceOUM SeminarSoon Seng Thah1OUMPresentation OutlineReview of LiteratureFindingsConclusionIntroduction1245Methodology3OUMIntroduction• On 16 March 2016, a Google Scholar  search on “online education” has yielded 3.38 million hits indicating vast potential  of online education.• Although there is an abundance of research available on online education, very little of it centres on the academic impact and learning effects of this modality on students (Jahng, 2007; Kirtman, 2009; Tallent-Runnels et al., 2006).• There is a need for research to improve our understanding of the elements which affect learners’ learning outcomes.3OUMIntroduction• What is learning outcome?The National Institute for Learning Outcomes Assessment (2012) defines learning outcomes as “the expected knowledge, skills, attitudes, competencies, and habits of mind that students are expected to acquire at an institution of higher education.” • The overall goal of education is learning; therefore, learning effectiveness must be the primary factor for which quality in education is measured or judged.4OUMIntroduction• A study of this nature is important as it exposes the fundamental issues of the ODL delivery method from the perspectives of OUM’s blended and online learning programmes.• Using advanced statistical analysis via Structural Equation Modeling a deterministic approach can be undertaken to determine factors and mediators which ascertain learner outcomes.• Findings can then be used by decision-makers to leverage further on the delivery approach and student learning  for instructional improvement.5OUMIntroduction• Research Questions To what extent do socio-demographic indicators affect OUM’s learning outcomes?; What are the factors which explain the effectiveness of OUM’s learning outcomes?; and How do these factors mediate learning outcomes in OUM’s academic programmes?6OUMReview of Literature• Learning outcomes have their roots on learning theories.  Of relevance is the social constructivist theory by Vygotsky.• Theory is related to constructivism, an important aspect of technology-based mediation of learning.• Social constructivism maintains that human acquisition of knowledge and skills are socially situated and can be “constructed” through interaction with others.• Groups construct knowledge for one another, collaboratively creating a shared culture. • Discussion and interactive discourse  within this “culture” promote learning because learners can interact with peers and facilitators within a learning platform, for e.g. through OUM’s MyVLE.7OUMReview of Literature• As with OUM’s approach to teaching and learning, we can also incorporate connectivism into its paradigm.• McHaney (2011)  and Siemens (2004) introduced the concept of connectivism, which conceptualises learning as a shared, community experience which is facilitated by technology. – Connectivism suggests that whereas the other learning theories assume that learning occurs within the individual, learning can also occur outside the individual and be stored and organised by technology. This learning theory suggests that due to the vast capability of technology to store and redeploy knowledge, what becomes a more important component of learning is how to find and access the information already available. 8OUMLiterature Review• Research shows that students who are characterised as the most successful in an online learning environment tend to be motivated, independent, and organised (Phipps & Merisotis, 1999) and have good self-regulation strategies (McMahon & Oliver, 2001).• McMahon & Oliver (2001) claimed that students in an online environment must accept a greater responsibility for their own learning since they have limited access to instructional support.• As Hannafin et al. (2003) indicates, attitudes toward web-based instruction impact students' abilities to learn in that media, so student data concerning distance learning is needed. – Based on their particular academic background, students who routinely use computers are better able to learn via computer lessons. 9OUMLiterature Review• Shelton & Saltsman (2005) found the most common complaints from faculty members regarding online education are a lack of understanding for this method of teaching, a lack of institutional support, and fear that the quality of education in the online environment suffers.• Research has found that online courses that utilise tools to augment interaction (student-to-student and student–to-instructor) and  engagement further enhance learning outcomes and overall satisfaction (McFarland & Hamilton, 2005; Palmer & Holt, 2008).• Research consistently shows that engagement and interaction in the online classroom leads to student learning outcomes, and a quality online learning experience (McFarland & Hamilton, 2005; Dykman & Davis, 2008; Palmer & Holt, 2008).10OUMMethodology• Survey was undertaken by IQRI/CRI in 2015 to study some aspects of learning outcomes in OUM’s e-learning programmes.• Survey research design used via the administration of a web-based questionnaire.• Survey instrument comprises demographic information, rating scale items (5-point Likert scale), and open-ended questions.• Sample comprising 397 learners from different faculties from OUM’s Learning Centres.11OUMMethodology• 1. All rating scale variables comprising 24 items were factor-analysed using the Principal Component Method to determine salient factors. 3 factors were extracted.• 2. The items nested within the 3 factors were then summed up to obtain the composite mean scores of individual responses.• 3. Reliability tests were undertaken to test the internal consistency of items which made up the 3 factors.• 4. SEM was used to construct the path diagram consisting of observed variables and latent constructs. This is checked for overall fit using Goodness-of-Fit statistics.• 5. The significant predictors and highly correlated variables were then used to construct the mediation framework derived from the path diagram.• 6. Inferences and conclusions were made from the empirical evidence derived from Step 1 to Step 5 above.12OUMMethodology• Very high reliability (Cronbach alpha) values were seen in the 3 constructs indicating high internal consistency:Constructs Cronbach'sAlphaCronbach'sAlpha Based on Standardised ItemsN of Items13Teaching & Learning Materials.944 .945 13e-Tutoring .964 .965 6Learning Experience .882 .879 5OUMFindings• Learners’ perceptions of teaching and learning materials are highest (mean = 3.69; S.D. = .616), followed by role of e-tutors (mean = 3.60; S.D. = .871) and learning experience (mean = 3.24; S.D. = .787).What is your gender? Teaching and materialsRole of the e-TutorLearning ExperienceFemale Mean143.7092 3.6393 3.2547N 286 286 286Std. Deviation .56345 .80684 .75395Male Mean 3.6386 3.4922 3.2050N 111 111 111Std. Deviation .73536 1.01351 .86914Total Mean 3.6895 3.5982 3.2408N 397 397 397Std. Deviation .61620 .87076 .78705OUMFindings• Female learners perceived teaching and learning materials, e-tutoring and learning experiences better than male studentsWhat is your gender? N Mean Std. DeviationStd. Error Mean15Teaching and materials Female 286 3.7092 .56345 .03332Male 111 3.6386 .73536 .06980e-Tutoring Female 286 3.6393 .80684 .04771Male 111 3.4922 1.01351 .09620Learning Experience Female 286 3.2547 .75395 .04458Male 111 3.2050 .86914 .08250OUMFindings• Using t-test for independent samples, the differences  in male and female perceptions are not significant in all the 3 constructs (p > .05)Levene's Test for Equality of Variances t-test for Equality of Means16F Sig. t dfSig. (2-tailed)Mean DifferenceStd. Error DifferenceTeaching and materialsEqual variances not assumed7.133 .008 .914 162.582 .362 .07065 .07734e-Tutoring Equal variances not assumed6.790 .010 1.370 166.870 .173 .14707 .10738Learning ExperienceEqual variances assumed2.429 .120 .563 395 .573 .04963 .08809OUMFindings• There is a tendency that the older the learners, the more positive their perceptions of teaching and learning materials, e-tutoring and learning experience.N Mean Std. Deviation Std. ErrorTeaching and materials18 to 24 76 3.7270 .52191 .0598725 to 34 177 3.6262 .68849 .0517535 to 44 106 3.7150 .60812 .0590745 to 54 30 3.8710 .39035 .071271755 to 64 8 3.7160 .44335 .15675Total 397 3.6895 .61620 .03093e-Tutoring 18 to 24 76 3.5848 .83727 .0960425 to 34 177 3.5420 .92442 .0694835 to 44 106 3.6602 .84858 .0824245 to 54 30 3.7889 .76180 .1390855 to 64 8 3.4317 .60256 .21304Total 397 3.5982 .87076 .04370Learning Experience18 to 24 76 3.2069 .78125 .0896225 to 34 177 3.2133 .82915 .0623235 to 44 106 3.3169 .77788 .0755545 to 54 30 3.2570 .61439 .1121755 to 64 8 3.1000 .65900 .23299Total 397 3.2408 .78705 .03950OUMFindings• No significant differences exist between age groups (p > .05).ANOVASum of Squares df Mean Square F Sig.Teaching and materialsBetween Groups 1.879 4 .470 1.240 .293Within Groups 148.482 392 .379Total 150.362 39618Role of the e-Tutor Between Groups 2.294 4 .573 .754 .556Within Groups 297.961 392 .760Total 300.254 396Learning Experience Between Groups 1.002 4 .250 .402 .807Within Groups 244.297 392 .623Total 245.299 396OUMFindings• Ratings for those with or without previous experience in online course were relatively high.Group StatisticsHave you taken a FULLY ONLINE course before? N Mean Std. DeviationStd. Error MeanTeaching and materials Yes 214 3.6728 .56549 .0386619No 183 3.7090 .67172 .04966e-Tutoring Yes 214 3.5863 .84655 .05787No 183 3.6121 .90038 .06656Learning Experience Yes 214 3.2245 .81961 .05603No 183 3.2598 .74896 .05536OUMFindings• There is no significant difference (p > .05) whether learners come in with or without previous experience in online course based on their ratings of : i. teaching and learning materials, ii.  e-tutoring, and iii. learning experience.Independent Samples TestLevene's Test for Equality of Variances t-test for Equality of MeansF Sig. t df Sig. (2-tailed)Mean DifferenceStd. Error Difference20Teaching and materialsEqual variances assumed .466 .495 -.582 395 .561 -.03613 .06209Equal variances not assumed -.574 357.310 .566 -.03613 .06293e-Tutoring Equal variances assumed .025 .873 -.294 395 .769 -.02576 .08777Equal variances not assumed -.292 377.058 .770 -.02576 .08820Learning ExperienceEqual variances assumed 3.956 .047 -.445 395 .656 -.03533 .07932Equal variances not assumed -.448 393.246 .654 -.03533 .07877OUMFindings• There is a tendency for those  who spent more time online to rate relatively higher.N Mean Std. Deviation Std. ErrorTeaching and materialsLess than 30 minutes104 3.4670 .70085 .0687230 to 60 minutes 183 3.7408 .52627 .038901 to 2 hours 69 3.8285 .58655 .070612 to 4 hours 25 3.8714 .62613 .12523More than 4 hours 16 3.6648 .76400 .1910021Total 397 3.6895 .61620 .03093e-Tutoring Less than 30 minutes104 3.2674 1.01129 .0991730 to 60 minutes 183 3.6664 .79460 .058741 to 2 hours 69 3.8269 .65634 .079012 to 4 hours 25 3.8200 .83322 .16664More than 4 hours 16 3.6354 1.06322 .26581Total 397 3.5982 .87076 .04370Learning Experience Less than 30 minutes104 3.0319 .82609 .0810030 to 60 minutes 183 3.2364 .70718 .052281 to 2 hours 69 3.5121 .69855 .084102 to 4 hours 25 3.4335 .99178 .19836More than 4 hours 16 3.1774 1.05158 .26289Total 397 3.2408 .78705 .03950OUMFindings• There is a significant difference in the time learners spent in logging-in to the online sessions with their ratings associated with i. teaching and learning materials ii.  e-tutoring and iii. learning experienceANOVASum of Squares df Mean Square F Sig.22Teaching and materials Between Groups 7.803 4 1.951 5.364 .000Within Groups 142.558 392 .364Total 150.362 396Role of the e-Tutor Between Groups 17.090 4 4.272 5.915 .000Within Groups 283.165 392 .722Total 300.254 396Learning Experience Between Groups 10.614 4 2.653 4.432 .002Within Groups 234.686 392 .599Total 245.299 396OUMFindings• Significant differences exist between time spent online and their ratings in Teaching and Learning materials for specific groups .(I) When I log on the fully online course, my fully online sessions averages...(J) When I log on the fully online course, my fully online sessions averages... Mean Difference (I-J) Std. Error Sig.Less than 30 minutes 30 to 60 minutes -.27387* .07405 .0021 to 2 hours -.36154* .09363 .0012 to 4 hours -.40448* .13433 .02323More than 4 hours -.19787 .16194 .73930 to 60 minutes Less than 30 minutes .27387* .07405 .0021 to 2 hours -.08766 .08519 .8422 to 4 hours -.13061 .12858 .848More than 4 hours .07601 .15722 .9891 to 2 hours Less than 30 minutes .36154* .09363 .00130 to 60 minutes .08766 .08519 .8422 to 4 hours -.04295 .14077 .998More than 4 hours .16367 .16733 .8652 to 4 hours Less than 30 minutes .40448* .13433 .02330 to 60 minutes .13061 .12858 .8481 to 2 hours .04295 .14077 .998More than 4 hours .20662 .19307 .822More than 4 hours Less than 30 minutes .19787 .16194 .73930 to 60 minutes -.07601 .15722 .9891 to 2 hours -.16367 .16733 .8652 to 4 hours -.20662 .19307 .822OUMFindings• Significant differences exist between time spent online and their ratings in e-tutoring for specific groups.(I) When I log on the fully online course, my fully online sessions averages...(J) When I log on the fully online course, my fully online sessions averages... Mean Difference (I-J) Std. Error Sig.Less than 30 minutes 30 to 60 minutes -.39895* .10437 .0011 to 2 hours -.55945* .13196 .0002 to 4 hours -.55259* .18931 .030More than 4 hours -.36800 .22824 .4902430 to 60 minutes Less than 30 minutes .39895* .10437 .0011 to 2 hours -.16050 .12007 .6682 to 4 hours -.15364 .18122 .915More than 4 hours .03094 .22157 1.0001 to 2 hours Less than 30 minutes .55945* .13196 .00030 to 60 minutes .16050 .12007 .6682 to 4 hours .00686 .19840 1.000More than 4 hours .19144 .23583 .9272 to 4 hours Less than 30 minutes .55259* .18931 .03030 to 60 minutes .15364 .18122 .9151 to 2 hours -.00686 .19840 1.000More than 4 hours .18458 .27211 .961More than 4 hours Less than 30 minutes .36800 .22824 .49030 to 60 minutes -.03094 .22157 1.0001 to 2 hours -.19144 .23583 .9272 to 4 hours -.18458 .27211 .961OUMFindings• Significant differences exist between time spent online and learners’ ratings on learning experience for specific groups.(I) When I log on the fully online course, my fully online sessions averages...(J) When I log on the fully online course, my fully online sessions averages... Mean Difference (I-J) Std. Error Sig.Less than 30 minutes 30 to 60 minutes -.20450 .09502 .2001 to 2 hours -.48020* .12014 .0012 to 4 hours -.40165 .17235 .137More than 4 hours -.14551 .20779 .95630 to 60 minutes Less than 30 minutes .20450 .09502 .200251 to 2 hours -.27570 .10931 .0882 to 4 hours -.19714 .16498 .754More than 4 hours .05899 .20172 .9981 to 2 hours Less than 30 minutes .48020* .12014 .00130 to 60 minutes .27570 .10931 .0882 to 4 hours .07855 .18062 .993More than 4 hours .33469 .21470 .5252 to 4 hours Less than 30 minutes .40165 .17235 .13730 to 60 minutes .19714 .16498 .7541 to 2 hours -.07855 .18062 .993More than 4 hours .25614 .24772 .839More than 4 hours Less than 30 minutes .14551 .20779 .95630 to 60 minutes -.05899 .20172 .9981 to 2 hours -.33469 .21470 .5252 to 4 hours -.25614 .24772 .839OUMFindings : Factors affecting Learners’ Outcomes• Factor analysis using IBM SPSS Statistics• Principal Component MethodMethod• Presence of correlations indicating appropriateness• A significance level of less than .05 (chi-sq=6471.362; df=276; p<.001)Bartlett’s test of sphericity • Determines sampling adequacy• Value is very high, i.e. .954 – Meritorious (Hair, et al. (2010) Kaiser-Meyer-Olkin Measure of Sampling Adequacy (MSA) 26OUMFindings : Factors affecting Learners’ Outcomes1stFactor• “Teaching and learning materials”• Contributes 56.028% to the variance• “E-tutoring”Key Success Factors2ndFactor• Contributes 7.685%  of the variance3rdFactor• “Learners experience”• Contributes 6.433% of the varianceFactor Determination  based on Percentage of Variance Method 27OUMFindings: Confirmatory SpecificationsStructural Equation Modeling (using AMOS)• Confirmatory analysis of key factors which influence learning outcomes.• Analysis would be a deterministic approach for decision-making.• A holistic view of the over-riding mechanism which leverages on building observed variables to be tested against latent constructs in a structural and measurement model.28OUMFindings: The Path Diagram.4229.36OUMFindings: Confirmatory Specifications showing Goodness-of-Fit• Goodness of Fit Indices (GFI)Model RMR GFI AGFI PGFIDefault model.065 .820 .781 .672Saturated model.000 1.000Independen30ce model.405 .123 .047 .113• Baseline Comparison: Comparative Fit IndicesModelNFIDelta1RFIrho1IFIDelta2TLIrho2CFIDefault model.887 .874 .912 .900 .911Saturated model1.000 1.000 1.000Independence model.000 .000 .000 .000 .000OUM Findings: Confirmatory Specifications• RMSEAModel RMSEA LO 90 HI 90 PCLOSEDefault model.091 .085 .096 .00031Independence model.287 .282 .292 .000OUM Findings: Mediation• Does e-tutoring in OUM mediate against Learning Experience?• Do OUM’s teaching and learning materials have a bearing on Learning Experience of learners?Answer: Yes. This mediation is partial.32OUM Findings: Mediation & Prediction• Which variables predict OUM’s ODL outcomes?Model SummaryModel R R SquareAdjusted R SquareStd. Error of the EstimateChange StatisticsR Square Change F Change df1 df2Sig. F Change1 .708a .502 .500 .55266 .502 341.189 1 339 .0002 .714b .509 .506 .54926 .008 5.211 1 338 .023a. Predictors: (Constant), T_L_Materials33b. Predictors: (Constant), T_L_Materials, E_TutoringANOVAaModelSum of Squares df Mean Square F Sig.1 Regression 104.211 1 104.211 341.189 .000bResidual 103.543 339 .305Total 207.754 3402 Regression 105.783 2 52.892 175.319 .000cResidual 101.971 338 .302Total 207.754 340a. Dependent Variable: Learning_Expb. Predictors: (Constant), T_L_Materialsc. Predictors: (Constant), T_L_Materials, E_TutoringOUMFindings: Mediation• SEM analysis proves the presence of partial mediation in students’ learning  experience.• The path regression weights (standardised beta coefficients) show that OUM’s teaching and learning meterials have a bearing on learners’ learning experience. T&L materials contribute 50.2% to the variation of Learning Experience.• When T&L materials  are included with e-tutoring, the contribute  of T&L materials and e-tutoring made up 50.9% to the learning experience. Hence of the 2, T&L contributes much more than e-tutoring itself. • Age, Gender, CGPA, Time online, whether a learner had experience in online learning or not do not significantly predict Learner Experience in OUM’s programmes.34OUMConclusions• SEM analysis confirms factor analysis extraction of factors.• The hypothesised model fits well with data structure.• Since data shows a generally successful implementation of ODL outcomes it can therefore be concluded that OUM’s ODL teaching and learning resources, e-tutoring system and learning experiences have resulted in effective and significant outcomes of the desired delivery system – from learners’ perspectives.35OUMThank YouEmail: soon@oum.edu.my36

Learning Outcomes in ODL: A Study of Learners’ Experience

http://library.oum.edu.my/repository/1044/1/library-document-1044.pdf

Learning Outcomes in ODL: A Study of Learners’ Experience

Abstract

Similar works

Full text

Available Versions

Open University Malaysia Knowledge Repository