Heleou, S., Drachsler, H., & Gillet, D. (2009). Evaluation of Recommender Systems for Technology-Enhanced Learning: Challenges and Possible Solutions. 1st workshop on Context-aware Recommender Systems for Learning at the Alpine Rendez-Vous. November, 30-December, 3, 2009, Garmisch-Patenkirchen, Germany.This paper discusses challenges and possible solutions of recommender
systems for Technology-Enhanced Learning (TEL). It also briefly presents
the the 3A contextual recommender system and explores its applicability
and evaluation in the context of learners using multiple Web 2.0
applications

Drachsler, Hendrik

Gillet, Dennis

Sandy, Heleau

DSpace at Open Universiteit Nederland

Evaluation of Recommender Systems for Technology-Enhanced 
Learning:Challenges and Possible Solutions 
 
 
Sandy El Helou1, Hendrik Drachsler2 ,Denis Gillet3 
 
 
1 Swiss Federal Institute of Lausanne (EPFL) 
1015 Lausanne, Switzerland 
+41 21 6936945,  sandy.elhelou@epfl.ch 
 
2Open University of the Netherlands (OUNL), 
P.O. Box 2960, 6401 DL Heerlen, The Netherlands 
+31 45 5762605, hendrik.drachsler@ou.nl 
 
3 Swiss Federal Institute of Lausanne (EPFL) 
1015 Lausanne, Switzerland 
+41 21 6935168,  denis.gillet@epfl.ch 
 
This paper discusses challenges and possible solutions of recommender 
systems for Technology-Enhanced Learning ( TEL). It also briefly presents 
the the 3A contextual recommender system and explores its applicability 
and evaluation in the context of learners using multiple Web 2.0 
applications. 
 
Background 
 
Web 2.0 social software applications are characterized by their low-entry 
barrier, user-friendly interfaces and easy-to-use features fostering user-
generated content, facilitating information sharing and dissimination as 
well as online feedback [1]. Their initial success in informal settings for 
maintaining and building friendship ties, has led to their emerging adoption 
in institutional and corporate environments (enterprise 2.0, eLearning 2.0) 
in order to enhance the learning experience. Learners use Web 2.0 
applications to find knowledge artifacts, learning activities, individuals and 
groups that can help them satisfy their learning needs. They also create, 
share, tag, rate and comment learning content and activities available online 
helping by that other learners to also find what they need. 
Web 2.0 tools open the door for a plethora of information that makes it hard 
on the learner to get an overview and select what can satisfy their learning 
needs. Consequently, personalized and contextualized recommender 
systems should be built on top of Web 2.0 tools, to help each learner find 
the most suitable information depending on his/her learning goals. As an 
example, the 3A contextual recommender system [2] simultaneously ranks 
knowledge artifacts, individuals and group learning activities according to 
their global popularity as well as their relevancy to the targer learner's 
context and their closeness to his/her "trusted" network. 
A major challenge related with the application of recommender systems in 
Technology-Enhanced Learning ( TEL), lies in finding proper evaluation 
frameworks to assess their efficiency and effectiveness [3]. 
 
 
Principal contributions 
 
In this paper, we discuss the challenges related to the evaluation of 
recommender systems for TEL. An evaluation method for the 3A contextual 
recommender system that tries to deal with these challenges is also 
presented. 
Researchers working on recommender systems do not necessarily have a 
social software application involving considerable number of users. That's 
why they have to rely on external applications that have richer and bigger 
datasets in order to test their algorithm. 
Nevertheless, popular social software applications do not all offer public 
API ( e.g. LinkedIn).  Thus, it is not always possible for "independent" 
third-party recommender systems to perform online longitudinal studies 
that involve huge and rich datasets with many users. Moreover, even when 
the API is public, only a subset of data and services are made accessible. In 
addition, since external recommender systems using API of popular 
platforms are not directly proposed (or imposed) by the hosting platforms, 
there ought to be compelling reasons for learners to add them, use them, 
and possibly provide a direct feedback. 
The evaluation of the 3A recommender system is planned to be carried out 
using the ReMashed environment where learners can specify certain Web 
2.0 tools and combine them in a Mash-Up Personal Learning Environment. 
Using ReMashed, we will be able to evaluate the algorithm using rich and 
large datasets fetched from different Web 2.0 tools such as Slideshare and 
Delicious. The fact that learners will be able to receive recommendations 
for data aggregated from mutiple Web sites  is believed to motivate them to 
use it, check the suggested content and eventually rate it. We are also 
considering to extend the ReMashed system to include social networking 
sites such as LinkedIn (if our request is accepted) and Facebook for a better 
evaluation of the 3A recommender system in a collaborative setting. 
The challenges, and eventual results obtained from this evaluation will be 
summarized in this paper. 
 
 
References 
[1] EL Helou, S., Gillet, D., and Yu, C.Towards a Cross Fertilization Between Social Software and 
CSCW. The 10th European Conference on Computer Supported Co-operative Work, Limerick, 
Ireland, Septembre 24-28, 2007. 
[2] El Helou, S., Gillet D., Salzmann C., and Sire S. The 3A Contextual Ranking System: 
Simultaneously Recommending Actors, Assets, and Group Activities. To be Published in 
Proceedings of  Recommender Systems 2009. ACM, 2009. 
[3]Drachsler, H., Pecceu, D., Arts, T., Hutten, E., Rutledge, L., Van Rosmalen, P., Hummel, H. G. K., 
& Koper, R. (2009). ReMashed - Recommendations for Mash-Up Personal Learning Environments. 
In U. Cress, V. Dimitrova, & M. Specht (Eds.), Learning in the Synergy of Multiple Disciplines, EC-
TEL 2009 (Vol. LNCS 5794). Nice, France: Springer. 
[4]Drachsler, H., Hummel, H. G. K., & Koper, R. (2009). Identifying the Goal, User model and 
Conditions of Recommender Systems for Formal and Informal Learning. Journal of Digital 
Information, 10. Retrieved July 7, 2009 from http://journals.tdl.org/jodi/article/view/442/279. 
[5]Li, L., Muller, M. J., Geyer, W., Dugan, C., Brownholtz, B., and Millen, D. R. 2007. Predicting 
individual priorities of shared activities using support vector machines. In Proceedings of the 
Sixteenth ACM Conference on Conference on information and Knowledge Management (Lisbon, 
Portugal, November 06 - 10, 2007). CIKM '07. ACM, New York, NY, 515-524. 
  


Evaluation of Recommender Systems for Technology-Enhanced Learning: Challenges and Possible Solutions

This paper discusses challenges and possible solutions of recommender systems for Technology-Enhanced Learning (TEL). It also briefly presents the the 3A contextual recommender system and explores its applicability and evaluation in the context of learners using multiple Web 2.0 applications

Open University of the Netherlands Research Portal

Open Universiteit www.ou.nl Evaluation of Recommender Systems for Technology-Enhanced LearningCitation for published version (APA):Sandy, H., Drachsler, H., & Gillet, D. (2009). Evaluation of Recommender Systems for Technology-EnhancedLearning: Challenges and Possible Solutions. Paper presented at STELLAR Alpine Rendez-Vous 2009,Garmisch-Patenkirchen, Bavaria, Germany.Document status and date:Published: 08/12/2009Document Version:Peer reviewed versionDocument license:CC BY-NC-NDPlease check the document version of this publication:• A submitted manuscript is the version of the article upon submission and before peer-review. There can be important differences betweenthe submitted version and the official published version of record. People interested in the research are advised to contact the author for thefinal version of the publication, or visit the DOI to the publisher's website.• The final author version and the galley proof are versions of the publication after peer review.• The final published version features the final layout of the paper including the volume, issue and page numbers.Link to publicationGeneral rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.• Users may download and print one copy of any publication from the public portal for the purpose of private study or research.• You may not further distribute the material or use it for any profit-making activity or commercial gain• You may freely distribute the URL identifying the publication in the public portal.If the publication is distributed under the terms of Article 25fa of the Dutch Copyright Act, indicated by the “Taverne” license above, pleasefollow below link for the End User Agreement:https://www.ou.nl/taverne-agreementTake down policyIf you believe that this document breaches copyright please contact us at:pure-support@ou.nlproviding details and we will investigate your claim.Downloaded from https://research.ou.nl/ on date: 16 Jul. 2023Evaluation of Recommender Systems for Technology-Enhanced Learning:Challenges and Possible Solutions   Sandy El Helou1, Hendrik Drachsler2 ,Denis Gillet3   1 Swiss Federal Institute of Lausanne (EPFL) 1015 Lausanne, Switzerland +41 21 6936945,  sandy.elhelou@epfl.ch  2Open University of the Netherlands (OUNL), P.O. Box 2960, 6401 DL Heerlen, The Netherlands +31 45 5762605, hendrik.drachsler@ou.nl  3 Swiss Federal Institute of Lausanne (EPFL) 1015 Lausanne, Switzerland +41 21 6935168,  denis.gillet@epfl.ch  This paper discusses challenges and possible solutions of recommender systems for Technology-Enhanced Learning ( TEL). It also briefly presents the the 3A contextual recommender system and explores its applicability and evaluation in the context of learners using multiple Web 2.0 applications.  Background  Web 2.0 social software applications are characterized by their low-entry barrier, user-friendly interfaces and easy-to-use features fostering user-generated content, facilitating information sharing and dissimination as well as online feedback [1]. Their initial success in informal settings for maintaining and building friendship ties, has led to their emerging adoption in institutional and corporate environments (enterprise 2.0, eLearning 2.0) in order to enhance the learning experience. Learners use Web 2.0 applications to find knowledge artifacts, learning activities, individuals and groups that can help them satisfy their learning needs. They also create, share, tag, rate and comment learning content and activities available online helping by that other learners to also find what they need. Web 2.0 tools open the door for a plethora of information that makes it hard on the learner to get an overview and select what can satisfy their learning needs. Consequently, personalized and contextualized recommender systems should be built on top of Web 2.0 tools, to help each learner find the most suitable information depending on his/her learning goals. As an example, the 3A contextual recommender system [2] simultaneously ranks knowledge artifacts, individuals and group learning activities according to their global popularity as well as their relevancy to the targer learner's context and their closeness to his/her "trusted" network. A major challenge related with the application of recommender systems in Technology-Enhanced Learning ( TEL), lies in finding proper evaluation frameworks to assess their efficiency and effectiveness [3].   Principal contributions  In this paper, we discuss the challenges related to the evaluation of recommender systems for TEL. An evaluation method for the 3A contextual recommender system that tries to deal with these challenges is also presented. Researchers working on recommender systems do not necessarily have a social software application involving considerable number of users. That's why they have to rely on external applications that have richer and bigger datasets in order to test their algorithm. Nevertheless, popular social software applications do not all offer public API ( e.g. LinkedIn).  Thus, it is not always possible for "independent" third-party recommender systems to perform online longitudinal studies that involve huge and rich datasets with many users. Moreover, even when the API is public, only a subset of data and services are made accessible. In addition, since external recommender systems using API of popular platforms are not directly proposed (or imposed) by the hosting platforms, there ought to be compelling reasons for learners to add them, use them, and possibly provide a direct feedback. The evaluation of the 3A recommender system is planned to be carried out using the ReMashed environment where learners can specify certain Web 2.0 tools and combine them in a Mash-Up Personal Learning Environment. Using ReMashed, we will be able to evaluate the algorithm using rich and large datasets fetched from different Web 2.0 tools such as Slideshare and Delicious. The fact that learners will be able to receive recommendations for data aggregated from mutiple Web sites  is believed to motivate them to use it, check the suggested content and eventually rate it. We are also considering to extend the ReMashed system to include social networking sites such as LinkedIn (if our request is accepted) and Facebook for a better evaluation of the 3A recommender system in a collaborative setting. The challenges, and eventual results obtained from this evaluation will be summarized in this paper.   References [1] EL Helou, S., Gillet, D., and Yu, C.Towards a Cross Fertilization Between Social Software and CSCW. The 10th European Conference on Computer Supported Co-operative Work, Limerick, Ireland, Septembre 24-28, 2007. [2] El Helou, S., Gillet D., Salzmann C., and Sire S. The 3A Contextual Ranking System: Simultaneously Recommending Actors, Assets, and Group Activities. To be Published in Proceedings of  Recommender Systems 2009. ACM, 2009. [3]Drachsler, H., Pecceu, D., Arts, T., Hutten, E., Rutledge, L., Van Rosmalen, P., Hummel, H. G. K., & Koper, R. (2009). ReMashed - Recommendations for Mash-Up Personal Learning Environments. In U. Cress, V. Dimitrova, & M. Specht (Eds.), Learning in the Synergy of Multiple Disciplines, EC-TEL 2009 (Vol. LNCS 5794). Nice, France: Springer. [4]Drachsler, H., Hummel, H. G. K., & Koper, R. (2009). Identifying the Goal, User model and Conditions of Recommender Systems for Formal and Informal Learning. Journal of Digital Information, 10. Retrieved July 7, 2009 from http://journals.tdl.org/jodi/article/view/442/279. [5]Li, L., Muller, M. J., Geyer, W., Dugan, C., Brownholtz, B., and Millen, D. R. 2007. Predicting individual priorities of shared activities using support vector machines. In Proceedings of the Sixteenth ACM Conference on Conference on information and Knowledge Management (Lisbon, Portugal, November 06 - 10, 2007). CIKM '07. ACM, New York, NY, 515-524.   

English

Evaluation of Recommender Systems for Technology-Enhanced Learning:Challenges and Possible Solutions

https://research.ou.nl/files/12472909/Helou_Drachsler_Gillet.pdf

Evaluation of Recommender Systems for Technology-Enhanced Learning: Challenges and Possible Solutions

Abstract

Similar works

Full text

Available Versions

DSpace at Open Universiteit Nederland

Open University of the Netherlands Research Portal

Open University of the Netherlands Research Portal