Off-line vs. On-line Evaluation of Recommender Systems in Small E-commerce

In this paper, we present our work towards comparing on-line and off-line evaluation metrics in the context of small e-commerce recommender systems. Recommending on small e-commerce enterprises is rather challenging due to the lower volume of interactions and low user loyalty, rarely extending beyond a single session. On the other hand, we usually have to deal with lower volumes of objects, which are easier to discover by users through various browsing/searching GUIs. The main goal of this paper is to determine applicability of off-line evaluation metrics in learning true usability of recommender systems (evaluated on-line in A/B testing). In total 800 variants of recommending algorithms were evaluated off-line w.r.t. 18 metrics covering rating-based, ranking-based, novelty and diversity evaluation. The off-line results were afterwards compared with on-line evaluation of 12 selected recommender variants and based on the results, we tried to learn and utilize an off-line to on-line results prediction model. Off-line results shown a great variance in performance w.r.t. different metrics with the Pareto front covering 68\% of the approaches. Furthermore, we observed that on-line results are considerably affected by the novelty of users. On-line metrics correlates positively with ranking-based metrics (AUC, MRR, nDCG) for novice users, while too high values of diversity and novelty had a negative impact on the on-line results for them. For users with more visited items, however, the diversity became more important, while ranking-based metrics relevance gradually decrease.Comment: Submitted to ACM Hypertext 2020 Conferenc

Measuring e-learning systems success

The education sector has been radically affected by developments in information technology. In the education arena, substantial funds have been invested in the systematic development of technology infrastructure. E-learning is believed to be the main platform for adopting and using new and more advanced IT in the education sector. However, measuring the success of e-learning systems is one of the key issues facing universities and educational institutions. Although considerable attention has been paid to the information systems success issue, there remain arguments about which factors are the most telling in measuring information system success. The issue of evaluation of the success of information systems generally, and e-learning systems in particular, has become more complicated due to the differing interests and needs of stakeholders. Different groups of stakeholders deal with e-learning systems in different ways - for instance, students, academic staff, ICT staff, management, and software developers. These stakeholders have substantially different objectives and often there are conflicts between their aims. This study proposes an evaluation methodology model to assess e-learning systems success. The model proposed is one which includes eight constructs: IT infrastructure services; system quality; information quality; service delivery quality; perceived usefulness; user satisfaction; customer value; and organisational value. A range of stakeholders such as students, academic staff, and ICT staff are considered in this model. Three instruments were designed to measure the perceptions of three different stakeholders towards e-learning system success. A quantitative study was conducted at University of Southern Queensland (USQ), with survey responses from 720 students who use the e-learning system, 110 academic staff members, and 22 ICT staff. The results confirm that the study model is valid and reliable to measure the success of e-learning systems from different points of view. Some of the relationships among the constructs in the study model were supported and some were not. The study contributed to the body of knowledge by providing a valid and reliable model to measure the success of e-learning systems. Moreover, this study contributes to the practitioners, recommending universities and educational institutions that develop and support e-learning systems

Building Bridges to an Uncertain Future Lived Now: Lessons from the Use of Participatory Action Research and Theory of Change Towards A Realistic Community-Based Participatory Monitoring and Evaluation System

Building on experience from the CGIAR Research Program on Aquatic Agricultural Systems implemented by WorldFish in the Visayas and Mindanao regions of the Philippines, known as the VisMin Hub, we describe the development and evolution of a monitoring and evaluation (M&E) system emerging from the facilitated action-reflection cycles of testing and adopting theories of change carried out with community partners through participatory action research (PAR). The former guides our community partners and us, as members of the potentially emergent PAR groups, towards the realization of the community’s vision; the latter facilitates learning to understand what, how and why change is unfolding. Unlike the conventional M&E system where indicators are pre-set at the beginning of program implementation, these processes result in an organically-evolved, communitybased participatory M&E system that is continuously revised according to contexts to guide communities towards realizing their visions. Its ultimate outcome is enhanced people’s capacity to own the product and process, giving rise to an internally-driven change. Towards the end, the paper offers an iterative discussion of learnings from implementing such an approach

Evaluating the success of e-learning systems : the case of Moodle LMS at the University of Warwick

E-learning is a direct result of the integration of education and technology, and is increasingly considered as a powerful medium for learning. The undeniable significance of e-learning in education has led to a large growth of e-learning courses and systems offering different types of service. Thus, evaluation of e-learning systems is vital in ensuring successful delivery, effective use, and positive impact on learners. In recent studies, the vast majority of universities report having adopted varieties of e-learning systems and platforms to facilitate the students’ learning process. However, while adopting e-learning systems is useful, it is not an end in itself. In reviewing the literature, studies have revealed many problems with these systems, such as meeting users’ requirements and the suitability of these systems for targeted users. In order to improve the current systems to satisfy users’ needs, it is important to understand the different aspects that influence the quality and success of these systems. Hence, a new model for evaluating the success of e-learning systems is introduced in this research. Based on an intensive review of the literature, four approaches were identified and analysed as a theoretical basis for the research: DeLone and McLean’s information systems success model; the Technology Acceptance Model; the User Satisfaction Models; and the E-learning Quality Models. In order to provide a general comprehensive definition of e-learning success measurements, the four approaches found in the literature were considered in developing our model. The proposed model includes eleven constructs: technical system quality; information quality; service quality; educational system quality; support system quality; learner quality; instructor quality; perceived satisfaction; perceived usefulness; system use; and benefits. The model is comprehensive, and not based on the number of constructs, but on the intention to provide a holistic picture and different levels of success related to a broad range of success determinants, rather than focusing on a specific construct. As such, it forms an original contribution to knowledge. To test the model, an empirical study was conducted. First, an instrument was designed to assess the perceptions of students towards e-learning system success. Second, an expert study with 30 e-learning experts was carried out to confirm the measurements and indicators. The model was then tested in the context of the University of Warwick by fitting the model to data collected from 563 students engaged with an e-learning system. Both quantitative and qualitative data were analysed. The results confirm that the model proposed in this study is valid and reliable. Thus, the study contributes to the growing body of knowledge with a valid and reliable model and an instrument to evaluate e-learning systems success (EESS model). Further, the study sheds light on important issues and recommendations that should be taken into consideration to improve the perceptions of satisfaction, usefulness, use, and benefits of the e-learning systems. The study further provides practitioners with several practical contributions

Real-world connections to sustainability: Using authentic learning activities to introduce students to systems thinking through green chemistry

Systems thinking refers to approaches to learning that emphasise the interdependence of components in dynamic systems and how they interact and influence one another (Mahaffy et al., 2019). Applying systems thinking to green chemistry teaching and learning can create a molecular basis for sustainability (Mahaffy et al., 2019) that is able to enhance undergraduate chemistry students’ multidimensional understanding of complex sustainability challenges (Smith, 2011). However, efforts to introduce sustainable systems thinking – specifically within first-year introductory chemistry courses – are particularly challenging, and past approaches have produced mixed success (Mahaffy et al., 2019; An et al., 2021). Consequently, this indicates an opportune space within undergraduate chemistry education research to explore alternative and multidisciplinary approaches towards teaching green chemistry and sustainability (Wissinger et al., 2021). In this research, we present the preliminary results of a trimester-long intervention using authentic learning activities to introduce first-year chemistry students to systems thinking, through the application of green chemistry concepts. To determine the effectiveness of the intervention, we are using a mixed-methods research design to assess the impact of the learning activities on students’ development of systems thinking skills. Student motivations and attitudes towards the subject of chemistry will also be evaluated via validated survey instruments (Guay et al., 2000; Liu et al., 2017). The learning activities have been designed and developed successfully, though the delivery of the intervention is currently ongoing. Preliminary results indicate that students are excited to learn about how chemistry can be more sustainable, and that they are engaging with the learning activities. The aim of this research is to provide rigorous evidence for using systems thinking as a tool to teach students about green chemistry, ‘future-proofing’ chemistry in a way that is relevant, meaningful, and authentic for today’s chemistry students. Outcomes from our data analysis will help inform the development of new undergraduate chemistry education curricula that align with contemporary sustainable challenges. REFERENCES An, J., Loppnow, G.R., & Holme, T. A. (2021). Measuring the impact of incorporating systems thinking into general chemistry on affective components of student learning. Canadian Journal of Chemistry, 99(8), 698–705. Fisher, M.A. (2019). Systems thinking and educating the heads, hands, and hearts of chemistry majors. Journal of Chemical Education, 96(12), 2715–2719. Guay, F., Vallerand, R. J., & Blanchard, C. (2000). On the assessment of situational intrinsic and extrinsic motivation: The Situational Motivation Scale (SIMS). Motivation and emotion, 24(3), 175–213. Liu, Y., Ferrell, B., Barbera, J., & Lewis, J. E. (2017). Development and evaluation of a chemistry-specific version of the academic motivation scale (AMS-Chemistry). Chemistry Education Research and Practice, 18(1), 191–213. Mahaffy, P. G., Matlin, S. A., Holme, T. A., & MacKellar, J. (2019). Systems thinking for education about the molecular basis of sustainability. Nature Sustainability, 2(5), 362–370. Smith, T. (2011). Using critical systems thinking to foster an integrated approach to sustainability: A proposal for development practitioners. Environment, development and sustainability, 13, 1–17. Wissinger, J. E., Visa, A., Saha, B. B., Matlin, S. A., Mahaffy, P. G., Kümmerer, K., & Cornell, S. (2021). Integrating sustainability into learning in chemistry. Journal of Chemical Education, 98(4), 1061–1063

The Automatic Assessment of Multiple Artefacts: An Investigation into Design Diagrams and Their Implementations

As the Higher Education sector has moved towards student-centred learning so too has the growth in electronic support for learning. E-assessment has been a part of this growth as increasingly assessment and its feedback is seen as an integral part of the students’ learning process. Mature e-assessment systems exist, particularly where answers to questions are restricted to a prescribed list of alternatives. However, for free response artefacts, where there is a limited restriction placed on answers to questions, automated assessment systems are embryonic. This dissertation presents an investigation into the automated assessment of free response artefacts. Design diagrams and their accompanying source code implementations are examples of free response artefacts. A case study is developed that investigates how to automatically generate formative feedback for a design diagram by utilizing its accompanying implementation. The dissertation presents a two-staged solution, initially analysing the design diagram in isolation before comparing it with the implementation. A framework for this approach has been developed and tested using a tool applied to coursework submitted by undergraduate computer science students. The tool was evaluated by comparing the formative feedback comments generated by the tool with those produced by a team of computer science educators. Evaluation was undertaken via two Likert questionnaires, one completed by students and one completed by a team of computer scientists. The results presented are favourable, with the majority of comments produced by the tool being seen to be as least as good as those generated by the computer science educators

Identification and prioritisation of variables influencing the cost of learning content development

A dominant factor in cost analyses of e-Learning programs at tertiary education institutions is the measurement of direct and indirect costs associated with digital learning content, or in brief, learning content. In broad terms, over 60% of total e-Learning costs are related to design, development, publication and evaluation of learning content. The inclusion of new and emerging concepts and technologies including Learning Objects (LOs) and Learning Content Management Systems (LCMSs) into e-Learning programs at universities has opened up new opportunities and increased the complexity of learning content cost analyses. This thesis constructs a model that offers a minimized cost for the design, development and publication of learning content in a typical e-Learning program. Steps towards building the target model include the investigation of factors that affect learning content development, identification and prioritization of variables influencing the development cost, discussion of the relationships between identified variables and the process of learning content development, and analyses of two extreme cost structures. Finally, by assigning relevant variables in the available cost structure, a cost-effective model that covers the three main processes for design, development and publication of learning content is constructed. The cost-effective model introduced in this thesis covers not only the main subject notes but also content materials that are widely used in e-Learning programs. The proposed model takes into account subject notes and other materials which include quizzes, tutorial questions, critical thinking tasks and assignments as Learning Objects. In addition, based on this model, an experimental platform is designed and implemented to support the practical aspects of the proposed model noting its limitations and constraints. The experimental platform allows for the implementation of a simple case study for verification of development strategies adopted in the proposed cost-effective model. The reusability of Learning Objects used in the case study, and standards compliance of the resultant learning packages with different e-Learning platforms are also tested and documented. As an efficient and effective method of enhancing and facilitating students’ learning, e-Learning has obtained a wider acceptance among higher education institutions. The concept of Learning Objects and underlying models and technologies, including the proposed cost model, represent a cost-effective approach for accelerating the design, development and publication of learning content that can be highlighted in the development of e-Learning programs worldwide

An extensive analysis of efficient bug prediction configurations

Background: Bug prediction helps developers steer maintenance activities towards the buggy parts of a software. There are many design aspects to a bug predictor, each of which has several options, i.e., software metrics, machine learning model, and response variable. Aims: These design decisions should be judiciously made because an improper choice in any of them might lead to wrong, misleading, or even useless results. We argue that bug prediction con?gurations are intertwined and thus need to be evaluated in their entirety, in contrast to the common practice in the ?eld where each aspect is investigated in isolation. Method: We use a cost-aware evaluation scheme to evaluate 60 di?erent bug prediction con?guration combinations on ?ve open source Java projects. Results:We ?nd out that the best choices for building a cost-e?ective bug predictor are change metrics mixed with source code metrics as independent variables, Random Forest as the machine learning model, and the number of bugs as the response variable. Combining these con?guration options results in the most e?cient bug predictor across all subject systems. Conclusions: We demonstrate a strong evidence for the interplay among bug prediction con?gurations and provide concrete guidelines for researchers and practitioners on how to build and evaluate e?cient bug predictors

A hybrid e-learning framework: Process-based, semantically-enriched and service-oriented

Despite the recent innovations in e-Learning, much development is needed to ensure better learning experience for everyone and bridge the research gap in the current state of the art e-Learning artefacts. Contemporary e-learning artefacts possess various limitations as follows. First, they offer inadequate variations of adaptivity, since their recommendations are limited to e-learning resources, peers or communities. Second, they are often overwhelmed with technology at the expense of proper pedagogy and learning theories underpinning e-learning practices. Third, they do not comprehensively capture the e-learning experiences as their focus shifts to e-learning activities instead of e-learning processes. In reality, learning is a complex process that includes various activities and interactions between different roles to achieve certain gaols in a continuously evolving environment. Fourth, they tend more towards legacy systems and lack the agility and flexibility in their structure and design. To respond to the above limitations, this research aims at investigating the effectiveness of combining three advanced technologies (i.e., Business Process Modelling and Enactment, Semantics and Service Oriented Computing – SOC–) with learning pedagogy in order to enhance the e-learner experience. The key design artefact of this research is the development of the HeLPS e-Learning Framework – Hybrid e-Learning Framework that is Process-based, Semantically-enriched and Service Oriented-enabled. In this framework, a generic e-learning process has been developed bottom-up based on surveying a wide range of e-learning models (i.e., practical artefacts) and their underpinning pedagogies/concepts (i.e., theories); and then forming a generic e-learning process. Furthermore, an e-Learning Meta-Model has been developed in order to capture the semantics of e-learning domain and its processes. Such processes have been formally modelled and dynamically enacted using a service-oriented enabled architecture. This framework has been evaluated using a concern-based evaluation employing both static and dynamic approaches. The HeLPS e-Learning Framework along with its components have been evaluated by applying a data-driven approach and artificially-constructed case study to check its effectiveness in capturing the semantics, enriching e-learning processes and deriving services that can enhance the e-learner experience. Results revealed the effectiveness of combining the above-mentioned technologies in order to enhance the e-learner experience. Also, further research directions have been suggested.This research contributes to enhancing the e-learner experience by making the e-learning artefacts driven by pedagogy and informed by the latest technologies. One major novel contribution of this research is the introduction of a layered architectural framework (i.e., HeLPS) that combines business process modelling and enactment, semantics and SOC together. Another novel contribution is adopting the process-based approach in e-learning domain through: identifying these processes and developing a generic business process model from a set of related e-learning business process models that have the same goals and associated objectives. A third key contribution is the development of the e-Learning Meta-Model, which captures a high-abstract view of learning domain and encapsulates various domain rules using the Semantic Web Rule Language. Additional contribution is promoting the utilisation of Service-Orientation in e-learning through developing a semantically-enriched approach to identify and discover web services from e-learning business process models. Fifth, e-Learner Experience Model (eLEM) and e-Learning Capability Maturity Model (eLCMM) have been developed, where the former aims at identifying and quantifying the e-learner experience and the latter represents a well-defined evolutionary plateau towards achieving a mature e-learning process from a technological perspective. Both models have been combined with a new developed data-driven Validation and Verification Model to develop a Concern-based Evaluation Approach for e-Learning artefacts, which is considered as another contribution