A Survey on Data-Driven Evaluation of Competencies and Capabilities Across Multimedia Environments

The rapid evolution of technology directly impacts the skills and jobs needed in the next decade. Users can, intentionally or unintentionally, develop different skills by creating, interacting with, and consuming the content from online environments and portals where informal learning can emerge. These environments generate large amounts of data; therefore, big data can have a significant impact on education. Moreover, the educational landscape has been shifting from a focus on contents to a focus on competencies and capabilities that will prepare our society for an unknown future during the 21st century. Therefore, the main goal of this literature survey is to examine diverse technology-mediated environments that can generate rich data sets through the users’ interaction and where data can be used to explicitly or implicitly perform a data-driven evaluation of different competencies and capabilities. We thoroughly and comprehensively surveyed the state of the art to identify and analyse digital environments, the data they are producing and the capabilities they can measure and/or develop. Our survey revealed four key multimedia environments that include sites for content sharing & consumption, video games, online learning and social networks that fulfilled our goal. Moreover, different methods were used to measure a large array of diverse capabilities such as expertise, language proficiency and soft skills. Our results prove the potential of the data from diverse digital environments to support the development of lifelong and lifewide 21st-century capabilities for the future society

Predicting Paid Certification in Massive Open Online Courses

Massive open online courses (MOOCs) have been proliferating because of the free or low-cost offering of content for learners, attracting the attention of many stakeholders across the entire educational landscape. Since 2012, coined as “the Year of the MOOCs”, several platforms have gathered millions of learners in just a decade. Nevertheless, the certification rate of both free and paid courses has been low, and only about 4.5–13% and 1–3%, respectively, of the total number of enrolled learners obtain a certificate at the end of their courses. Still, most research concentrates on completion, ignoring the certification problem, and especially its financial aspects. Thus, the research described in the present thesis aimed to investigate paid certification in MOOCs, for the first time, in a comprehensive way, and as early as the first week of the course, by exploring its various levels. First, the latent correlation between learner activities and their paid certification decisions was examined by (1) statistically comparing the activities of non-paying learners with course purchasers and (2) predicting paid certification using different machine learning (ML) techniques. Our temporal (weekly) analysis showed statistical significance at various levels when comparing the activities of non-paying learners with those of the certificate purchasers across the five courses analysed. Furthermore, we used the learner’s activities (number of step accesses, attempts, correct and wrong answers, and time spent on learning steps) to build our paid certification predictor, which achieved promising balanced accuracies (BAs), ranging from 0.77 to 0.95. Having employed simple predictions based on a few clickstream variables, we then analysed more in-depth what other information can be extracted from MOOC interaction (namely discussion forums) for paid certification prediction. However, to better explore the learners’ discussion forums, we built, as an original contribution, MOOCSent, a cross- platform review-based sentiment classifier, using over 1.2 million MOOC sentiment-labelled reviews. MOOCSent addresses various limitations of the current sentiment classifiers including (1) using one single source of data (previous literature on sentiment classification in MOOCs was based on single platforms only, and hence less generalisable, with relatively low number of instances compared to our obtained dataset;) (2) lower model outputs, where most of the current models are based on 2-polar iii iv classifier (positive or negative only); (3) disregarding important sentiment indicators, such as emojis and emoticons, during text embedding; and (4) reporting average performance metrics only, preventing the evaluation of model performance at the level of class (sentiment). Finally, and with the help of MOOCSent, we used the learners’ discussion forums to predict paid certification after annotating learners’ comments and replies with the sentiment using MOOCSent. This multi-input model contains raw data (learner textual inputs), sentiment classification generated by MOOCSent, computed features (number of likes received for each textual input), and several features extracted from the texts (character counts, word counts, and part of speech (POS) tags for each textual instance). This experiment adopted various deep predictive approaches – specifically that allow multi-input architecture - to early (i.e., weekly) investigate if data obtained from MOOC learners’ interaction in discussion forums can predict learners’ purchase decisions (certification). Considering the staggeringly low rate of paid certification in MOOCs, this present thesis contributes to the knowledge and field of MOOC learner analytics with predicting paid certification, for the first time, at such a comprehensive (with data from over 200 thousand learners from 5 different discipline courses), actionable (analysing learners decision from the first week of the course) and longitudinal (with 23 runs from 2013 to 2017) scale. The present thesis contributes with (1) investigating various conventional and deep ML approaches for predicting paid certification in MOOCs using learner clickstreams (Chapter 5) and course discussion forums (Chapter 7), (2) building the largest MOOC sentiment classifier (MOOCSent) based on learners’ reviews of the courses from the leading MOOC platforms, namely Coursera, FutureLearn and Udemy, and handles emojis and emoticons using dedicated lexicons that contain over three thousand corresponding explanatory words/phrases, (3) proposing and developing, for the first time, multi-input model for predicting certification based on the data from discussion forums which synchronously processes the textual (comments and replies) and numerical (number of likes posted and received, sentiments) data from the forums, adapting the suitable classifier for each type of data as explained in detail in Chapter 7

Delving into instructor‐led feedback interventions informed by learning analytics in massive open online courses

Producción CientíficaBackground:Providing feedback in massive open online courses (MOOCs) is chal-lenging due to the massiveness and heterogeneity of learners' population. Learninganalytics (LA) solutions aim at scaling up feedback interventions and supportinginstructors in this endeavour.Paper Objectives:This paper focuses on instructor-led feedback mediated by LAtools in MOOCs. Our goal is to answer how, to what extent data-driven feedback isprovided to learners, and what its impact is.Methods:We conducted a systematic literature review on the state-of-the-art LA-informed instructor-led feedback in MOOCs. From a pool of 227 publications, weselected 38 articles that address the topic of LA-informed feedback in MOOCs medi-ated by instructors. We applied etic content analysis to the collected data.Results and Conclusions:The results revealed a lack of empirical studies exploring LA todeliver feedback, and limited attention on pedagogy to inform feedback practices. Our find-ings suggest the need for systematization and evaluation of feedback. Additionally, there isa need for conceptual tools to guide instructors' in the design of LA-based feedback.Takeaways:We point out the need for systematization and evaluation of feedback. Weenvision that this research can support the design of LA-based feedback, thus contribut-ing to bridge the gap between pedagogy and data-driven practice in MOOCs.Consejo de Investigación de Estonia (PSG286)Ministerio de Ciencia e Innovación - Fondo Europeo de Desarrollo Regional y la Agencia Nacional de Investigación (grant PID2020-112584RB-C32) and (grant TIN2017-85179-C3-2-R)Junta de Castilla y León - Fondo Social Europeo y el Consejo Regional de Educación (grant E-47-2018-0108488

Digitalisation, Artificial Intelligence and Vocational Occupations and Skills: What are the needs for training Teachers and Trainers?

The paper seeks to explore the impact AI and automation have on vocational occupations and skills and to examine what that means for teachers and trainers in VET. It looks at how AI can be used to shape learning and teaching processes, through for example, digital assistants which support teachers. It also focuses on the transformative power of AI that promises profound changes in employment and work tasks. The paper is based on research being undertaken through the EU Erasmus+ Taccle AI project. It presents the results of an extensive literature review and of interviews with VET managers, teachers and AI experts in five countries. It asks whether machines will complement or replace humans in the workplace before going to look at developments in using AI for teaching and learning in VET. Finally, it proposes extensions to the EU DigiCompEdu Framework for training teachers and trainers in using technology

Practical and Ethical Challenges of Large Language Models in Education: A Systematic Scoping Review

Educational technology innovations leveraging large language models (LLMs) have shown the potential to automate the laborious process of generating and analysing textual content. While various innovations have been developed to automate a range of educational tasks (e.g., question generation, feedback provision, and essay grading), there are concerns regarding the practicality and ethicality of these innovations. Such concerns may hinder future research and the adoption of LLMs-based innovations in authentic educational contexts. To address this, we conducted a systematic scoping review of 118 peer-reviewed papers published since 2017 to pinpoint the current state of research on using LLMs to automate and support educational tasks. The findings revealed 53 use cases for LLMs in automating education tasks, categorised into nine main categories: profiling/labelling, detection, grading, teaching support, prediction, knowledge representation, feedback, content generation, and recommendation. Additionally, we also identified several practical and ethical challenges, including low technological readiness, lack of replicability and transparency, and insufficient privacy and beneficence considerations. The findings were summarised into three recommendations for future studies, including updating existing innovations with state-of-the-art models (e.g., GPT-3/4), embracing the initiative of open-sourcing models/systems, and adopting a human-centred approach throughout the developmental process. As the intersection of AI and education is continuously evolving, the findings of this study can serve as an essential reference point for researchers, allowing them to leverage the strengths, learn from the limitations, and uncover potential research opportunities enabled by ChatGPT and other generative AI models

Knowledge Communities in Online Education and (Visual) Knowledge Management: 19. Workshop GeNeMe‘16 as part of IFKAD 2016: Proceedings of 19th Conference GeNeMe

Communities in New Media started in 1998 as a workshop series at TU Dresden, and since then has annually dealt with online communities at the interface between several disciplines such as education and economics, computer science, social and communication sciences, and more. (See Köhler, Kahnwald & Schoop, 2015). The workshop is traditionally a forum for interdisciplinary dialogue between science and business and serves to share experiences and knowledge among participants from different disciplines, organisations, and institutions. In addition to the core themes of knowledge management and communities (in the chapters of the same name), the main focus of the conference is also on the support of knowledge and learning processes in the field of (media-assisted) higher education. This is complemented by an informational perspective when it comes to more functional and methodological approaches - use cases, workflows, and automation in knowledge management. In addition, systems and approaches for feedback, exchange, and ideas are presented. With the focus of knowledge media design and visual research as well as creative processes, this time there is also a highlight on visual aspects of knowledge management and mediation. For IFKAD 2016, three GeNeMe tracks were accepted which focus on the interface of knowledge communities and knowledge management as well as knowledge media design in science, business, or education. In this conference volume you will find detailed information about these three tracks: -- Knowledge Communities I: Knowledge Management -- Knowledge Communities II: Online Education -- Visual Knowledge Management [From the Preface.]:Preface IX Vorwort XIII Knowledge Communities I: Knowledge Management 1 Process Learning Environments 1 Two Steps to IT Transparency: A Practitioner’s Approach for a Knowledge Based Analysis of Existing IT Landscapes in SME 13 Social Media and Sustainable Communication. Rethinking the Role of Research and Innovation Networks 26 Consolidating eLearning in a Higher Education Institution: An Organisational Issue integrating Didactics, Technology, and People by the Means of an eLearning Strategy 39 How to treat the troll? An empirical analysis of counterproductive online behavior, personality traits and organizational behavior 51 Knowledge Communities II: Online Education 64 Sifa-Portfolio – a Continuing Education Concept for Specialists on Industrial Safety Combining Formal and Informal Learning 64 Analysing eCollaboration: Prioritisation of Monitoring Criteria for Learning Analytics in the Virtual Classroom 78 Gamifying Higher Education. Beyond Badges, Points and Leaderboards 93 Virtual International Learning Experience in Formal Higher Education – A Case Study from Jordan 105 Migration to the Flipped Classroom – Applying a Scalable Flipped Classroom Arrangement 117 MOOC@TU9 – Common MOOC Strategy of the Alliance of Nine Leading German Institutes of Technology 131 A Survey on Knowledge Management in Universities in the QS Rankings: E-learning and MOOCs 144 Visual Knowledge Media 157 Generating implications for design in practice: How different stimuli are retrieved and transformed to generate ideas 157 Behind the data – preservation of the knowledge in CH Visualisations 170 Building a Wiki resource on digital 3D reconstruction related knowledge assets 184 Visual media as a tool to acquire soft skills — cross-disciplinary teaching-learning project SUFUvet 196 Graphing Meeting Records - An Approach to Visualize Information in a Multi Meeting Context 209 HistStadt4D – A four dimensional access to history 221 Ideagrams: A digital tool for observing ideation processes 234 Adress- und Autorenverzeichnis 251Gemeinschaften in Neuen Medien hat 1998 als Workshop-Reihe an der TU Dresden begonnen und seither jährlich das Thema Online-Communities an der Schnittstelle mehrerer Disziplinen wie Informatik, Bildungs- und Wirtschaftswissenschaften, Informatik sowie Sozial-und Kommunikationswissenschaft u.a.m. thematisiert (vgl. Köhler, Kahnwald & Schoop, 2015). Der Workshop ist traditionell ein Forum für den interdisziplinären Dialog zwischen Wissenschaft und Wirtschaft und dient dazu, Erfahrungen und Wissen unter den Teilnehmern aus verschiedenen Disziplinen, Organisationen und Institutionen zu teilen. Die inhaltlichen Schwerpunkte der Konferenz widmen sich neben den Kernthemen Wissensmanagement und Communities (in den gleichnamigen Kapiteln) auch der Unterstützung von Wissens- und Lernprozessen im Bereich der (mediengestützten) Hochschullehre. Ergänzt wird diese eher organisationswissenschaftliche durch eine informatorische Perspektive, wenn es um stärker funktionale bzw. auch methodische Ansätze geht – Use Cases, Workflows und Automatisierung im Wissensmanagement. Darüber hinaus werden Systeme und Ansätze für Feedback, Austausch und Ideenfindung vorgestellt. Mit den Schwerpunkten der Wissensmediengestaltung und visuellen Forschungs- sowie Kreativprozessen wird diesmal auch ein Schlaglicht auf visuelle Aspekte von Wissensmanagement und -vermittlung geworfen. Für die IFKAD 2016 wurden drei GeNeMe-Tracks angenommen, die sich auf das Interface von Wissensgemeinschaften und Wissensmanagement sowie die Wissensmediengestaltung in Wissenschaft, Wirtschaft oder Bildung konzentrieren. Im vorliegenden Tagungsband finden Sie detaillierte Informationen zu diesen drei Tracks: -- Knowledge Communities I: Knowledge Management -- Knowledge Communities II: Online Education -- Visual Knowledge Management [Aus dem Vorwort.]:Preface IX Vorwort XIII Knowledge Communities I: Knowledge Management 1 Process Learning Environments 1 Two Steps to IT Transparency: A Practitioner’s Approach for a Knowledge Based Analysis of Existing IT Landscapes in SME 13 Social Media and Sustainable Communication. Rethinking the Role of Research and Innovation Networks 26 Consolidating eLearning in a Higher Education Institution: An Organisational Issue integrating Didactics, Technology, and People by the Means of an eLearning Strategy 39 How to treat the troll? An empirical analysis of counterproductive online behavior, personality traits and organizational behavior 51 Knowledge Communities II: Online Education 64 Sifa-Portfolio – a Continuing Education Concept for Specialists on Industrial Safety Combining Formal and Informal Learning 64 Analysing eCollaboration: Prioritisation of Monitoring Criteria for Learning Analytics in the Virtual Classroom 78 Gamifying Higher Education. Beyond Badges, Points and Leaderboards 93 Virtual International Learning Experience in Formal Higher Education – A Case Study from Jordan 105 Migration to the Flipped Classroom – Applying a Scalable Flipped Classroom Arrangement 117 MOOC@TU9 – Common MOOC Strategy of the Alliance of Nine Leading German Institutes of Technology 131 A Survey on Knowledge Management in Universities in the QS Rankings: E-learning and MOOCs 144 Visual Knowledge Media 157 Generating implications for design in practice: How different stimuli are retrieved and transformed to generate ideas 157 Behind the data – preservation of the knowledge in CH Visualisations 170 Building a Wiki resource on digital 3D reconstruction related knowledge assets 184 Visual media as a tool to acquire soft skills — cross-disciplinary teaching-learning project SUFUvet 196 Graphing Meeting Records - An Approach to Visualize Information in a Multi Meeting Context 209 HistStadt4D – A four dimensional access to history 221 Ideagrams: A digital tool for observing ideation processes 234 Adress- und Autorenverzeichnis 25