Analysing Qualitative Data Using Facial Expressions in an Educational Scenario

In communication, both verbal and non-verbal means ensure that a message is conveyed, and facial expressions are acknowledged as one of the most influential factors in non-verbal communication. Facial Analysis Coding System (FACS) is a tool to analyse data other than the spoken language to improve a researcher's reading of an interviewee's emotions, and proposes a methodology to support the annotation process of facial expressions in a piece of communication. This study investigates an applied framework for FACS in an educational scenario. The study combines both the computerised and manual entries in the applied method. The study addresses the challenges, findings and recommendations of this applied method

Annotation of multimedia learning materials for semantic search

Multimedia is the main source for online learning materials, such as videos, slides and textbooks, and its size is growing with the popularity of online programs offered by Universities and Massive Open Online Courses (MOOCs). The increasing amount of multimedia learning resources available online makes it very challenging to browse through the materials or find where a specific concept of interest is covered. To enable semantic search on the lecture materials, their content must be annotated and indexed. Manual annotation of learning materials such as videos is tedious and cannot be envisioned for the growing quantity of online materials. One of the most commonly used methods for learning video annotation is to index the video, based on the transcript obtained from translating the audio track of the video into text. Existing speech to text translators require extensive training especially for non-native English speakers and are known to have low accuracy. This dissertation proposes to index the slides, based on the keywords. The keywords extracted from the textbook index and the presentation slides are the basis of the indexing scheme. Two types of lecture videos are generally used (i.e., classroom recording using a regular camera or slide presentation screen captures using specific software) and their quality varies widely. The screen capture videos, have generally a good quality and sometimes come with metadata. But often, metadata is not reliable and hence image processing techniques are used to segment the videos. Since the learning videos have a static background of slide, it is challenging to detect the shot boundaries. Comparative analysis of the state of the art techniques to determine best feature descriptors suitable for detecting transitions in a learning video is presented in this dissertation. The videos are indexed with keywords obtained from slides and a correspondence is established by segmenting the video temporally using feature descriptors to match and align the video segments with the presentation slides converted into images. The classroom recordings using regular video cameras often have poor illumination with objects partially or totally occluded. For such videos, slide localization techniques based on segmentation and heuristics is presented to improve the accuracy of the transition detection. A region prioritized ranking mechanism is proposed that integrates the keyword location in the presentation into the ranking of the slides when searching for a slide that covers a given keyword. This helps in getting the most relevant results first. With the increasing size of course materials gathered online, a user looking to understand a given concept can get overwhelmed. The standard way of learning and the concept of “one size fits all” is no longer the best way to learn for millennials. Personalized concept recommendation is presented according to the user’s background knowledge. Finally, the contributions of this dissertation have been integrated into the Ultimate Course Search (UCS), a tool for an effective search of course materials. UCS integrates presentation, lecture videos and textbook content into a single platform with topic based search capabilities and easy navigation of lecture materials

Smart labs and social practice: social tools for pervasive laboratory workspaces: a position paper

The emergence of pervasive and ubiquitous computing stimulates a view of future work environments where sharing of information, data and knowledge is easy and commonplace, particularly in highly interactive settings. Much of the work in this area focuses on tool development to support activities such as data collection, data recording and sharing, and so on. We are interested in this kind of technical development, which is both challenging and essential for science communities. But we are also interested in a broader interpretation of knowledge sharing and the human/social side of tools we develop to support this. We are keen to know more about how groups of different kinds of scientists can make their work understandable and shareable with each other in a multidisciplinary setting. This is a complex task because boundaries and barriers can emerge between disciplines engendered by differences in discourses and practices, which may not easily translate into other discipline areas. In the worst case, there may be some hostility between disciplines, or at least doubt and scepticism. Nevertheless, sharing approaches to research, research expertise, data and methods across disciplines can be a very fruitful exercise, and encouragement to engage in this activity is particularly pertinent in the digital era. Issues of privacy and security are also key aspects – knowing when and how to release data or information to other groups is crucial to providing a safe environment for people to work, and there are several sensitivities to be explored here. In this paper we describe an evolving situation that captures many of these issues, which we aim to track longitudinally

Cognitive emotions in e-learning processes and their potential relationship with students’ academic adjustment

In times of growing importance and emphasis on improving academic outcomes for young people, their academic selves/lives are increasingly becoming more central to their understanding of their own wellbeing. How they experience and perceive their academic successes or failures, can influence their perceived self-efficacy and eventual academic achievement. To this end, ‘cognitive emotions’, elicited to acquire or develop new skills/knowledges, can play a crucial role as they indicate the state or the “flow” of a student’s emotions, when facing challenging tasks. Within innovative teaching models, measuring the affective components of learning have been mainly based on self-reports and scales which have neglected the real-time detection of emotions, through for example, recording or measuring facial expressions. The aim of the present study is to test the reliability of an ad hoc software trained to detect and classify cognitive emotions from facial expressions across two different environments, namely a video-lecture and a chat with teacher, and to explore cognitive emotions in relation to academic e-selfefficacy and academic adjustment. To pursue these goals, we used video-recordings of ten psychology students from an online university engaging in online learning tasks, and employed software to automatically detect eleven cognitive emotions. Preliminary results support and extend prior studies, illustrating how exploring cognitive emotions in real time can inform the development and success of academic e-learning interventions aimed at monitoring and promoting students’ wellbeing.peer-reviewe

A Survey of Smart Classroom Literature

Recently, there has been a substantial amount of research on smart classrooms, encompassing a number of areas, including Information and Communication Technology, Machine Learning, Sensor Networks, Cloud Computing, and Hardware. Smart classroom research has been quickly implemented to enhance education systems, resulting in higher engagement and empowerment of students, educators, and administrators. Despite decades of using emerging technology to improve teaching practices, critics often point out that methods miss adequate theoretical and technical foundations. As a result, there have been a number of conflicting reviews on different perspectives of smart classrooms. For a realistic smart classroom approach, a piecemeal implementation is insufficient. This survey contributes to the current literature by presenting a comprehensive analysis of various disciplines using a standard terminology and taxonomy. This multi-field study reveals new research possibilities and problems that must be tackled in order to integrate interdisciplinary works in a synergic manner. Our analysis shows that smart classroom is a rapidly developing research area that complements a number of emerging technologies. Moreover, this paper also describes the co-occurrence network of technological keywords using VOSviewer for an in-depth analysis

Artificial Intelligence methodologies to early predict student outcome and enrich learning material

L'abstract è presente nell'allegato / the abstract is in the attachmen

Novel Datasets, User Interfaces and Learner Models to Improve Learner Engagement Prediction on Educational Videos

With the emergence of Open Education Resources (OERs), educational content creation has rapidly scaled up, making a large collection of new materials made available. Among these, we find educational videos, the most popular modality for transferring knowledge in the technology-enhanced learning paradigm. Rapid creation of learning resources opens up opportunities in facilitating sustainable education, as the potential to personalise and recommend specific materials that align with individual users’ interests, goals, knowledge level, language and stylistic preferences increases. However, the quality and topical coverage of these materials could vary significantly, posing significant challenges in managing this large collection, including the risk of negative user experience and engagement with these materials. The scarcity of support resources such as public datasets is another challenge that slows down the development of tools in this research area. This thesis develops a set of novel tools that improve the recommendation of educational videos. Two novel datasets and an e-learning platform with a novel user interface are developed to support the offline and online testing of recommendation models for educational videos. Furthermore, a set of learner models that accounts for the learner interests, knowledge, novelty and popularity of content is developed through this thesis. The different models are integrated together to propose a novel learner model that accounts for the different factors simultaneously. The user studies conducted on the novel user interface show that the new interface encourages users to explore the topical content more rigorously before making relevance judgements about educational videos. Offline experiments on the newly constructed datasets show that the newly proposed learner models outperform their relevant baselines significantly

Video Augmentation in Education: in-context support for learners through prerequisite graphs

The field of education is experiencing a massive digitisation process that has been ongoing for the past decade. The role played by distance learning and Video-Based Learning, which is even more reinforced by the pandemic crisis, has become an established reality. However, the typical features of video consumption, such as sequential viewing and viewing time proportional to duration, often lead to sub-optimal conditions for the use of video lessons in the process of acquisition, retrieval and consolidation of learning contents. Video augmentation can prove to be an effective support to learners, allowing a more flexible exploration of contents, a better understanding of concepts and relationships between concepts and an optimization of time required for video consumption at different stages of the learning process. This thesis focuses therefore on the study of methods for: 1) enhancing video capabilities through video augmentation features; 2) extracting concept and relationships from video materials; 3) developing intelligent user interfaces based on the knowledge extracted. The main research goal is to understand to what extent video augmentation can improve the learning experience. This research goal inspired the design of EDURELL Framework, within which two applications were developed to enable the testing of augmented methods and their provision. The novelty of this work lies in using the knowledge within the video, without exploiting external materials, to exploit its educational potential. The enhancement of the user interface takes place through various support features among which in particular a map that progressively highlights the prerequisite relationships between the concepts as they are explained, i.e., following the advancement of the video. The proposed approach has been designed following a user-centered iterative approach and the results in terms of effect and impact on video comprehension and learning experience make a contribution to the research in this field

Correlating Visual Speaker Gestures with Measures of Audience Engagement to Aid Video Browsing

In this thesis, we argue that in the domains of educational lectures and political debates, speaker gestures can be a source of semantic cues for video browsing. We hypothesize that certain human gestures, which can be automatically identified through techniques of computer vision, can convey significant information that are correlated to audience engagement. We present a joint-angle descriptor derived from an automatic upper body pose estimation framework to train an SVM which identifies point and spread poses in extracted video frames of an instructor giving a lecture. Ground-truth is collected in the form of 2500 manually annotated frames covering 20 minutes of a video lecture. Cross validation on the ground-truth data showed classifier F-scores of 0.54 and 0.39 for point and spread poses, respectively. We also derive an attribute for gestures which measures the angular variance of the arm movements from this system (analogous to arm waving). We present a method for tracking hands which succeeds even when left and right hands are clasping and occluding each other. We evaluate on a ground-truth dataset of 698 images with 1301 annotated left and right hands, mostly clasped. Our method performs better than baseline on recall (0.66 vs. 0.53) without sacrificing precision (0.65 for both) toward the goal of recognizing clasped hands. For tracking, it results in an improvement over a baseline method with an F-score of 0.59 vs. 0.48. From this, we are able to derive hand motion-based gesture attributes such as velocity, direction change and extremal pose. In ground-truth studies, we manually annotate and analyze the gestures of two instructors, each in a 75-minute computer science lecture using a 14-bit pose vector. We observe "pedagogical" gestures of punctuation and encouragement in addition to traditional classes of gestures such as deictic and metaphoric. We also introduce a tool to facilitate the manual annotations of gestures in video and present results on their frequencies and co-occurrences. In particular, we find that 5 poses represent 80% of the variation in the annotated ground truth. We demonstrate a correlation between the angular variance of arm movements and the presence of those conjunctions that are used to contrast connected clauses ("but", "neither", etc.) in the accompanying speech. We do this by training an AdaBoost-based binary classifier using decision trees as weak learners. On a ground-truth database of 4243 video clips totaling 3.83 hours, each with subtitles, training on sets of conjunctions indicating contrast produces classifiers capable of achieving 55% accuracy on a balanced test set. We study two different presentation methods: an attribute graph which shows a normalized measure of the visual attributes across an entire video, as well as emphasized subtitles, where individual words are emphasized (resized) based on their accompanying gestures. Results from 12 subjects show supportive ratings given for the browsing aids in the task of providing keywords for video under time constraints. Subjects' keywords are also compared to independent ground-truth, resulting in precisions from 0.50-0.55, even when given less than half real time to view the video. We demonstrate a correlation between gesture attributes and a rigorous method of measuring audience engagement: electroencephalography (EEG). Our 20 subjects watch 61 minutes of video of the 2012 U.S. Presidential Debates while under observation through EEG. After discarding corrupted recordings, we retain 47 minutes worth of EEG data for each subject. The subjects are examined in aggregate and in subgroups according to gender and political affiliation. We find statistically significant correlations between gesture attributes (particularly extremal pose) and our feature of engagement derived from EEG. For all subjects watching all videos, we see a statistically significant correlation between gesture and engagement with a Spearman rank correlation of rho = 0.098 with p < 0.05, Bonferroni corrected. For some stratifications, correlations reach as high as rho = 0.297. From these results, we conclude what gestures can be used to measure engagement