Ranking Feature Sets for Emotion Models Used in Classroom Based Intelligent Tutoring Systems

Abstract. Recent progress has been made in using sensors with Intel-ligent Tutoring Systems in classrooms in order to predict the affective state of students users. If tutors were able to interpret sensor data with new students based on past experience, rather than having to be indi-vidually trained, then tutor developers could evaluate various methods of adapting to each student’s affective state using consistent predictions. Our classifiers for emotion have predicted student emotions with an accu-racy between 78 % and 87%. However, it is still unclear which sensors are best, and the educational technology community needs to know this to develop better than baseline classifiers, e.g. ones that use only frequency of emotional occurrence to predict affective state. This paper suggests a method for comparing classifiers using different sensors as well as a method for validating the classifiers on a novel population. This involves training our classifiers on data collected in the Fall of 2008 and testing them on data collected in the Spring of 2009. Results of the comparison show that the classifiers for some affective states are significantly better than the baseline, and a validation study found that not all classifier rankings generalize to new settings. The analysis suggests that though there is some benefit gained from simple linear classifiers, more advanced methods are needed for better results.

Individual and Inter-related Action Unit Detection in Videos for Affect Recognition

The human face has evolved to become the most important source of non-verbal information that conveys our affective, cognitive and mental state to others. Apart from human to human communication facial expressions have also become an indispensable component of human-machine interaction (HMI). Systems capable of understanding how users feel allow for a wide variety of applications in medical, learning, entertainment and marketing technologies in addition to advancements in neuroscience and psychology research and many others. The Facial Action Coding System (FACS) has been built to objectively define and quantify every possible facial movement through what is called Action Units (AU), each representing an individual facial action. In this thesis we focus on the automatic detection and exploitation of these AUs using novel appearance representation techniques as well as incorporation of the prior co-occurrence information between them. Our contributions can be grouped in three parts. In the first part, we propose to improve the detection accuracy of appearance features based on local binary patterns (LBP) for AU detection in videos. For this purpose, we propose two novel methodologies. The first one uses three fundamental image processing tools as a pre-processing step prior to the application of the LBP transform on the facial texture. These tools each enhance the descriptive ability of LBP by emphasizing different transient appearance characteristics, and are proven to increase the AU detection accuracy significantly in our experiments. The second one uses multiple local curvature Gabor binary patterns (LCGBP) for the same problem and achieves state-of-the-art performance on a dataset of mostly posed facial expressions. The curvature information of the face, as well as the proposed multiple filter size scheme is very effective in recognizing these individual facial actions. In the second part, we propose to take advantage of the co-occurrence relation between the AUs, that we can learn through training examples. We use this information in a multi-label discriminant Laplacian embedding (DLE) scheme to train our system with SIFT features extracted around the salient and transient landmarks on the face. The system is first validated on a challenging (containing lots of occlusions and head pose variations) dataset without the DLE, then we show the performance of the full system on the FERA 2015 challenge on AU occurence detection. The challenge consists of two difficult datasets that contain spontaneous facial actions at different intensities. We demonstrate that our proposed system achieves the best results on these datasets for detecting AUs. The third and last part of the thesis contains an application on how this automatic AU detection system can be used in real-life situations, particularly for detecting cognitive distraction. Our contribution in this part is two-fold: First, we present a novel visual database of people driving a simulator while being induced visual and cognitive distraction via secondary tasks. The subjects have been recorded using three near-infrared camera-lighting systems, which makes it a very suitable configuration to use in real driving conditions, i.e. with large head pose and ambient light variations. Secondly, we propose an original framework to automatically discriminate cognitive distraction sequences from baseline sequences by extracting features from continuous AU signals and by exploiting the cross-correlations between them. We achieve a very high classification accuracy in our subject-based experiments and a lower yet acceptable performance for the subject-independent tests. Based on these results we discuss how facial expressions related to this complex mental state are individual, rather than universal, and also how the proposed system can be used in a vehicle to help decrease human error in traffic accidents

Modeling Learner Mood In Realtime Through Biosensors For Intelligent Tutoring Improvements

Computer-based instructors, just like their human counterparts, should monitor the emotional and cognitive states of their students in order to adapt instructional technique. Doing so requires a model of student state to be available at run time, but this has historically been difficult. Because people are different, generalized models have not been able to be validated. As a person’s cognitive and affective state vary over time of day and seasonally, individualized models have had differing difficulties. The simultaneous creation and execution of an individualized model, in real time, represents the last option for modeling such cognitive and affective states. This dissertation presents and evaluates four differing techniques for the creation of cognitive and affective models that are created on-line and in real time for each individual user as alternatives to generalized models. Each of these techniques involves making predictions and modifications to the model in real time, addressing the real time datastream problems of infinite length, detection of new concepts, and responding to how concepts change over time. Additionally, with the knowledge that a user is physically present, this work investigates the contribution that the occasional direct user query can add to the overall quality of such models. The research described in this dissertation finds that the creation of a reasonable quality affective model is possible with an infinitesimal amount of time and without “ground truth” knowledge of the user, which is shown across three different emotional states. Creation of a cognitive model in the same fashion, however, was not possible via direct AI modeling, even with all of the “ground truth” information available, which is shown across four different cognitive states

Professional development for using technology in mathematics teaching in Ghana

Drawing on the framework of TPACK (technological pedagogical content knowledge) and the principles of effective mathematics pedagogy, this study set out with two aims. First, shifts in teachers’ technology dispositions (beliefs, attitudes, and knowledge) were explored after engagement in the professional development programme mediated with GeoGebra software. Second, typical features and nuances of the complexities of enacting effective mathematics pedagogy in a GeoGebra learning environment were examined. Eleven in-service mathematics teachers from a senior high school in Ghana were engaged in a professional development programme for 12 months. They were introduced to the use of GeoGebra software in mathematics teaching, and then designed GeoGebra-based mathematics lessons, which they taught to their peers and subsequently their students in the mathematics classroom. Self-report questionnaire, interviews, focus group discussions, lesson plans, and lesson observations were used for data collection. The results provided evidence that within Geogebra-based mathematics lessons, teachers were able to enact, to different degrees, five practices central to effective mathematics pedagogy: creating mathematical setting, providing useful mathematical tasks, orchestrating mathematical discussions, making mathematical connections, and assessing students’ learning. Further analysis of the data provided evidence for theorising 31 core practices across these central themes of effective mathematics pedagogy. Following their engagement in the professional development, the teachers enacted these practices to greater or lesser extents. However, it was problematic for most teachers to effectively engage their students in deep mathematical discussion. Engaging teachers to design and teach with GeoGebra in the mathematics content area offered a unique lens for understanding the shift in the teachers’ dispositions towards the use of technology in mathematics teaching and learning. As teachers engaged in using GeoGebra, their knowledge and perceived beliefs about the usefulness and nature of technology in mathematics education became profound. The teachers improved their technological pedagogical content knowledge (TPACK) during the study. Analysis of the components of TPACK showed that they improved their knowledge of the mathematics content as well as knowledge of technology, teaching, and students’ learning. However, their intention to put new pedagogical approaches into classroom practice in the future depended on multiple contextual factors including administrative support, continual professional training, and provision of adequate technology facilities. The findings from this study have implications for Ghana’s senior high school mathematics education, TPACK, effective mathematics pedagogy, professional development, and research methodology