Associating Facial Expressions and Upper-Body Gestures with Learning Tasks for Enhancing Intelligent Tutoring Systems

Learning involves a substantial amount of cognitive, social and emotional states. Therefore, recognizing and understanding these states in the context of learning is key in designing informed interventions and addressing the needs of the individual student to provide personalized education. In this paper, we explore the automatic detection of learner’s nonverbal behaviors involving hand-over-face gestures, head and eye movements and emotions via facial expressions during learning. The proposed computer vision-based behavior monitoring method uses a low-cost webcam and can easily be integrated with modern tutoring technologies. We investigate these behaviors in-depth over time in a classroom session of 40 minutes involving reading and problem-solving exercises. The exercises in the sessions are divided into three categories: an easy, medium and difficult topic within the context of undergraduate computer science. We found that there is a significant increase in head and eye movements as time progresses, as well as with the increase of difficulty level. We demonstrated that there is a considerable occurrence of hand-over-face gestures (on average 21.35%) during the 40 minutes session and is unexplored in the education domain. We propose a novel deep learning approach for automatic detection of hand-over-face gestures in images with a classification accuracy of 86.87%. There is a prominent increase in hand-over-face gestures when the difficulty level of the given exercise increases. The hand-over-face gestures occur more frequently during problem-solving (easy 23.79%, medium 19.84% and difficult 30.46%) exercises in comparison to reading (easy 16.20%, medium 20.06% and difficult 20.18%)

Single Epoch Analysis and Bi-hemisphere Study of Magnetoencephalographic (MEG) Signals using Vector Signal Transformation V3 and Magnetic Field Tomography (MFT)

The biomagnetic inverse problem has no unique solution, nevertheless even a cursory look at the features shown in raw signal can often suffice to highlight strong superficial activity. To do a proper single epoch analysis is normally prohibitively expensive in terms of computing demands. Hence the original aim of this thesis was to use simple efficient signal transformations to characterize superficial generators and contrast the single epoch signature with that extracted from the average signal. The results have intrigued us sufficiently to go beyond the original goal and extract very preliminary estimates of activity across the cerebral hemisphere in single trials. The original tool, and one that we have used for much of the work, is a simple vector signal transformation called V3. This signal transformation highlights nearby sources; it is a crude but quick estimator of generators directly from the raw MEG signals. Together with Magnetic Field Tomography (MFT), which relies on distributed source analysis of the MEG signals, we have tackled the following specific problems relating to aspects of normal brain function: efficient estimation of generators of magnetic fields; relationship between the average signal and single trials; and interhemispheric differences and relationship between the activity in the left and right hemispheres of the brain. During the project, we have used as examples auditory evoked MEG measurements obtained from two multichannel systems and applied the V3 and MFT analysis to both the average and single trial signals. In particular, we chose the 40-Hz (or gamma band) auditory response as the study subject. We found that in single epochs similar patterns of high frequency activity are observed in the area around the auditory cortex well before, close to and well after stimulus onset; the sequence of events observed in the average can only represent the evolution of events in single trials in a statistical way; and deep and central areas of the brain may be the seeds for the main deflections observed in the auditory responses

Characterising the neck motor system of the blowfly

Flying insects use visual, mechanosensory, and proprioceptive information to control their movements, both when on the ground and when airborne. Exploiting visual information for motor control is significantly simplified if the eyes remain aligned with the external horizon. In fast flying insects, head rotations relative to the body enable gaze stabilisation during highspeed manoeuvres or externally caused attitude changes due to turbulent air. Previous behavioural studies into gaze stabilisation suffered from the dynamic properties of the supplying sensor systems and those of the neck motor system being convolved. Specifically, stabilisation of the head in Dipteran flies responding to induced thorax roll involves feed forward information from the mechanosensory halteres, as well as feedback information from the visual systems. To fully understand the functional design of the blowfly gaze stabilisation system as a whole, the neck motor system needs to be investigated independently. Through X-ray micro-computed tomography (μCT), high resolution 3D data has become available, and using staining techniques developed in collaboration with the Natural History Museum London, detailed anatomical data can be extracted. This resulted in a full 3- dimensional anatomical representation of the 21 neck muscle pairs and neighbouring cuticula structures which comprise the blowfly neck motor system. Currently, on the work presented in my PhD thesis, μCT data are being used to infer function from structure by creating a biomechanical model of the neck motor system. This effort aims to determine the specific function of each muscle individually, and is likely to inform the design of artificial gaze stabilisation systems. Any such design would incorporate both sensory and motor systems as well as the control architecture converting sensor signals into motor commands under the given physical constraints of the system as a whole.Open Acces

Beyond Usability: An Alternative Usability Evaluation Method, PUT-Q2

Usability can be thought of as a measure or degree to which a system satisfies the needs of the human. Usability is a quality inherent to any given system, which assists in determining the efficiency, effectiveness and satisfaction levels of those involved in the interaction. Everyday we are bombarded with interactions and experiences that shape our thoughts, values, and judgments as well as test our limits of interaction with technology. These interactions have progressed at such an intense pace that humans have become practically slaves to technological innovation. Humans are forced to conform with needs of technology, rather then technology conforming to human needs. This fact must be rectified and becomes the primary focus of this thesis. Current models in usability evaluation methods (UEMs) analyze the quantitative data collected during testing. These statistical studies provide insight into limited aspects of usability, and most overlook human dimensions, including perception and affective responses; thus leaving a glaring pitfall in the overall analysis of system usability. By analyzing a new qualitative channel of data, this research attempts to explain these human-dimensional factors. Up to this point no evaluation model has been largely accepted which attempts to fuse both qualitative and quantitative data. This research proposes an alternative UEM, incorporating both qualitative and quantitative data, called the Perception and Usability Testing combining Qualitative and Quantitative data, or PUT-Q2. This new usability evaluation method presents complex qualitative and quantitative data in graphical visualizations and matrices that assist the usability expert in uncovering additional correlations and usability issues with their system

Electrophysiology

The outstanding evolution of recording techniques paved the way for better understanding of electrophysiological phenomena within the human organs, including the cardiovascular, ophthalmologic and neural systems. In the field of cardiac electrophysiology, the development of more and more sophisticated recording and mapping techniques made it possible to elucidate the mechanism of various cardiac arrhythmias. This has even led to the evolution of techniques to ablate and cure most complex cardiac arrhythmias. Nevertheless, there is still a long way ahead and this book can be considered a valuable addition to the current knowledge in subjects related to bioelectricity from plants to the human heart

Efficient techniques for soft tissue modeling and simulation

Performing realistic deformation simulations in real time is a challenging problem in computer graphics. Among numerous proposed methods including Finite Element Modeling and ChainMail, we have implemented a mass spring system because of its acceptable accuracy and speed. Mass spring systems have, however, some drawbacks such as, the determination of simulation coefficients with their iterative nature. Given the correct parameters, mass spring systems can accurately simulate tissue deformations but choosing parameters that capture nonlinear deformation behavior is extremely difficult. Since most of the applications require a large number of elements i. e. points and springs in the modeling process it is extremely difficult to reach realtime performance with an iterative method. We have developed a new parameter identification method based on neural networks. The structure of the mass spring system is modified and neural networks are integrated into this structure. The input space consists of changes in spring lengths and velocities while a "teacher" signal is chosen as the total spring force, which is expressed in terms of positional changes and applied external forces. Neural networks are trained to learn nonlinear tissue characteristics represented by spring stiffness and damping in the mass spring algorithm. The learning algorithm is further enhanced by an adaptive learning rate, developed particularly for mass spring systems. In order to avoid the iterative approach in deformation simulations we have developed a new deformation algorithm. This algorithm defines the relationships between points and springs and specifies a set of rules on spring movements and deformations. These rules result in a deformation surface, which is called the search space. The deformation algorithm then finds the deformed points and springs in the search space with the help of the defined rules. The algorithm also sets rules on each element i. e. triangle or tetrahedron so that they do not pass through each other. The new algorithm is considerably faster than the original mass spring systems algorithm and provides an opportunity for various deformation applications. We have used mass spring systems and the developed method in the simulation of craniofacial surgery. For this purpose, a patient-specific head model was generated from MRI medical data by applying medical image processing tools such as, filtering, the segmentation and polygonal representation of such model is obtained using a surface generation algorithm. Prism volume elements are generated between the skin and bone surfaces so that different tissue layers are included to the head model. Both methods produce plausible results verified by surgeons

Machine learning methods detect arm movement impairments in a patient with parieto-occipital lesion using only early kinematic information

Patients with lesions of the parieto-occipital cortex typically misreach visual targets that they correctly perceive (optic ataxia). Although optic ataxia was described more than 30 years ago, distinguishing this condition from physiological behavior using kinematic data is still far from being an achievement. Here, combining kinematic analysis with machine learning methods, we compared the reaching performance of a patient with bilateral occipitoparietal damage with that of 10 healthy controls. They performed visually guided reaches toward targets located at different depths and directions. Using the horizontal, sagittal, and vertical deviation of the trajectories, we extracted classification accuracy in discriminating the reaching performance of patient from that of controls. Specifically, accurate predictions of the patient's deviations were detected after the 20% of the movement execution in all the spatial positions tested. This classification based on initial trajectory decoding was possible for both directional and depth components of the movement, suggesting the possibility of applying this method to characterize pathological motor behavior in wider frameworks

FINITE ELEMENT MODELING AND SIMULATION OF OCCUPANT RESPONSES IN HIGHWAY CRASHES

Roadside barrier systems play an important role in reducing the number of fatalities and the severity of injuries in highway crashes. After decades of work by researchers and engineers, roadside barriers have been improved and are generally effective in preventing head-on collisions and thus crash fatalities. To further improve the performance of highway safety devices and develop new systems, a good understanding of occupant injuries is required. Although incorporating occupant responses and/or injuries into the design of safety devices is highly recommended by the current safety regulations, there are currently no studies that can be used to develop official guidelines or standards. Despite its usefulness in understanding the crash mechanism and improving vehicle crashworthiness, crash testing is very expensive and restricted by the crash scenarios that can be investigated. In addition, no crash test dummy is incorporated in majority of the crash testing of roadside barriers. With the recent advances in high performance computing and numerical codes, computer modeling and simulation are playing an important role in crash analysis and roadside safety research. In this study, the finite element model of a Hybrid III 50th percentile male dummy was developed for studying the driver’s responses in vehicular crashes into highway barriers. After validation by standard crash tests, the dummy model was combined with the finite element model of a 2006 Ford F250 pickup truck and used in simulations of the vehicle impacting a concrete barrier and a W-beam guiderail under different impact speeds and angles. Finally, the dummy responses in these simulations were analyzed by correlating with existing human injury criteria so as to correlate impact severity to vehicular responses and ultimately to barrier performances