Sympathetic Loading in Critical Tasks

In this dissertation I developed or perfected unobtrusive methods to quantify sympathetic arousals. Furthermore, I used these methods to study the sympathetic system's role on critical activities, arriving at intriguing conclusions. Sympathetic arousals occur during states of mental, emotional, and/or sensorimotor strain resulting from adverse or demanding circumstances. They are key elements of human physiology's coping mechanism, shoring up resources to a good effect. When the intensity and duration of these arousals are overwhelming, however, then they may block memory and disrupt rational thought or actions at the moment they are needed the most. Arousals abound in three types of critical activities: high-stakes situations, challenging tasks, and critical multitasking. Accordingly, my research was based on three studies representative of these three activity types: `Subject Screening', `Educational Exam', and `Distracted Driving'. In the first study I investigated the association of sympathetic arousals with deceptive behavior in interrogations. In the second study, I investigated the relationship between sympathetic arousals and exam performance. In the third study, I investigated the interaction between sympathetic arousals and driving performance under cognitive, emotional, and sensorimotor distractions. In the interrogation study, I used for the first time a contact-free electrodermal activity measurement method to quantify arousals. The method detected deceptive behavior based on differential sympathetic responses in well-structured interviews. In the exam study, I documented that sympathetic arousals positively correlate with students' exam performance, dispelling the myth of `easy going' super achievers. Finally, in the driving study, my results revealed that not only apparent sensorimotor stressors (texting while driving) but also hidden stressors (cognitive or emotional) could have a significant effect on driving performance.Computer Science, Department o

Machine Learning-based Lie Detector applied to a Novel Annotated Game Dataset

Lie detection is considered a concern for everyone in their day to day life given its impact on human interactions. Thus, people normally pay attention to both what their interlocutors are saying and also to their visual appearances, including faces, to try to find any signs that indicate whether the person is telling the truth or not. While automatic lie detection may help us to understand this lying characteristics, current systems are still fairly limited, partly due to lack of adequate datasets to evaluate their performance in realistic scenarios. In this work, we have collected an annotated dataset of facial images, comprising both 2D and 3D information of several participants during a card game that encourages players to lie. Using our collected dataset, We evaluated several types of machine learning-based lie detectors in terms of their generalization, person-specific and cross-domain experiments. Our results show that models based on deep learning achieve the best accuracy, reaching up to 57\% for the generalization task and 63\% when dealing with a single participant. Finally, we also highlight the limitation of the deep learning based lie detector when dealing with cross-domain lie detection tasks

Implementation of wavelet analysis on thermal images for affective states recognition of children with autism spectrum disorder

Children with Autism Spectrum Disorder are identified as a group of people who has difficulties in socio-emotional interaction. Most of them lack the proper context in producing social response through facial expression and speech. Since emotion is the key for effective social interaction, it is justifiably vital for them to comprehend the correct emotion expressions and recognitions. Emotion is a type of affective states and can be detected through physical reaction and physiological signals. In general, recognition of affective states from physical reaction such as facial expression and speech for autistic children is often unpredictable. Hence, an alternative method of identifying the affective states through physiological signals is proposed. Though considered non-invasive, most of the current recognition methods require sensors to be patched on to the skin body to measure the signals. This would most likely cause discomfort to the children and mask their 'true' affective states. The study proposed the use of thermal imaging modality as a passive medium to analyze the physiological signals associated with the affective states nonobtrusively. The study hypothesized that, the impact of cutaneous temperature changes due to the pulsating blood flow in the blood vessels at the frontal face area measured from the modality could have a direct impact to the different affective states of autistic children. A structured experimental setup was designed to measure thermal imaging data generated from different affective state expressions induced using different sets of audio-video stimuli. A wavelet-based technique for pattern detection in time series was deployed to spot the changes measured from the region of interest. In the study, the affective state model for typical developing children aged between 5 and 9 years old was used as the baseline to evaluate the performance of the affective state classifier for autistic children. The results from the classifier showed the efficacy of the technique and accorded good performance of classification accuracy at 88% in identifying the affective states of autistic children. The results were momentous in distinguishing basic affective states and the information could provide a more effective response towards improving social-emotion interaction amongst the autistic children

Using Polygraph to Detect Passengers Carrying Illegal Items

The present study examined the effectiveness of a Modified-Comparison Questions Technique, used in conjunction with the polygraph, to differentiate between common travelers, drug traffickers, and terrorists at transportation hubs. Two experiments were conducted using a mock crime paradigm. In Experiment 1, we randomly assigned 78 participants to either a drug condition, where they packed and lied about illicit drugs in their luggage, or a control condition, where they did not pack or lie about any illegal items. In Experiment 2, we randomly assigned 164 participants to one of the two conditions in Experiment 1 or an additional bomb condition, where they packed and lied about a bomb in their luggage. For both experiments, we assessed participants’ RR interval, heart rate, peak-to-peak amplitude of Galvanic Skin Response (GSR) and all three combined, using Discriminant Analyses to determine the classification accuracy of participants in each condition. In both experiments, we found decelerated heart rates and increased peak-to-peak amplitude of GSR in guilty participants when lying in response to questions regarding their crime. We also found accurate classifications of participants, in both Experiment 1 (drug vs. control: 84.2% vs. 82.5%) and Experiment 2 (drug vs. control: 82:1% vs. 95.1%; bomb vs. control: 93.2% vs. 95.1%; drug vs. bomb: 92.3% vs. 90.9%), above chance level. These findings indicate that Modified-CQT, combined with a polygraph test, is a viable method for investigating suspects of drug trafficking and terrorism at transportation hubs such as train stations and airports

Infrared thermal imaging in affective neuroscience: insights to the self from the peripheral nervous system.

Changes in peripheral physiology lay in the unconscious and occur as a response to external challenges, whether is to fight a virus (e.g. fever) a predator (e.g Fight or Flight) or even to face a social challenge. Autonomic adaptation carries its own physiological print and by harnessing the power given by homeostatic balance, distinctions can be made between arousal and parasympathetic restoration. Conventional physiological methods restrict the way in which experimental designs can be performed. Functional Infrared Thermal Imaging (fITI) provides an alternative for physiological monitoring that enables experimental paradigms that resemble real life situations. With the use of thermal imaging the following studies were set to examine self-conscious emotions in a naturalistic experimental setting while advancing methodologically the technique of fITI. In the following chapters the potentialities and limits of fITI are illustrated (Chapter 2) and three studies are presented where fITI has been applied to investigate the autonomic signature of guilt in children (Chapter 3); the facial imprints of autonomic contagion in mother and child (Chapter 4); the role of social proximity and gaze in modulating facial temperature (Chapter 5). FITI has managed to reliably and systematically collect physiological thermal changes between affective states illustrating a new pathway for contact-free autonomic monitoring in the arena of self-conscious emotions

Mobile Thermography-based Physiological Computing for Automatic Recognition of a Person’s Mental Stress

This thesis explores the use of Mobile Thermography1, a significantly less investigated sensing capability, with the aim of reliably extracting a person’s multiple physiological signatures and recognising mental stress in an automatic, contactless manner. Mobile thermography has greater potentials for real-world applications because of its light-weight, low computation-cost characteristics. In addition, thermography itself does not necessarily require the sensors to be worn directly on the skin. It raises less privacy concerns and is less sensitive to ambient lighting conditions. The work presented in this thesis is structured through a three-stage approach that aims to address the following challenges: i) thermal image processing for mobile thermography in variable thermal range scenes; ii) creation of rich and robust physiology measurements; and iii) automated stress recognition based on such measurements. Through the first stage (Chapter 4), this thesis contributes new processing techniques to address negative effects of environmental temperature changes upon automatic tracking of regions-of-interest and measuring of surface temperature patterns. In the second stage (Chapters 5,6,7), the main contributions are: robustness in tracking respiratory and cardiovascular thermal signatures both in constrained and unconstrained settings (e.g. respiration: strong correlation with ground truth, r=0.9987), and investigation of novel cortical thermal signatures associated with mental stress. The final stage (Chapters 8,9) contributes automatic stress inference systems that focus on capturing richer dynamic information of physiological variability: firstly, a novel respiration representation-based system (which has achieved state-of-the-art performance: 84.59% accuracy, two stress levels), and secondly, a novel cardiovascular representation-based system using short-term measurements of nasal thermal variability and heartrate variability from another sensing channel (78.33% accuracy achieved from 20seconds measurements). Finally, this thesis contributes software libraries and incrementally built labelled datasets of thermal images in both constrained and everyday ubiquitous settings. These are used to evaluate performance of our proposed computational methods across the three-stages