Finding the Hidden: Detecting Atypical Affective States from Physiological Signals

In cognitive science, intuition is described as a strategy of processing information that relies on people's instinctive and emotional criteria. When compared with the deliberate choices made after conscious reasoning, the quick and intuitive decision making strategies can be more effective. The intuitive thinking provokes changes in human physiological responses which can be measured by sensors. Utilising physiological reactions, previous work shows that atypical patterns such as emotion expressions and image manipulations can be identified. This thesis expands the exploration to examine whether more atypical human behaviour can be recognised from physiological signals. The examined subtly atypical behaviour includes depression, doubt and deception, Depression is a serious chronic mental disease and is considered as an atypical health condition in people. Doubt is defined as a non-deliberate attempt to mislead others and is a passive form of deception, representing an atypicality from honest behaviours. Deception is a more purposeful attempt to deceive, and thus is a distinct type of atypicality than honest communication. Through examining physiological reactions from presenters who have a particular atypical behaviour or condition, and observers who view behaviours of presenters, this research aims to recognise atypicality in human behaviour. A collection of six user studies are conducted. In two user studies, presenters are asked to conduct doubting and deceiving behaviours, while the remaining user studies involve observers watching behaviours of presenters who suffer from depression, have doubt, or have conducted deception. Physiological reactions of both presenters and observers are collected, including Blood Volume Pulse, Electrodermal Activity, Skin Temperature and Pupillary Responses. Observers are also asked to explicitly evaluate whether the viewed presenters were being depressed, doubting, or deceiving. Investigations upon physiological data in this thesis finds that detectable cues corresponding with depression, doubt and deception can be found. Viewing depression provokes visceral physiological reactions in observers that can be measured. Such physiological responses can be used to derive features for machine learning models to accurately distinguish between healthy individuals and people with depression. By contrast, depression does not provoke strong conscious recognition in observers, resulting in a conscious evaluation accuracy slightly above chance level. Similar results are also found in detecting doubt and deception. People with doubt and deceit elicit consistent physiological reactions within themselves. These bodily responses can be utilised by machine learning models or deep learning models to recognise doubt or deception. The doubt and deceit in presenters can also be recognised using physiological signals in observers, with excellent recognition rates which are higher when compared with the conscious judgments from the same group of observers. The results indicate that atypicality in presenters can both be captured by physiological signals of presenters and observers. Presenters' physiological reactions contribute to higher recognition of atypicality, but observers' physiological responses can serve as a comparable alternative. The awareness of atypicality among observers happens physiologically, so can be used by machine learning models, even when they do not reach the consciousness of the person. The research findings lead to a further discussion around the implications of observers' physiological responses. Decision support applications which utilise a quantifiable measure of people's unconscious and intuitive 'gut feeling' can be developed based on the work reported here to assist people with medical diagnosis, information credibility evaluation, and criminal detection. Further research suggests exploring more atypical behaviours in the wild

Predictive cognition in dementia: the case of music

The clinical complexity and pathological diversity of neurodegenerative diseases impose immense challenges for diagnosis and the design of rational interventions. To address these challenges, there is a need to identify new paradigms and biomarkers that capture shared pathophysiological processes and can be applied across a range of diseases. One core paradigm of brain function is predictive coding: the processes by which the brain establishes predictions and uses them to minimise prediction errors represented as the difference between predictions and actual sensory inputs. The processes involved in processing unexpected events and responding appropriately are vulnerable in common dementias but difficult to characterise. In my PhD work, I have exploited key properties of music – its universality, ecological relevance and structural regularity – to model and assess predictive cognition in patients representing major syndromes of frontotemporal dementia – non-fluent variant PPA (nfvPPA), semantic-variant PPA (svPPA) and behavioural-variant FTD (bvFTD) - and Alzheimer’s disease relative to healthy older individuals. In my first experiment, I presented patients with well-known melodies containing no deviants or one of three types of deviant - acoustic (white-noise burst), syntactic (key-violating pitch change) or semantic (key-preserving pitch change). I assessed accuracy detecting melodic deviants and simultaneously-recorded pupillary responses to these deviants. I used voxel-based morphometry to define neuroanatomical substrates for the behavioural and autonomic processing of these different types of deviants, and identified a posterior temporo-parietal network for detection of basic acoustic deviants and a more anterior fronto-temporo-striatal network for detection of syntactic pitch deviants. In my second chapter, I investigated the ability of patients to track the statistical structure of the same musical stimuli, using a computational model of the information dynamics of music to calculate the information-content of deviants (unexpectedness) and entropy of melodies (uncertainty). I related these information-theoretic metrics to performance for detection of deviants and to ‘evoked’ and ‘integrative’ pupil reactivity to deviants and melodies respectively and found neuroanatomical correlates in bilateral dorsal and ventral striatum, hippocampus, superior temporal gyri, right temporal pole and left inferior frontal gyrus. Together, chapters 3 and 4 revealed new hypotheses about the way FTD and AD pathologies disrupt the integration of predictive errors with predictions: a retained ability of AD patients to detect deviants at all levels of the hierarchy with a preserved autonomic sensitivity to information-theoretic properties of musical stimuli; a generalized impairment of surprise detection and statistical tracking of musical information at both a cognitive and autonomic levels for svPPA patients underlying a diminished precision of predictions; the exact mirror profile of svPPA patients in nfvPPA patients with an abnormally high rate of false-alarms with up-regulated pupillary reactivity to deviants, interpreted as over-precise or inflexible predictions accompanied with normal cognitive and autonomic probabilistic tracking of information; an impaired behavioural and autonomic reactivity to unexpected events with a retained reactivity to environmental uncertainty in bvFTD patients. Chapters 5 and 6 assessed the status of reward prediction error processing and updating via actions in bvFTD. I created pleasant and aversive musical stimuli by manipulating chord progressions and used a classic reinforcement-learning paradigm which asked participants to choose the visual cue with the highest probability of obtaining a musical ‘reward’. bvFTD patients showed reduced sensitivity to the consequence of an action and lower learning rate in response to aversive stimuli compared to reward. These results correlated with neuroanatomical substrates in ventral and dorsal attention networks, dorsal striatum, parahippocampal gyrus and temporo-parietal junction. Deficits were governed by the level of environmental uncertainty with normal learning dynamics in a structured and binarized environment but exacerbated deficits in noisier environments. Impaired choice accuracy in noisy environments correlated with measures of ritualistic and compulsive behavioural changes and abnormally reduced learning dynamics correlated with behavioural changes related to empathy and theory-of-mind. Together, these experiments represent the most comprehensive attempt to date to define the way neurodegenerative pathologies disrupts the perceptual, behavioural and physiological encoding of unexpected events in predictive coding terms

Neural correlates of visual awareness

openL'elaborato si propone di esporre le attuali evidenze riguardanti il modo in cui i contenuti soggettivi di consapevolezza visiva sono codificati a livello neurale. Sebbene i meccanismi neurali della percezione visiva siano ampiamente conosciuti, rimane ancora da chiarire come l'informazione visiva entri a far parte dei contenuti della coscienza. Per identificare i correlati neurali della coscienza (CNC), che rappresentano la minima attività neurale per una specifica esperienza conscia, vengono messe in relazione misure comportamentali di consapevolezza, limitatamente a stimoli presentati in un contesto sperimentale, con i sottostanti meccanismi neurali. Attraverso paradigmi sperimentali come la rivalità binoculare e tecniche di mascheramento visivo è possibile provare ad identificare i CNC contenuto-specifici utilizzando misure neurofisiologiche e tecniche di neuroimaging. Tali tecniche forniscono infatti utili informazioni circa le basi neuroanatomiche e funzionali dell'esperienza sotto esame. Sebbene i meccanismi che sottendono l’attenzione siano spesso associati all'esperienza cosciente, evidenze sperimentali suggeriscono una separazione tra i due processi. Le ricerche sui correlati neurali della consapevolezza visiva indicano come l’attività di una singola area cerebrale non possa essere necessaria e sufficiente a spiegare le qualità dei contenuti coscienti. Sembrerebbe invece essere necessaria una rappresentazione della scena visiva distribuita nella corteccia visiva primaria (V1) e nelle aree visive ventrali con attivazione di regioni temporo-parietali. Misure elettrofisiologiche come la visual awareness negativity (VAN) sono state correlate alla consapevolezza visiva mentre altri indicatori sembrerebbero essere maggiormente legati a processi attentivi. Diversi modelli teorici offrono spiegazioni empiriche sull’emergenza della coscienza dall’attività cerebrale. Nel caso della consapevolezza visiva, alcuni modelli teorici rilevanti sono la teoria dello spazio di lavoro neurale globale, la quale sottolinea la necessità di condivisione dell'informazione tra ampie aree cerebrali e la teoria dell'elaborazione ricorrente che si concentra invece sul feedback proveniente a V1 dalle aree extrastriate. Inoltre, il modello dell’”elaborazione predittiva” descrive la percezione cosciente come il risultato di un processo attivo in cui il cervello crea costantemente previsioni sull’ambiente circostante. Allo stato attuale, la ricerca sui correlati neurali della consapevolezza visiva evidenzia dunque come un network di regioni cerebrali posteriori sia fondamentale per avere esperienze visive coscienti. Inoltre, i segnali di feedback sembrano svolgere un ruolo cruciale, evidenziando le complesse interazioni tra dinamiche neurali e percezione cosciente.The paper aims to present the current evidence regarding how subjective contents of visual awareness are encoded at the neural level. While the neural mechanisms of visual perception are well understood, it remains unclear how visual information becomes part of consciousness. To identify the neural correlates of consciousness (NCC), representing the minimum neural activity for a specific conscious experience, behavioral measures of awareness are related to underlying neural mechanisms, limited to stimuli presented in an experimental context. Through experimental paradigms such as binocular rivalry and visual masking techniques, it is possible to attempt to identify content-specific NCC using neurophysiological measures and neuroimaging techniques. These techniques indeed provide valuable information about the neuroanatomical and functional basis of the examined experience. Although mechanisms underlying attention are often associated with conscious experience, experimental evidence suggests a separation between the two processes. Research on the neural correlates of visual awareness indicates that the activity of a single brain area may not be necessary and sufficient to explain the qualities of conscious contents. Instead, a distributed representation of the visual scene in the primary visual cortex (V1) and ventral visual areas with activation of temporo-parietal regions seems to be necessary. Electrophysiological measures such as Visual Awareness Negativity (VAN) have been correlated with visual awareness, while other indicators appear to be more related to attentional processes. Various theoretical models offer empirical explanations of the emergence of consciousness from brain activity. In the case of visual awareness, some relevant theoretical models include the global neural workspace theory, which emphasizes the need for information sharing among extensive brain areas, and the recurrent processing theory, which focuses on feedback from extrastriate areas to V1. Additionally, the predictive processing model describes conscious perception as the result of an active process in which the brain constantly generates predictions about the surrounding environment. Currently, research on the neural correlates of visual awareness highlights the importance of a network of posterior brain regions for conscious visual experiences. Furthermore, feedback signals appear to play a crucial role, highlighting the complex interactions between neural dynamics and conscious perception

Inferring implicit relevance from physiological signals

Ongoing growth in data availability and consumption has meant users are increasingly faced with the challenge of distilling relevant information from an abundance of noise. Overcoming this information overload can be particularly difficult in situations such as intelligence analysis, which involves subjectivity, ambiguity, or risky social implications. Highly automated solutions are often inadequate, therefore new methods are needed for augmenting existing analysis techniques to support user decision making. This project investigated the potential for deep learning to infer the occurrence of implicit relevance assessments from users' biometrics. Internal cognitive processes manifest involuntarily within physiological signals, and are often accompanied by 'gut feelings' of intuition. Quantifying unconscious mental processes during relevance appraisal may be a useful tool during decision making by offering an element of objectivity to an inherently subjective situation. Advances in wearable or non-contact sensors have made recording these signals more accessible, whilst advances in artificial intelligence and deep learning have enhanced the discovery of latent patterns within complex data. Together, these techniques might make it possible to transform tacit knowledge into codified knowledge which can be shared. A series of user studies recorded eye gaze movements, pupillary responses, electrodermal activity, heart rate variability, and skin temperature data from participants as they completed a binary relevance assessment task. Participants were asked to explicitly identify which of 40 short-text documents were relevant to an assigned topic. Investigations found this physiological data to contain detectable cues corresponding with relevance judgements. Random forests and artificial neural networks trained on features derived from the signals were able to produce inferences with moderate correlations with the participants' explicit relevance decisions. Several deep learning algorithms trained on the entire physiological time series data were generally unable to surpass the performance of feature-based methods, and instead produced inferences with low correlations with participants' explicit personal truths. Overall, pupillary responses, eye gaze movements, and electrodermal activity offered the most discriminative power, with additional physiological data providing diminishing or adverse returns. Finally, a conceptual design for a decision support system is used to discuss social implications and practicalities of quantifying implicit relevance using deep learning techniques. Potential benefits included assisting with introspection and collaborative assessment, however quantifying intrinsically unknowable concepts using personal data and abstruse artificial intelligence techniques were argued to pose incommensurate risks and challenges. Deep learning techniques therefore have the potential for inferring implicit relevance in information-rich environments, but are not yet fit for purpose. Several avenues worthy of further research are outlined

Knowledge management framework based on brain models and human physiology

The life of humans and most living beings depend on sensation and perception for the best assessment of the surrounding world. Sensorial organs acquire a variety of stimuli that are interpreted and integrated in our brain for immediate use or stored in memory for later recall. Among the reasoning aspects, a person has to decide what to do with available information. Emotions are classifiers of collected information, assigning a personal meaning to objects, events and individuals, making part of our own identity. Emotions play a decisive role in cognitive processes as reasoning, decision and memory by assigning relevance to collected information. The access to pervasive computing devices, empowered by the ability to sense and perceive the world, provides new forms of acquiring and integrating information. But prior to data assessment on its usefulness, systems must capture and ensure that data is properly managed for diverse possible goals. Portable and wearable devices are now able to gather and store information, from the environment and from our body, using cloud based services and Internet connections. Systems limitations in handling sensorial data, compared with our sensorial capabilities constitute an identified problem. Another problem is the lack of interoperability between humans and devices, as they do not properly understand human’s emotional states and human needs. Addressing those problems is a motivation for the present research work. The mission hereby assumed is to include sensorial and physiological data into a Framework that will be able to manage collected data towards human cognitive functions, supported by a new data model. By learning from selected human functional and behavioural models and reasoning over collected data, the Framework aims at providing evaluation on a person’s emotional state, for empowering human centric applications, along with the capability of storing episodic information on a person’s life with physiologic indicators on emotional states to be used by new generation applications

Alterations in The States and Contents of Consciousness: Empirical and Theoretical Aspects

The main purpose of the present doctoral thesis is to investigate subjective experiences and cognitive processes in four different types of altered states of consciousness: naturally occurring dreaming, cognitively induced hypnosis, pharmacologically induced sedation, and pathological psychosis. Both empirical and theoretical research is carried out, resulting in four empirical and four theoretical studies. The thesis begins with a review of the main concepts used in consciousness research, the most influential philosophical and neurobiological theories of subjective experience, the classification of altered states of consciousness, and the main empirical methods used to study consciousness alterations. Next, findings of the original studies are discussed, as follows. Phenomenal consciousness is found to be dissociable from responsiveness, as subjective experiences do occur in unresponsive states, including anaesthetic-induced sedation and natural sleep, as demonstrated by post-awakening subjective reports. Two new tools for the content analysis of subjective experiences and dreams are presented, focusing on the diversity, complexity and dynamics of phenomenal consciousness. In addition, a new experimental paradigm of serial awakenings from non-rapid eye movement sleep is introduced, which enables more rapid sampling of dream reports than has been available in previous studies. It is also suggested that lucid dreaming can be studied using transcranial brain stimulation techniques and systematic analysis of pre-lucid dreaming. For blind judges, dreams of psychotic patients appear to be indistinguishable from waking mentation reports collected from the same patients, which indicates a close resemblance of these states of mind. However, despite phenomenological similarities, dreaming should not be treated as a uniform research model of psychotic or intact consciousness. Contrary to this, there seems to be a multiplicity of routes of how different states of consciousness can be associated. For instance, seemingly identical time perception distortions in different alterations of consciousness may have diverse underlying causes for these distortions. It is also shown that altered states do not necessarily exhibit impaired cognitive processing compared to a baseline waking state of consciousness: a case study of time perception in a hypnotic virtuoso indicates a more consistent perceptual timing under hypnosis than in a waking state. The thesis ends with a brief discussion of the most promising new perspectives for the study of alterations of consciousness.Siirretty Doriast

Attention Restraint, Working Memory Capacity, and Mind Wandering: Do Emotional Valence or Intentionality Matter?

Attention restraint appears to mediate the relationship between working memory capacity (WMC) and mind wandering (Kane et al., 2016). Prior work has identifed two dimensions of mind wandering—emotional valence and intentionality. However, less is known about how WMC and attention restraint correlate with these dimensions. Te current study examined the relationship between WMC, attention restraint, and mind wandering by emotional valence and intentionality. A confrmatory factor analysis demonstrated that WMC and attention restraint were strongly correlated, but only attention restraint was related to overall mind wandering, consistent with prior fndings. However, when examining the emotional valence of mind wandering, attention restraint and WMC were related to negatively and positively valenced, but not neutral, mind wandering. Attention restraint was also related to intentional but not unintentional mind wandering. Tese results suggest that WMC and attention restraint predict some, but not all, types of mind wandering

Affective Computing

This book provides an overview of state of the art research in Affective Computing. It presents new ideas, original results and practical experiences in this increasingly important research field. The book consists of 23 chapters categorized into four sections. Since one of the most important means of human communication is facial expression, the first section of this book (Chapters 1 to 7) presents a research on synthesis and recognition of facial expressions. Given that we not only use the face but also body movements to express ourselves, in the second section (Chapters 8 to 11) we present a research on perception and generation of emotional expressions by using full-body motions. The third section of the book (Chapters 12 to 16) presents computational models on emotion, as well as findings from neuroscience research. In the last section of the book (Chapters 17 to 22) we present applications related to affective computing