Hidden Markov modeling of eye movements with image information leads to better discovery of regions of interest

Conference Theme: Integrating Psychological, Philosophical, Linguistic, Computational and Neural PerspectivesPoster Session 2: no. 56Hidden Markov models (HMM) can describe the spatial and temporal characteristics of eye-tracking recordings in cognitive tasks. Here, we introduce a new HMM approach. We developed HMMs based on fixation locations and we also used image information as an input feature. We demonstrate the benefits of the newly proposed model in a face recognition study wherein an HMM was developed for every subject. Discovery of regions of interest on facial stimuli is improved compared to earlier approaches. Moreover, clustering of the newly developed HMMs lead to very distinct groups. The newly developed approach also allows reconstructing image information at fixation.postprin

Hidden Markov model analysis reveals better eye movement strategies in face recognition

Conference Theme: Mind, Technology, and SocietyHere we explored eye movement strategies that lead to better performance in face recognition with hidden Markov models (HMMs). Participants performed a standard face recognition memory task with eye movements recorded. The durations and locations of the fixations were analyzed using HMMs for both the study and the test phases. Results showed that in the study phase, the participants who looked more often at the eyes and shifted between different regions on the face with long fixation durations had better performances. The test phase analyses revealed that an efficient, short first orienting fixation followed by a more analytic pattern focusing mainly on the eyes led to better performances. These strategies could not be revealed by analysis methods that do not take individual differences in both temporal and spatial dimensions of eye movements into account, demonstrating the power of the HMM approach.postprin

Eye movement pattern in face recognition is associated with cognitive decline in the elderly

Conference Theme: Mind, Technology, and SocietyThe present study investigated the relationship between eye movement pattern in face recognition and cognitive perform-ance during natural aging through modeling and comparing eye movement of young (18-24 years) and older (65-81 years) adults using Hidden Markov Model (HMM) based approach. Young adults recognized faces better than older adults, particularly when measured by the false alarm rate. Older adults’ recognition performance, on the other hand, correlated with their cognitive status assessed by the Montreal Cognitive Assessment (MoCA). Eye movement analysis with HMM revealed two different strategies, namely “analytic” and “holistic”. Participants using the analytic strategy had better recognition performance (particularly in the false alarm rate) than those using the holistic strategy. Significantly more young adults adopted the analytic strategy; whereas more older adults adopted the holistic strategy. Interestingly, older adults with lower cognitive status were associated with higher likelihood of using the holistic strategy. These results suggest an association between holistic eye movement patterns and cognitive decline in the elderly.postprin

Mind reading: discovering individual preferences from eye movements using switching hidden Markov models

Conference Theme: Integrating Psychological, Philosophical, Linguistic, Computational and Neural PerspectivesPoster Session 3: no. 33Here we used a hidden Markov model (HMM) based approach to infer individual choices from eye movements in preference decision-making. We assumed that during a decision making process, participants may switch between exploration and decision-making periods, and this behavior can be better captured with a Switching HMM (SHMM). Through clustering individual eye movement patterns described in SHMMs, we automatically discovered two groups of participants with different decision making behavior. One group showed a strong and early bias to look more often at the to-be chosen stimulus (i.e., the gaze cascade effect; Shimojo et al., 2003) with a short final decision-making period. The other group showed a weaker cascade effect with a longer final decision- making period. The SHMMs also showed capable of inferring participants’ preference choice on each trial with high accuracy. Thus, our SHMM approach made it possible to reveal individual differences in decision making and discover individual preferences from eye movement data.postprin

Analytic eye movement patterns in face recognition are associated with better performance and more top-down control of visual attention: an fMRI study

Conference Theme: Integrating Psychological, Philosophical, Linguistic, Computational and Neural PerspectivesPoster Session 3: no. 57Recent research has revealed two different eye movement patterns during face recognition: holistic and analytic. The present study investigated the neural correlates of these two patterns through functional magnetic resonance imaging (fMRI). A more holistic pattern was associated with more activation in the face-selective perceptual areas, including the occipital face area and fusiform face area. In contrast, participants using a more analytic pattern demonstrated more activation in areas important for top-down control of visual attention, including the frontal eye field and intraparietal sulcus. In addition, participants using the analytic patterns had better recognition performance than those showing holistic patterns. These results suggest that analytic eye movement patterns are associated with more engagement of top-down control of visual attention, which may consequently enhance recognition performance.postprin

Explanation Strategies for Image Classification in Humans vs. Current Explainable AI

Explainable AI (XAI) methods provide explanations of AI models, but our understanding of how they compare with human explanations remains limited. In image classification, we found that humans adopted more explorative attention strategies for explanation than the classification task itself. Two representative explanation strategies were identified through clustering: One involved focused visual scanning on foreground objects with more conceptual explanations diagnostic for inferring class labels, whereas the other involved explorative scanning with more visual explanations rated higher for effectiveness. Interestingly, XAI saliency-map explanations had the highest similarity to the explorative attention strategy in humans, and explanations highlighting discriminative features from invoking observable causality through perturbation had higher similarity to human strategies than those highlighting internal features associated with higher class score. Thus, humans differ in information and strategy use for explanations, and XAI methods that highlight features informing observable causality match better with human explanations, potentially more accessible to users