Relevance Prediction from Eye-movements Using Semi-interpretable Convolutional Neural Networks

We propose an image-classification method to predict the perceived-relevance of text documents from eye-movements. An eye-tracking study was conducted where participants read short news articles, and rated them as relevant or irrelevant for answering a trigger question. We encode participants' eye-movement scanpaths as images, and then train a convolutional neural network classifier using these scanpath images. The trained classifier is used to predict participants' perceived-relevance of news articles from the corresponding scanpath images. This method is content-independent, as the classifier does not require knowledge of the screen-content, or the user's information-task. Even with little data, the image classifier can predict perceived-relevance with up to 80% accuracy. When compared to similar eye-tracking studies from the literature, this scanpath image classification method outperforms previously reported metrics by appreciable margins. We also attempt to interpret how the image classifier differentiates between scanpaths on relevant and irrelevant documents

Moderating effects of self-perceived knowledge in a relevance assessment task : an EEG study

Relevance assessment, a crucial Human-computer Information Retrieval (HCIR) aspect, denotes how well retrieved information meets the user’s information need (IN). Recently, user-centred research benefited from the employment of brain imaging, which contributed to our understanding of relevance assessment and associated cognitive processes. However, the effect of contextual aspects, such as the searcher’s self-perceived knowledge (SPK) on relevance assessment and its underlying neurocognitive processes, has not been studied. This work investigates the impact of users’ SPK about a topic (i.e. ‘knowledgeable’ vs. ‘not knowledgeable’) on relevance assessments (i.e. ‘relevant’ vs. ‘non-relevant’). To do so, using electroencephalography (EEG), we measured the neural activity of twenty-five participants while they provided relevance assessments during the Question and Answering (Q/A) Task. In the analysis, we considered the effects of SPK and specifically how it modulates the brain activity underpinning relevance judgements. Data-driven analysis revealed significant event-related potential differences (P300/CPP, N400, LPC), which were modulated by searchers’ SPK in the context of relevance assessment. We speculate that SPK affects distinct cognitive processes associated with attention, semantic integration and categorisation, memory, and decision formation that underpin relevance assessment formation. Our findings are an important step toward a better understanding of the role users’ SPK plays during relevance assessment

Differentiating Between Empirical and Preferential Decision Strategies

Visual decision-making is a common action that recruits a complex set of cognitive processes. When first presented with an option set from which to choose, participants can rely on one of two distinct decision strategies, preferential and empirical. In a preferential choice, participants choose their most preferred option; there is frequently a so-called gaze bias effect in such choices, where the gaze directed at the chosen option is longer than the gaze at unchosen options. In empirical choices, participants select an objectively correct choice from a set of distractors; these decisions have been shown to produce similar or weaker effects of the gaze bias. Although both forms of decision-making are the subject of scientific investigation, there are no studies that directly compare and contrast the two types. My project is the first to investigate the two decision types using a within-participants experimental design. Participants chose between option pairs in a 2-alternate-forced-choice task with trials grouped into 2 blocks: empirical and preferential. In the empirical block, option pairs contained one correct and one incorrect choice whereas in the preferential condition, option pairs were equal in value (i.e., no correct or incorrect choice). Reaction times for each choice, the number of looks, and duration of gaze for each option were recorded using a computer and eye tracker. To test whether the gaze bias effect occurs equally across these two decision types, different decision stimuli (features, math expressions, and words related to social biases) were used. These experiments thus help to differentiate between preferential and empirical decision using a novel method. Further, by introducing social influences in the manipulations, this research also extends our understanding of social influences on decision-making

iMind: Uma ferramenta inteligente para suporte de compreensão de conteúdo

Usually while reading, content comprehension difficulty affects individual performance. Comprehension difficulties, e. g., could lead to a slow learning process, lower work quality, and inefficient decision-making. This thesis introduces an intelligent tool called “iMind” which uses wearable devices (e.g., smartwatches) to evaluate user comprehension difficulties and engagement levels while reading digital content. Comprehension difficulty can occur when there are not enough mental resources available for mental processing. The mental resource for mental processing is the cognitive load (CL). Fluctuations of CL lead to physiological manifestation of the autonomic nervous system (ANS), which can be measured by wearables, like smartwatches. ANS manifestations are, e. g., an increase in heart rate. With low-cost eye trackers, it is possible to correlate content regions to the measurements of ANS manifestation. In this sense, iMind uses a smartwatch and an eye tracker to identify comprehension difficulty at content regions level (where the user is looking). The tool uses machine learning techniques to classify content regions as difficult or non-difficult based on biometric and non-biometric features. The tool classified regions with a 75% accuracy and 80% f-score with Linear regression (LR). With the classified regions, it will be possible, in the future, to create contextual support for the reader in real-time by, e.g., translating the sentences that induced comprehension difficulty.Normalmente durante a leitura, a dificuldade de compreensão pode afetar o desempenho da leitura. A dificuldade de compreensão pode levar a um processo de aprendizagem mais lento, menor qualidade de trabalho ou uma ineficiente tomada de decisão. Esta tese apresenta uma ferramenta inteligente chamada “iMind” que usa dispositivos vestíveis (por exemplo, smartwatches) para avaliar a dificuldade de compreensão do utilizador durante a leitura de conteúdo digital. A dificuldade de compreensão pode ocorrer quando não há recursos mentais disponíveis suficientes para o processamento mental. O recurso usado para o processamento mental é a carga cognitiva (CL). As flutuações de CL levam a manifestações fisiológicas do sistema nervoso autônomo (ANS), manifestações essas, que pode ser medido por dispositivos vestíveis, como smartwatches. As manifestações do ANS são, por exemplo, um aumento da frequência cardíaca. Com eye trackers de baixo custo, é possível correlacionar manifestação do ANS com regiões do texto, por exemplo. Neste sentido, a ferramenta iMind utiliza um smartwatch e um eye tracker para identificar dificuldades de compreensão em regiões de conteúdo (para onde o utilizador está a olhar). Adicionalmente a ferramenta usa técnicas de machine learning para classificar regiões de conteúdo como difíceis ou não difíceis com base em features biométricos e não biométricos. A ferramenta classificou regiões com uma precisão de 75% e f-score de 80% usando regressão linear (LR). Com a classificação das regiões em tempo real, será possível, no futuro, criar suporte contextual para o leitor em tempo real onde, por exemplo, as frases que induzem dificuldade de compreensão são traduzidas

EYE-AS-AN-INPUT FOR IMPROVING INTERACTIVE INFORMATION RETRIEVAL

In this work, Publication Access Through Tiered Interaction and Exploration (PATTIE) is presented with the eye as an additional input modality. PATTIE is built upon the scatter/gather information retrieval paradigm where users can explore a visual and interactive table-of-contents metaphor for large-scale document collections in an iterative manner. Additionally, the prototype has been integrated with eye-tracking through the web camera and experimental findings are provided to demonstrate a proof-of-concept for interest modeling at the term level and implicit relevance feedback on the gold standard inaugural 2019 Text REtrieval Conference Precision Medicine dataset (TREC PM). Low error rates for gaze tracking, and acceptable performance on binary classification of interest are reported as well as statistically significant increases in precision and recall performance for relevant information on a TREC PM task when PATTIE is used with eye-as-an-input versus a baseline PATTIE system.Doctor of Philosoph