Inferring User Knowledge Level from Eye Movement Patterns

The acquisition of information and the search interaction process is influenced strongly by a person’s use of their knowledge of the domain and the task. In this paper we show that a user’s level of domain knowledge can be inferred from their interactive search behaviors without considering the content of queries or documents. A technique is presented to model a user’s information acquisition process during search using only measurements of eye movement patterns. In a user study (n=40) of search in the domain of genomics, a representation of the participant’s domain knowledge was constructed using self-ratings of knowledge of genomics-related terms (n=409). Cognitive effort features associated with reading eye movement patterns were calculated for each reading instance during the search tasks. The results show correlations between the cognitive effort due to reading and an individual’s level of domain knowledge. We construct exploratory regression models that suggest it is possible to build models that can make predictions of the user’s level of knowledge based on real-time measurements of eye movement patterns during a task session

Real-time inference of word relevance from electroencephalogram and eye gaze

Objective. Brain-computer interfaces can potentially map the subjective relevance of the visual surroundings, based on neural activity and eye movements, in order to infer the interest of a person in real-time. Approach. Readers looked for words belonging to one out of five semantic categories, while a stream of words passed at different locations on the screen. It was estimated in real-time which words and thus which semantic category interested each reader based on the electroencephalogram (EEG) and the eye gaze. Main results. Words that were subjectively relevant could be decoded online from the signals. The estimation resulted in an average rank of 1.62 for the category of interest among the five categories after a hundred words had been read. Significance. It was demonstrated that the interest of a reader can be inferred online from EEG and eye tracking signals, which can potentially be used in novel types of adaptive software, which enrich the interaction by adding implicit information about the interest of the user to the explicit interaction. The study is characterised by the following novelties. Interpretation with respect to the word meaning was necessary in contrast to the usual practice in brain-computer interfacing where stimulus recognition is sufficient. The typical counting task was avoided because it would not be sensible for implicit relevance detection. Several words were displayed at the same time, in contrast to the typical sequences of single stimuli. Neural activity was related with eye tracking to the words, which were scanned without restrictions on the eye movements.EC/FP7/611570/EU/Symbiotic Mind Computer Interaction for Information Seeking/MindSe

Relevance Prediction from Eye-movements Using Semi-interpretable Convolutional Neural Networks

We propose an image-classification method to predict the perceived-relevance of text documents from eye-movements. An eye-tracking study was conducted where participants read short news articles, and rated them as relevant or irrelevant for answering a trigger question. We encode participants' eye-movement scanpaths as images, and then train a convolutional neural network classifier using these scanpath images. The trained classifier is used to predict participants' perceived-relevance of news articles from the corresponding scanpath images. This method is content-independent, as the classifier does not require knowledge of the screen-content, or the user's information-task. Even with little data, the image classifier can predict perceived-relevance with up to 80% accuracy. When compared to similar eye-tracking studies from the literature, this scanpath image classification method outperforms previously reported metrics by appreciable margins. We also attempt to interpret how the image classifier differentiates between scanpaths on relevant and irrelevant documents

Las instrucciones de relevancia en combinación con la interrogación elaborativa facilitan la lectura estratégica: evidencias desde los movimientos oculares

The aim of the present study was to examine effects of relevance instructions and elaborative interrogation on the processing of and memory for expository texts. Eye movements of 132 undergraduate students were tracked while they read expository texts. After reading each text, they produced an oral summary. Participants were divided into four experimental conditions that differed by the presence or absence of the why question and the specific or general relevance instruction they received. Results showed that readers who received the why question embedded in the texts and also received the specific instruction of answering the question demonstrated more strategic reading, as reflected in their first-pass and look-back reading times and also in their better recall of question-relevant information. These results can be readily applied to real-life learning contexts, as they suggest that employing specific relevance instructions in combination with elaborative interrogation may elicit more efficient and strategic reading.El objetivo del presente estudio fue examinar el efecto de las Instrucciones de relevancia y de Interrogación elaborativa en el procesamiento y el posterior recuerdo de textos expositivos. A tal fin se registraron los movimientos oculares de 132 estudiantes universitarios mientras leían textos expositivos. Después de leer cada texto, realizaron un resumen oral del mismo. Se asignó a los participantes a cuatro condiciones experimentales que se diferenciaban por la presencia o ausencia de una pregunta de tipo “por qué” y la instrucción específica o general de relevancia que recibieron. Los resultados mostraron que los lectores que recibieron la pregunta de tipo “por qué” insertada en los textos y que también recibieron la instrucción específica de contestar a la pregunta mostraron unos patrones de lectura más estratégicos, como quedó reflejado en sus tiempos de lectura inicial y de refijaciones y también en su mejor recuerdo de la información relacionada con la pregunta. Estos resultados pueden aplicarse fácilmente a contextos reales de aprendizaje, ya que sugieren que emplear instrucciones específicas de relevancia en combinación con interrogación elaborativa puede facilitar y potenciar estilos de lectura más eficientes y estratégicosThe work reported in this manuscript was supported by Grant PSI2013-47219-P from the Ministry of Economic and Competitiveness (MINECO) of Spai

Individual differences in processing written irony

Theoretical accounts of irony comprehension assume that when an ironic utterance is unfamiliar and the context does not prime for ironic interpretation, processing should take longer than when reading the same utterance with a literal meaning. This slowdown reflects problems in integrating the utterance into the developing text representation, which results in a reanalysis of the utterance. Similar assumptions are made about other forms of figurative language, such as metaphors, although studies have shown that there are differences in the cognitive demands of different forms of figurative language. Until fairly recently, most of the studies have ignored possible individual differences in irony comprehension among healthy adults. Recent results have suggested that there might be individual differences in irony comprehension related to working memory capacity (WMC) and emotion processing. In the present thesis, I wanted to answer the following questions: 1) How do readers resolve the meaning of irony? 2) How do individual differences in WMC and the ability to process emotional information affect the processing of irony? and 3) Does the processing of irony differ from the processing of other forms of figurative language, namely metaphors? These questions were examined in four studies using eye-tracking to tap into the detailed time-course of resolving the meaning of irony. The results of these studies showed that readers need to reprocess the ironic utterance to achieve the intended meaning, as suggested by the theories on irony comprehension. WMC aids this process by helping readers to keep contextual information in their mind while they integrate the meaning of the utterance with the context and/or inhibit a more salient literal interpretation while making the inference. Emotion processing abilities help to recognize the emotional cues of irony; readers with a poorer ability to process emotional information need to rely more on textual context to resolve the ironic meaning. Finally, resolving the ironic meaning differs from resolving other forms of figurative language, namely metaphors. Metaphors are easier to comprehend, and the processing of the intended meaning of metaphors starts at an earlier stage of reading. Moreover, emotion processing abilities are related to the processing of irony, but not metaphors. Based on the findings of this thesis, I present a new theoretical framework, the Cumulative Evidence Model.Yksilölliset erot kirjoitetun ironian prosessoinnissa Ironian ymmärtämisen teorioiden mukaan ironisen lausahduksen merkityksen ratkaiseminen kestää pidempään kuin saman lausahduksen prosessointi kirjaimellisessa merkityksessä, mikäli lausahdus ei ole entuudestaan tuttu ja jos sitä ei ole voinut ennakoida. Tämän hidastumisen ajatellaan heijastavan virkkeen uuden merkityksen liittämistä aiempaan tekstikontekstiin, mikä edellyttää virkkeen uudelleen prosessointia. Muiden kuvaannollisen kielen muotojen, kuten metaforien, merkityksen ymmärtämisen on oletettu tapahtuvan samankaltaisen prosessin kautta. Lisäksi pitkään on oletettu, että terveet aikuiset ymmärtävät ironiaa samalla tavoin. Viimeaikaiset tutkimukset ovat osoittaneet, että eri kuvaannollisen kielen muotojen ymmärtäminen edellyttää lukijalta erilaisia kognitiivisia taitoja. Lisäksi viimeaikaiset tutkimustulokset antavat viitteitä siitä, että ironian ymmärtämisessä on yksilöllisiä eroja, jotka voisivat liittyä työmuistin kapasiteettiin ja emotionaalisen tiedon prosessointikykyyn. Tässä tutkielmassa halusin selvittää: 1) kuinka lukijat ratkaisevat ironisen lausahduksen merkityksen, 2) kuinka yksilölliset erot työmuistin kapasiteetissa ja emotionaalisen informaation prosessoinnissa vaikuttavat ironian prosessointiin ja 3) kuinka ironian prosessointi eroaa muiden kuvaannollisen kielen muotojen, kuten metaforien, prosessoinnista. Näitä tutkimuskysymyksiä selvitettiin neljässä osatutkimuksessa silmänliikemenetelmän avulla. Menetelmän avulla on mahdollista seurata reaaliajassa ymmärtämisprosessin etenemistä. Tutkielman tulokset osoittavat, että lukijat prosessoivat ironisen virkkeen uudelleen ymmärtääkseen sen tarkoitetun merkityksen. Työmuistin kapasiteetti auttaa tässä prosessissa joko auttamalla lukijoita pitämään mielessään konteksti-informaatiota, jotta he voivat integroida uuden merkityksen siihen, tai auttamalla lukijoita torjumaan ironisen lausahduksen kirjaimellisen merkityksen tulkintaprosessin aikana. Emotionaalisen informaation prosessointikyky auttaa lukijaa tunnistamaan ironisen kommentin emotionaalisen viestin; ne lukijat joiden emootioiden prosessointikyky on heikko, joutuvat tulkintaa tehdessään nojaamaan enemmän kontekstissa esitettyihin vihjeisiin. Tutkimuksen tulokset osoittavat myös, että ironian prosessointi eroaa metaforien prosessoinnista. Metaforat ovat helpompia ymmärtää ja niiden merkityksen prosessointi alkaa varhaisemmassa vaiheessa. Lisäksi emotionaalisen informaation prosessointikyky liittyy vain ironian prosessointiin. Tämän tutkielman löydöksiin perustuen esitän uuden teoreettisen viitekehyksen ironian ymmärtämiseen, kumuloituvien todisteiden mallin

Exploring the applicability of implicit relevance measures in varying reading speed for adaptive I.R. systems

Projecte realitzat en el marc d’un programa de mobilitat amb la University of Helsinki. Faculty of Science. Department of Computer ScienceThis thesis goes further in the study of implicit indicators used to infer interest in documents for information retrieval tasks. We study the behavior of two different categories of implicit indicators: fixation-derived features and physiology (pupil size, electrodermal activity). Based on the limited number of participants at our disposal we study how these measures react when addressing documents at three different reading rates. Most of the fixation-derived features are reported to differ significantly when reading at different speeds. Furthermore, the ability of pupil size and electrodermal activity to indicate perceived relevance is found intrinsically dependent on speed of reading. That is, when users read at comfortable reading speed, these measures are found to be able to correctly discriminate relevance judgments, but fail when increasing the addressed speed of reading. Therefore, the outcomes of this thesis strongly suggest to take into account reading speed when designing highly adaptive information retrieval systems

Factuality Checking in News Headlines with Eye Tracking

We study whether it is possible to infer if a news headline is true or false using only the movement of the human eyes when reading news headlines. Our study with 55 participants who are eye-tracked when reading 108 news headlines (72 true, 36 false) shows that false headlines receive statistically significantly less visual attention than true headlines. We further build an ensemble learner that predicts news headline factuality using only eye-tracking measurements. Our model yields a mean AUC of 0.688 and is better at detecting false than true headlines. Through a model analysis, we find that eye-tracking 25 users when reading 3-6 headlines is sufficient for our ensemble learner.Comment: Accepted to SIGIR 202