Towards Active Vision in the DUAL Cognitive Architecture

The paper describes an extension of the cognitive architecture DUAL with a model of visual attention and perception. The goal of this attempt is to account for the construction and the categorization of object and scene representations derived from visual stimuli in the TextWorld microdomain. Low-level parallel computations are combined with an active serial deployment of visual attention enabling the construction of abstract symbolic representations. A limited-capacity short-term visual store holding information across attention shifts forms the core of the model interfacing between the low-level representation of the stimulus and DUAL’s semantic memory. The model is validated by comparing the results of a simulation with real data from an eye movement experiment with human subjects

Visual attention method based on vertex ranking of graphs by heterogeneous image attributes

В статье рассматривается разработка метода визуального внимания на основе ранжирования вершин графа по разнородным признакам изображений. Целью исследований является создание метода, позволяющего с высокой точностью обнаруживать объекты на изображениях с низким цветовым контрастом выделяемых и фоновых областей. Для вычисления области значимости изображение предварительно сегментируется на регионы. На основе регионов строится граф. Каждый регион связан со смежными регионами, а также с областями, примыкающими к смежным регионам. Регионы являются вершинами графа. Вершины графа ранжируются по признакам соответствующих областей изображения. Область значимости выделяется на основе запросов фоновых областей. К фоновым областям относятся регионы, примыкающие к краям изображения. В существующем подходе визуального внимания на основе ранжирования вершин графа использовались только цветовые признаки изображения. В предлагаемом методе для повышения точности дополнительно используются текстурные признаки и признаки формы. Для вычисления текстурных признаков используется функция энергии Габора. При анализе формы рассчитывается расстояние между центрами регионов. Результаты экспериментов представлены на тестовых изображениях. Построены кривые точности-полноты, показывающие преимущество разработанного метода.Работа выполнена при финансовой поддержке Министерства науки и высшего образования РФ (Госзадание ВлГУ ГБ-1187/20)

The Emergence of Adaptive Eye Movements in Reading

Abstract Simulations were completed using artificial reading "agents" that are subject to known physiological (e.g., limited visual acuity) and psychological (e.g., limited attention) constraints and capable of learning to move their eyes and allocate attention to read as efficiently as possible. These simulations indicate that agents learn when and where to move their eyes to attain maximal reading efficiency, generalize this behavior from training sentences to novel test sentences, and use word length to predict word-identification times and thereby make optimal decisions about when to initiate saccadic programming-even if word length is only moderately predictive of word-identification times. These results suggest that humans may exploit even modestly informative cues in learning to decide when to move their eyes during reading

Characteristics of Eye Movements and Models of Reading Based on the Use of Eye-Tracking

U posljednjih su nekoliko desetljeća razvijene brojne metode s ciljem rasvjetljavanja mehanizama jezične obrade. Ugrubo se mogu podijeliti na metode koje se provode u odgođenom i u stvarnom vremenu (u ovome će se radu koristiti izvorni engleski izrazi off-line i on-line), pri čemu ih razlikuje način primjene te vrsta i obilježja podataka koje pružaju. Metoda praćenja pokreta očiju (engl. eye-tracking) danas se smatra jednom od najboljih metoda za proučavanje jezične obrade. Ona daje uvid u procese kognitivne obrade u stvarnom vremenu, a za razliku od off-line, ali i nekih on-line metoda, npr. metode mjerenja evociranih potencijala ili funkcionalne magnetske rezonancije, naročito je pogodna za istraživanje procesa koji se protežu u vremenu. Budući da je utemeljena na pretpostavkama koje povezuju fiziološku razinu kontrole pokreta oka s kognitivnim procesima koji su u pozadini, a pritom je i neinvazivna, danas je ovo sve zastupljenija metoda u proučavanju procesa vezanih za jezičnu obradu. Glavni je cilj ovog rada dati kratki pregled metoda istraživanja jezične obrade s posebnim naglaskom na on-line metodu praćenja pokreta očiju, predstaviti obilježja pokreta očiju tijekom čitanja te pružiti prikaz nekih od danas najaktualnijih modela čitanja. Iako se jezična obrada može proučavati i putem čitanja i putem slušanja, za potrebe ovog rada naglasak je stavljen na paradigmu čitanja.Over the last few decades many methods for studying the underlying mechanisms of language processing have been developed. They can roughly be divided into off-line and on-line methods, and differences between them involve how they are carried out, as well as the type and characteristics of data they provide. The eye-tracking method is considered one of the best methods for studying language processing. It provides insights into cognitive processes in real time. As opposed to off-line, and also some on-line methods, i.e. cognitive evoked potentials or functional magnetic resonance, it is especially suitable for studying processes that spread over time. Considering the fact that it is based on assumptions that connect the physiological level of eye movement control with the underlying cognitive processes, and is also non invasive, it is becoming more and more present when it comes to studying language processing. The main aim of this paper was to provide a brief overview of the methods for studying language processing with special emphasis on the on-line eye-tracking method, to introduce the characteristics of eye movements during reading and to present some of the most influential models of reading. Even though language processing in eye-tracking can be studied through either reading or listening, for the purpose of this paper the emphasis has been put on the reading paradigm

MARLUI: Multi-Agent Reinforcement Learning for Adaptive UIs

Adaptive user interfaces (UIs) automatically change an interface to better support users' tasks. Recently, machine learning techniques have enabled the transition to more powerful and complex adaptive UIs. However, a core challenge for adaptive user interfaces is the reliance on high-quality user data that has to be collected offline for each task. We formulate UI adaptation as a multi-agent reinforcement learning problem to overcome this challenge. In our formulation, a user agent mimics a real user and learns to interact with a UI. Simultaneously, an interface agent learns UI adaptations to maximize the user agent's performance. The interface agent learns the task structure from the user agent's behavior and, based on that, can support the user agent in completing its task. Our method produces adaptation policies that are learned in simulation only and, therefore, does not need real user data. Our experiments show that learned policies generalize to real users and achieve on par performance with data-driven supervised learning baselines

SEAM: An Integrated Activation-Coupled Model of Sentence Processing and Eye Movements in Reading

Models of eye-movement control during reading, developed largely within psychology, usually focus on visual, attentional, lexical, and motor processes but neglect post-lexical language processing; by contrast, models of sentence comprehension processes, developed largely within psycholinguistics, generally focus only on post-lexical language processes. We present a model that combines these two research threads, by integrating eye-movement control and sentence processing. Developing such an integrated model is extremely challenging and computationally demanding, but such an integration is an important step toward complete mathematical models of natural language comprehension in reading. We combine the SWIFT model of eye-movement control (Seelig et al., 2020, doi:10.1016/j.jmp.2019.102313) with key components of the Lewis and Vasishth sentence processing model (Lewis & Vasishth, 2005, doi:10.1207/s15516709cog0000_25). This integration becomes possible, for the first time, due in part to recent advances in successful parameter identification in dynamical models, which allows us to investigate profile log-likelihoods for individual model parameters. We present a fully implemented proof-of-concept model demonstrating how such an integrated model can be achieved; our approach includes Bayesian model inference with Markov Chain Monte Carlo (MCMC) sampling as a key computational tool. The integrated model, SEAM, can successfully reproduce eye movement patterns that arise due to similarity-based interference in reading. To our knowledge, this is the first-ever integration of a complete process model of eye-movement control with linguistic dependency completion processes in sentence comprehension. In future work, this proof of concept model will need to be evaluated using a comprehensive set of benchmark data

Tracking the mind during reading via eye movements: Comments on Kliegl, Nuthmann, and Engbert (2006)

Kliegl, Nuthmann, and Engbert (2006) reported an impressive set of data analyses dealing with the influence of the prior, present, and next word on the duration of the current eye fixation during reading. They argued that outcomes of their regression analyses indicate that lexical processing is distributed across a number of words during reading. In this comment, we question their conclusions and address four different issues: (1) whether there is evidence for distributed lexical processing, (2) whether so-called parafoveal-on-foveal effects are widespread, (3) the role of correlational analyses in reading research, and (4) problems in their analyses with only using cases where words are fixated exactly once

Linguistic processes do not beat visuo-motor constraints, but they modulate where the eyes move regardless of word boundaries: Evidence against top-down word-based eye-movement control during reading

International audienceWhere readers move their eyes, while proceeding forward along lines of text, has long been assumed to be determined in a top-down word-based manner. According to this classical view, readers of alphabetic languages would invariably program their saccades towards the center of peripheral target words, as selected based on the (expected) needs of ongoing (word-identification) processing, and the variability in within-word landing positions would exclusively result from systematic and random errors. Here we put this predominant hypothesis to a strong test by estimating the respective influences of language-related variables (word frequency and word predictability) and lower-level visuo-motor factors (word length and saccadic launch-site distance to the beginning of words) on both word-skipping likelihood and within-word landing positions. Our eye-movement data were collected while forty participants read 316 pairs of sentences, that differed only by one word, the prime; this was either semantically related or unrelated to a following test word of variable frequency and length. We found that low-level visuo-motor variables largely predominated in determining which word would be fixated next, and where in a word the eye would land. In comparison, language-related variables only had tiny influences. Yet, linguistic variables affected both the likelihood of word skipping and within-word initial landing positions, all depending on the words’ length and how far on average the eye landed from the word boundaries, but pending the word could benefit from peripheral preview. These findings provide a strong case against the predominant word-based account of eye-movement guidance during reading, by showing that saccades are primarily driven by low-level visuo-motor processes, regardless of word boundaries, while being overall subject to subtle, one-off, language-based modulations. Our results also suggest that overall distributions of saccades’ landing positions, instead of truncated within-word landing-site distributions, should be used for a better understanding of eye-movement guidance during reading