This on-going research paper explores (the possibilities to integrate eye tracking (ET) and electroencephalogram (EEG) for cartographic usability research. While ET, on one hand, provides observations and measurements related to gaze movements, EEG, on the other hand, helps to monitor and measure electrical activity occurring at different locations in the brain with a high temporal resolution. Therefore, combining ET and EEG introduces a holistic approach enabling to measure both overt and covert attention, and additionally, may reveal insights on individual’s different strategies of spatial cognition, if there is any. In this context, we introduce the experimental design settings for visual search task on simplified 2D static maps considering expert and novice participants, outlining methodological proposal and possible analyses. The paper mainly discusses the technical and theoretical issues of ET-EEG integration and mentions potential benefits of implementing EEG in cartographic usability research to indicate its value for future studies

Keskin, Merve

Ooms, Kristien

Ghent University Academic Bibliography

Possibilities of eye tracking and EEG integra-tion for visual search on 2D maps Merve Keskin*, Kristien Ooms* * Geography Department of Ghent University, Ghent, Belgium, merve.keskin@ugent.be, kristien.ooms@ugent.be  Abstract. This on-going research paper explores (the possibilities to inte-grate eye tracking (ET) and electroencephalogram (EEG) for cartographic usability research. While ET, on one hand, provides observations and measurements related to gaze movements, EEG, on the other hand, helps to monitor and measure electrical activity occurring at different locations in the brain with a high temporal resolution. Therefore, combining ET and EEG introduces a holistic approach enabling to measure both overt and covert attention, and additionally, may reveal insights on individual’s dif-ferent strategies of spatial cognition, if there is any. In this context, we in-troduce the experimental design settings for visual search task on simplified 2D static maps considering expert and novice participants, outlining meth-odological proposal and possible analyses. The paper mainly discusses the technical and theoretical issues of ET-EEG integration and mentions poten-tial benefits of implementing EEG in cartographic usability research to in-dicate its value for future studies.  Keywords. eye tracking, EEG, visual search, usability, cognitive cartog-raphy   1. Introduction It is important to choose the right methods to measure overt and covert attention those unveiling task-related user behaviors. On one hand, eye tracking provides valuable insights of cognitive processes related to overt attention by making sense of fixations and saccadic eye movements. It has been applied in many cartographic usability researches as a standalone technique, or combined with other methods such as think-aloud, question-naire, and interview (e.g. Çöltekin et al., 2010; Elzakker, et al,. 2008; Fabri-kant, et al., 2010; Ooms, 2012;). On the other hand, brain imaging tech-niques like electroencephalogram (EEG) can be used to measure covert at-tention (Treder, 2011; Kulke, 2016) and possibly to obtain additional in-sights (i.e. cognitive load, timing of brain processes, etc.) while working with cartographic products and related geographic applications.  Although the EEG data is overwhelming by nature and promising in terms of revealing human neural dynamics, it is crucial to set goals and ask re-search questions which could be answered with EEG. For instance, EEG can be employed to measure cognitive load with a high temporal resolution and the multidimensionality of EEG data (space, time, frequency, power and phase) provides highly independent information associated with a large variety of perception, cognition, and actions (Cohen, 2014; Kida et al., 2015). However, it is unlikely to use EEG to answer questions requiring high spatial resolution. The reason is that EEG reflects a mixture of brain activities from multiple areas at various distances from the electrode (see Cohen, 2014).  Although, to our knowledge, there exists only one study that made use of ET and EEG integration for cartographic usability research (Maggi & Fabri-kant, 2014), these two techniques are often combined in medical, experi-mental psychology, marketing, sports, and usability domains. Therefore, we propose to evaluate the possibilities regarding this integration further for its use in usability studies of cartographic products. The motivation behind the ET and EEG combination arises from the need of gaining deeper insight of map user’s search behaviors when they interact with the visual information presented via maps. Our main research ques-tions are as follows: Do search strategies of map-related information differ for experts and novices when working with simple 2D maps? Does the cog-nitive load differ between expert and novices? These research questions will not be answered in this paper, however, they can be in the near future with the proposed methodology and analyses. Therefore, we try to explain how this integration can be achieved, which issues needed to be considered and how we can make sense of collected ET and EEG data. 2. Methodology 2.1. Experimental design and procedure We designed a mixed methods user experiment, integrating ET, EEG and post-test questionnaire. The experiment was carried out with 31 novices who are undergraduate students from Faculty of Economics and Business Administration of Ghent University, and gained credits in return of their participation and 23 experts who have at least a master degree on geogra-phy, GIS or related areas, and are affiliated with Geography department of Ghent University.  Stimuli were shown on a 22” color monitor with 1680 x 1050 resolution and participants were asked to perform a visual search on 2D maps with a sim-ple design and content. In total, they saw three maps, each showing a list of three labels alongside of it, and they had to locate the corresponding labels on the map with a mouse click.   Simultaneously, we recorded participants’ eye movements using a SMI RED 250 eye tracker mounted to the participant monitor, and the brain activity data by using EEG modules of BIOPAC through signals stemmed from 16 electrodes of the 10-20 International system EEG cap. The synchronization was established through TTL (transistor-transistor logic) which is a widely-used technology to make integrated circuits. Finally, participants performed a post-test questionnaire including questions about their age, gender, high-est level of education, and so forth.  2.2. Design issues The design of the experiment kept simple in order to be able to analyze the EEG data. Even within simple maps and tasks, differences between individ-ual performances can be observed. For instance, despite the simple design, Ooms (2012) has already identified differences in the attentive behavior between experts and novices. The other most important design issue is the selection of EEG electrodes to be used. As we intend to find out whether there is something interesting behind this integration, we involved the maximum number of electrodes we can, which was limited to 16 electrodes in our case. Hence, we had to dis-card some of the electrodes. Based on the literature, we know that during visual attention, superior colliculus (center), FEF (Frontal eye field) and LIP (Lateral intraparietal area) (perietal) network play an important role (see Bisley et al, 2011; Esterman, et al., 2015; Krauzlis, et al., 2013). In addi-tion to this, we included the occipital electrodes (responsible for visual pro-cessing) and some temporal lobes (responsible for verbal understanding) (Nolte, 2002).  Furthermore, we imposed a baseline image to observe the participants’ rest-ing period of physiology. The baseline was simply a cross, and participants were instructed to look at it for some seconds. It is useful to remove a mean baseline value from each epoch in the preprocessing stage, if baseline dif-ferences between data epochs are present. 2.3. Preprocessing Merging and aligning the synchronized EEG and ET data is the most chal-lenging step of the integration. It requires data management (i.e. writing scripts to convert EEG, ET and event data in compatible formats), theoreti-cal knowledge and expertise (i.e. assessing the quality of synchronization, deciding on rejecting bad data) and a huge amount of time because each participant data has to be processed separately. Preprocessing is essential to eliminate parts of the EEG record that contains noises originated from varying sources such as eye blinks, heartbeat, and muscle activity or environmental electrical activity (e.g. line–frequency noise from monitors) (Handy, 2005). Hence, filtering EEG data helps re-move high frequency artifacts and low frequency drifts (Cohen, 2014).  Figure 2. An example of a synchronized EEG and ET data We are currently processing the participants’ data using an open source MATLAB toolbox called EEGLAB (Delorme & Makeig, 2004) with the EYEEEG extension (Dimigen, et al., 2011). Figure 2 illustrates a portion of EEG recording merged and aligned with eye movement data (saccades and fixations) through shared events. The horizontal axis represents time (s) and vertical axis shows amplitude (µV) i.e. the amount of energy in fre-quency bands listed on the left-hand side of the graph. Amplitude scale was adjusted such that the EEG waves are clearly visible but not overlap. Events (indicated as 108 in Figure 4) represent time-points when the stimu-lus was shown to the participant. EEG waves on the graph do not reflect the raw EEG data that had high frequency power (line-frequency) noise, most likely at 50 Hz. To reveal the EEG signal hidden in the noise, a notch filter at 50 Hz was applied. On the other hand, thick blue noisy waves belong to a bad channel due to bad contact that can be visually detected and excluded. 2.4. Possible Analyses Due to high temporal resolution of EEG, we will focus on event-related po-tentials (ERP: EEG activity that is time-locked to an event), not the whole EEG recording. Therefore, we plan to apply time frequency-based ap-proaches. Once we determine which brain activity is originated from sac-cadic movements and which from the fixations, that are visual processing related, we can sort EEG data based on saccadic and fixation related com-ponents. Then it will be possible to match ERPs at desired fixations and explore if there are similar patterns for different participants. This will give the opportunity to answer our first research question: Do search strategies of map-related information differ for experts and novices?  EEG is also sensitive to measure attention and cognitive load, which can be derived by analyzing the activity of different frequency bands (i.e. alpha, beta, delta, gamma, and theta). By using a moving window to look at fre-quency, phase and amplitude, it is possible to correlate of perception that is not so time resolved. For instance, we can anticipate the cognitive load when participants think their next eye movement is important by making a synchronization between gamma frequencies in different areas. To do so, we intend to answer our second research question: Does the cognitive load differ between expert and novices? 3. Summary and Outlook We presented our exploratory study on EEG and ET integration for carto-graphic usability research, which has a potential value unveiling some addi-tional insights on different map users’ cognitive behaviors.  In this context, we proposed a methodological approach that may be useful for further user experiments by discussing the issues of ET-EEG integration and mention-ing possible analyses with the synchronized data. Gaining a deeper under-standing on cognitive limitations and abilities of different groups of map users will enable us to design and create more usable, effective, and user-specific cartographic visualizations, which inherently cause less cognitive load. References Bisley, J. W., Mirpour, K., Arcizet, F., & Ong, W. S. (2011). The role of the lateral intraparie-tal area in orienting attention and its implications for visual search. European Journal of Neuroscience, 33(11), 1982-1990. Cohen, M. X. (2014). Analyzing neural time series data: theory and practice. MIT Press. Çöltekin, A., Fabrikant, S. I., & Lacayo, M. (2010). Exploring the efficiency of users' visual analytics strategies based on sequence analysis of eye movement recordings. International Journal of Geographical Information Science, 24(10), 1559-1575. Delorme A. & Makeig S. (2004). EEGLAB: an open source toolbox for analysis of single-trial EEG dynamics. Journal of Neuroscience Methods 134:9-21 Dimigen, O., Sommer, W., Hohlfeld, A., Jacobs, A. M., & Kliegl, R. (2011). Coregistration of eye movements and EEG in natural reading: analyses and review. Journal of Experimental Psychology: General, 140(4), 552. Elzakker, van C.P.J.M., I. Delikostidis & P. van Oosterom. (2008). Field-based Usability Evaluation Methodology for Mobile Geo-applications. The Cartographic Journal, 45(2), 139-149. Esterman, M., Liu, G., Okabe, H., Reagan, A., Thai, M., & DeGutis, J. (2015). Frontal eye field involvement in sustaining visual attention: evidence from transcranial magnetic stimulation. Neuroimage, 111, 542-548. Fabrikant, S. I., Hespanha, S. R., & Hegarty, M. (2010). Cognitively inspired and perceptual-ly salient graphic displays for efficient spatial inference making. Annals of the Association of American Geographers, 100(1), 13-29. Handy, T. C. (Ed.). (2005). Event-related potentials: A methods handbook. MIT press. Ignashchenkova, A., Dicke, P. W., Haarmeier, T., & Thier, P. (2004). Neuron-specific contri-bution of the superior colliculus to overt and covert shifts of attention. Nature neurosci-ence, 7(1), 56. Kida, T., Tanaka, E., & Kakigi, R. (2015). Multi-dimensional dynamics of human electro-magnetic brain activity. Frontiers in human neuroscience, 9. Krauzlis, R. J., Lovejoy, L. P., & Zénon, A. (2013). Superior colliculus and visual spatial at-tention. Annual review of neuroscience, 36, 165-182. Kulke, L. V., Atkinson, J., & Braddick, O. (2016). Neural differences between covert and overt attention studied using EEG with simultaneous remote eye tracking. Frontiers in human neuroscience, 10. Maggi, S., & Fabrikant, S. I. (2014). Triangulating Eye Movement Data of Animated Dis-plays. In ET4S@ GIScience (pp. 27-31). Nolte, J. (2002). The human brain: an introduction to its functional anatomy. Ooms, K. (2012). Maps, how do users see them?: an in depth investigation of the map users' cognitive processes (Doctoral dissertation). Ghent University, Belgium Treder, M. S., Schmidt, N. M., & Blankertz, B. (2011). Gaze-independent brain–computer interfaces based on covert attention and feature attention. Journal of neural engineering, 8(6), 066003 

Possibilities of eye tracking and EEG integration for visual search on 2D maps

Repository for Publications and Research Data

https://biblio.ugent.be/publication/8619106/file/8619110

Possibilities of eye tracking and EEG integration for visual search on 2D maps

Abstract

Similar works

Full text

Available Versions

Ghent University Academic Bibliography

Repository for Publications and Research Data