Extracting and Visualizing Data from Mobile and Static Eye Trackers in R and Matlab

Eye tracking is the process of measuring where people are looking at with an eye tracker device. Eye tracking has been used in many scientific fields, such as education, usability research, sports, psychology, and marketing. Eye tracking data are often obtained from a static eye tracker or are manually extracted from a mobile eye tracker. Visualization usually plays an important role in the analysis of eye tracking data. So far, there existed no software package that contains a whole collection of eye tracking data processing and visualization tools. In this dissertation, we review the eye tracking technology, the eye tracking techniques, the existing software related to eye tracking, and the research on eye tracking for posters and related media. We then discuss the three main goals we have achieved in this dissertation: (i) development of a Matlab toolbox for automatically extracting mobile eye tracking data; (ii) development of the linked microposter plots family as new means for the visualization of eye tracking data; (iii) development of an R package for automatically extracting and visualizing data from mobile and static eye trackers

Visual communication of technology: its impact on designing and innovation in industrial and engineering design education

Visual communication (VC) resources can be seen as playing an increasingly important role in delivery and learning systems in today s design and technology education. The performance of current tools and resources is the primary concern of this research, and particularly whether they take full advantage of VC when delivering technological information to industrial design (ID) and engineering design (ED) students. This thesis sought key principles behind the visual communication of technology (VCT) and its association to designing, creativity and innovation through a literature survey. The findings concluded that there were many such assertions made with little evidence concerning the associations suggested. Some guiding sources and key emerging principles (KEPs) for good VCT practices were established. A miniature-kite-designing exercise was conducted as a case study for the purpose of examining the links between VCT, designing and creativity and/or innovation. Kite-technological-information posters were used as the VCT tool for the kite-designing case. A comparative study of kite-designing was conducted in Malaysia to check the reliability of the study, and another validation study was carried out for the purpose of establishing the validity of the data gathering. Visual technological information (VTI) for kite design (or a kite-poster) was refined accordingly to the KEPs established from the literature review, and its visual impact was tested through the use of eye-tracking technology. Some selected current and historical visual tools, which have been used in design and technology communication and were recognised as having positive impacts were analysed and articulated in order to reveal a deeper understanding of the KEPs. These were further validated through eye-tracking of reading patterns of participants on those selected visuals. The perceptual responses toward those visuals were also recorded and analysed. A theoretical research framework was established to investigate VTI representation used in books by Ashby (1999) and Ashby and Johnson (2002), in new authors scholarly papers (METU, 2010), and of the author s analysis and redesign of some of those studied VTIs based on the KEPs emerging from the research. A questionnaire survey was conducted within a number of higher education institutions in 3 regions around the world in order to achieve reliable data gathering. This third case study was validated through experts discussion of the findings and related issues. Within these three case studies, a mixture of scientific (using the eye-tracker device) and conventional methods (questionnaires, interviews, discussion group and comparative studies), and also others methods such as design workshops, analysing existing resources, using own practice of design-and-redesign activities were conducted to provide quantitative and qualitative measurements to empirically validate the literature search. Evidence of links between VCT, designerly activities which involved knowledge, skills and values within the technological communication, and of facilitating creativity was obtained. Empirical evidence showed that VTIs were effective in communicating knowledge, skills and values; where the KEPs criteria had played essential roles in enriching the visual emphasis of VTIs. The redesigning exercise using the author s own practice, which articulated the KEPs through the redesign of the existing VTIs for the purpose of more effective VCT, again obtained significant evidence of visual effectiveness and easy understanding capability. Evidence from the analysis of 2 books on materials technology for ID and ED students, views from the 2 materials experts, and the literature review suggested that ID and ED students require difference types of representational models and graphical strategies of VCT in their learning. However, the empirical data from the research, which was supported by one of the materials experts, suggested that ID and ED students even with different cultural backgrounds did not require different VTIs or the use of different VCT strategies for effective communication

IDATER online conference: graphicacy and modelling 2010

IDATER online conference: graphicacy and modelling 201

Understanding search behaviour on mobile devices

Web search on hand-held devices has become enormously common and popular. Although a number of studies have revealed how users interact with search engine result pages (SERPs) on desktop monitors, there are still only few studies related to user interaction in mobile web search, and search results are shown in a similar way whether on a mobile phone or a desktop. Therefore, it is still difficult to know what happens between users and SERPs while searching on small screens, and this means that the current presentation of SERPs on mobile devices may not be the best. According to the findings from previous studies, including our earlier work, we can confirm that search behaviour on touch-enabled mobile devices is different from behaviour with desktop screens, and so we need to consider a different SERP presentation design for mobile devices. In this thesis, we explore several user interactions during search with the aim of improving search experience on smartphones. First, one remarkable trend of mobile devices is their enlargement of screen sizes during the last few years. This leads us to look for differences in search behaviour on different sized small screens, and if there are any, to suggest better presentation of search results for each screen size. In the first study, we investigated search performance, behaviour, and user satisfaction on three small screens (3.6 inches for early smartphones, 4.7 inches for recent smart-phones and 5.5 inches for phablets). We found no significant differences with respect to the efficiency of carrying out tasks. However, participants exhibited different search behaviours on the small, medium, and large sizes of small screens, respectively: a higher chance of scrolling with the worst user satisfaction on the smallest screen; fast information extraction with some hesitation before selecting a link on the medium screen; and less eye movements on top links on the largest screen. These results suggest that the presentation of web search results for each screen size needs to take into account differences in search behaviour. Second, although people are familiar with turning pages horizontally while reading books, vertical scrolling is the standard option that people have available while searching on mobile devices. So following a suggestion from the first study, in the second study we explored the effect of horizontal and vertical viewport control types (pagination versus scrolling) with various positions of a correct answer in mobile web search. Our findings suggest that although users are more familiar with scrolling, participants spent less time to find the correct answer with pagination, especially when the relevant result is located beyond the page fold. In addition, participants using scrolling exhibited less interest in lower-ranked results even if the documents were relevant. The overall result indicates that it is worthwhile providing different viewport controls for better search experiences in mobile web search. Third, snippets occupy the biggest space in each search result. Results from a previous study suggested that snippet length affects search performance on a desktop monitor. Due to the smaller screen, the effect seems to be much larger on smartphones. As one possible idea for a SERP presentation design from the first study, we investigated appropriate snippet lengths on mobile devices in the third study. We compared search behaviour with three different snippet lengths, that is, one line, two to three lines, and six or more lines of snippets on mobile SERPs. We found that with long snippets, participants needed longer search time for a particular task type, and the longer time consumption provided no better search accuracy. Our findings suggest that this search performance is related to viewport movements and user attention. We expect that our proposed approaches provide ways to understand mobile web search behaviour, and that the findings can be applied to a wide range of research areas such as human-computer integration, information retrieval, and even social science for a better presentation design of SERP on mobile devices

The Artful Eye: Exploring Visual Engagement with Artworks in Different Contexts

Artworks are increasingly experienced in non-traditional platforms, from digital collections on museum websites to virtual gallery tours, making it important to investigate the context-dependent and context-independent aspects of aesthetic experience. While some studies have shown that artworks in the museum elicit a higher visual engagement than when presented on a screen, others reported divergent findings. This thesis suggests that such discrepancies may be due to the interaction between the artwork's physical and contextual characteristics and investigates how diverse aspects of viewing behaviour change between the museum, on-screen laboratory, and virtual gallery laboratory contexts. Fifteen paintings by different Australian artists from the Art Gallery of New South Wales (AGNSW) were included as stimuli for the studies in this thesis. Mobile and screen-based eye movement recordings were used to index visual engagement (number of fixations, total and average fixation duration) with artworks across the three different contexts. Our first study (Chapter 2) compared the visual engagement of museum visitors in the AGNSW to that of participants looking at their digital reproductions in laboratory. We focused on how aspects of viewing behaviour, including viewing distance in the gallery condition and eye gaze measures such as fixation count, total fixation duration and average fixation duration are affected by the artworks’ physical characteristics, including size and image statistics properties such as Fourier amplitude spectrum, fractal dimension and entropy. The effects of these factors on visual engagement were then explored in a virtual gallery replica of the exhibition (Chapter 3). In a virtual gallery context, we also tested the impact of two additional context-dependent factors: the curatorial arrangement and further manipulations of the relative size of the paintings. Overall, the results show significant differences in viewing behaviour across different contexts, but also that the effects of presentation contexts are modulated by the artworks’ physical characteristics. In the final two studies, the thesis explores the effect of mere exposure on viewing behaviour in different contexts (Chapter 4) and the spatial and temporal image statistics of fixated compared to non-fixated regions of artworks in both the museum and on- screen viewing contexts (Chapter 5). The results show that visual engagement in the museum, but not on-screen, is enhanced by previous exposure to digital reproductions of artworks. Finally, Chapter 5 demonstrates that fixated and randomly selected regions differed in both spatial and temporal image statistics with more pronounced differences in the on-screen viewing condition. In sum, the thesis demonstrates that a combination of context-dependent variables (e.g., navigation, curatorial setting and relative size) and the low-level properties (e.g., fractal dimension, amplitude spectrum, entropy) of artworks influence visual engagement

Human Machine Interfaces for Teleoperators and Virtual Environments

In Mar. 1990, a meeting organized around the general theme of teleoperation research into virtual environment display technology was conducted. This is a collection of conference-related fragments that will give a glimpse of the potential of the following fields and how they interplay: sensorimotor performance; human-machine interfaces; teleoperation; virtual environments; performance measurement and evaluation methods; and design principles and predictive models

Paying attention to the evidence: a comparison of perception and decision making processes in novice and experienced scene of crime officers using eye tracking in simulated crime scene scenarios

Research on crime scene investigation has strongly focused on the technical aspects of the process, while cognitive aspects (searching, reasoning and perception) have often been overlooked. Textbooks on forensic sciences tend to focus on identifying and processing evidence, and the use of equipment while it can be argued that cognitive factors in processing such evidence and using equipment are equally important. This thesis studies the cognitive aspects of crime scene investigation by comparing eye movement patterns in experts and novices. Studies in various domains, including surgery, sports, and chess playing have shown that eye movements differ between experts and novices, providing a tool towards a more objective assessment of skill than is possible with peer assessment. In four experiments eye movements of experts and novices were examined during (1) inspection of photographs of crime scenes on a computer screen (2) a change blindness task on crime and non-crime scene images, (3) active exploration of a simulated crime scene and (4) the assessment of emotional crime and natural scenes. While some trends in eye movement differences, such as a tendency on longer fixation durations and a broader focus on the overall scene and less on the direct evidence could be found in experts compared to novices, differences between experts and novices were considerably smaller than in other domains, despite the broad range of measures extracted from the data. This lack of clear expertise effects may relate to the rather diverse range of perceptual layouts of crime scenes, reducing possible top-down effects of expertise on the deployment of attention. The results will be discussed with a view of possible directions of future research in this domain

Investigating non-visual eye movements non-intrusively: Comparing manual and automatic annotation styles

Non-visual eye-movements (NVEMs) are eye movements that do not serve the provision of visual information. As of yet, their cognitive origins and meaning remain under-explored in eye-movement research. The first problem presenting itself in pursuit of their study is one of annotation: in virtue of their being non-visual, they are not necessarily bound to a specific surface or object of interest, rendering conventional eye-trackers nonideal for their study. This, however, makes it potentially viable to investigate them without requiring high resolution data. In this report, we present two approaches to annotating NVEM data – one of them grid-based, involving manual annotation in ELAN (Max Planck Institute for Psycholinguistics: The Language Archive, 2019), the other one Cartesian coordinate-based, derived algorithmically through OpenFace (Baltrušaitis et al., 2018). We evaluated a) the two approaches in themselves, e.g. in terms of consistency, as well as b) their compatibility, i.e. the possibilities of mapping one to the other. In the case of a), we found good overall consistency in both approaches, in the case of b), there is evidence for the eventual possibility of mapping the OpenFace gaze estimations onto the manual coding grid