iMind: Uma ferramenta inteligente para suporte de compreensão de conteúdo

Usually while reading, content comprehension difficulty affects individual performance. Comprehension difficulties, e. g., could lead to a slow learning process, lower work quality, and inefficient decision-making. This thesis introduces an intelligent tool called “iMind” which uses wearable devices (e.g., smartwatches) to evaluate user comprehension difficulties and engagement levels while reading digital content. Comprehension difficulty can occur when there are not enough mental resources available for mental processing. The mental resource for mental processing is the cognitive load (CL). Fluctuations of CL lead to physiological manifestation of the autonomic nervous system (ANS), which can be measured by wearables, like smartwatches. ANS manifestations are, e. g., an increase in heart rate. With low-cost eye trackers, it is possible to correlate content regions to the measurements of ANS manifestation. In this sense, iMind uses a smartwatch and an eye tracker to identify comprehension difficulty at content regions level (where the user is looking). The tool uses machine learning techniques to classify content regions as difficult or non-difficult based on biometric and non-biometric features. The tool classified regions with a 75% accuracy and 80% f-score with Linear regression (LR). With the classified regions, it will be possible, in the future, to create contextual support for the reader in real-time by, e.g., translating the sentences that induced comprehension difficulty.Normalmente durante a leitura, a dificuldade de compreensão pode afetar o desempenho da leitura. A dificuldade de compreensão pode levar a um processo de aprendizagem mais lento, menor qualidade de trabalho ou uma ineficiente tomada de decisão. Esta tese apresenta uma ferramenta inteligente chamada “iMind” que usa dispositivos vestíveis (por exemplo, smartwatches) para avaliar a dificuldade de compreensão do utilizador durante a leitura de conteúdo digital. A dificuldade de compreensão pode ocorrer quando não há recursos mentais disponíveis suficientes para o processamento mental. O recurso usado para o processamento mental é a carga cognitiva (CL). As flutuações de CL levam a manifestações fisiológicas do sistema nervoso autônomo (ANS), manifestações essas, que pode ser medido por dispositivos vestíveis, como smartwatches. As manifestações do ANS são, por exemplo, um aumento da frequência cardíaca. Com eye trackers de baixo custo, é possível correlacionar manifestação do ANS com regiões do texto, por exemplo. Neste sentido, a ferramenta iMind utiliza um smartwatch e um eye tracker para identificar dificuldades de compreensão em regiões de conteúdo (para onde o utilizador está a olhar). Adicionalmente a ferramenta usa técnicas de machine learning para classificar regiões de conteúdo como difíceis ou não difíceis com base em features biométricos e não biométricos. A ferramenta classificou regiões com uma precisão de 75% e f-score de 80% usando regressão linear (LR). Com a classificação das regiões em tempo real, será possível, no futuro, criar suporte contextual para o leitor em tempo real onde, por exemplo, as frases que induzem dificuldade de compreensão são traduzidas

The Empirical Analysis of the Comprehensibility of Process Models created by Process Mining

Companies use process models to specify their operational processes. With the help of process models, the business processes in a company are analysed by process mining techniques to optimise them. The subdiscipline of process discovery identifies the actual state of business processes and enables them to be examined. Various tools and algorithms can be used, which lead to different process visualisations. The type of process visualisation has a major influence on the comprehensibility of process models. The objective of this thesis is to investigate the comprehensibility of process models generated by process mining. For this purpose, an exploratory eye-tracking study is conducted with fifteen participants. The study examines process models from two scenarios - a vaccination process and an insurance process. The corresponding process models are created manually, and event logs are generated from them using self-created applications. These event logs are loaded into the process mining tools Celonis Snap, Disco, ProM, Apromore and PM4Py and process models are generated from them. A selection of the resulting process models is then tested for comprehensibility in the user study. The analysis of variance (ANOVA) shows no significant differences between the different generated process models. Finally, with the Pearson correlation’s help, the participants’ subjective ranking is highly significantly related to the level of acceptability and cognitive load. The correlation between the time spent looking at the process models and the number of correctly answered comprehension questions is interesting. From this correlation, it can be concluded that understanding process models requires a certain amount of time. An astonishing result of the study is that the quality between manually created models and models generated by process mining is similarly high. Despite interesting results, further studies are needed, as the study is confronted with some limitations (particularly the number of participants). The results can be used as a basis for future studies to further explore this field of research

Approaches for Eye-Tracking While Reading

In this thesis, we developed an algorithm to detect the correct line being read by participants. The comparisons of the reading line classification algorithms are demonstrated using eye-tracking data collected from a realistic reading experiment in front of a low-cost desktop-mounted eye-tracker. With the development of eye-tracking techniques, research begins to aim at trying to understand information from the eyes. However, state of the art in eye-tracking applications is affected by a large amount of measurement noise. Even the expensive eye-trackers still suffer significant noise. In addition, the inherent characteristics of gaze movement increase the difficulty of obtaining valuable information from gaze measurements. We first discussed an improved Kalman smoother called slip-Kalman smoother, designed to separate eye-gaze data corresponding to correct text lines and reduce measurement noise. Next, two different classifiers are applied to be trained; one is Gaussian discriminant based while the other is support vector machine based. As a result, our algorithm improved the performance of eye gaze classification in the reading scenario and beat the previous method

An eye tracking based framework for safety improvement of offshore operations

Offshore drilling operations consist of complex and high-risk processes. Lack of situational awareness in drilling operations has become an important human factor issue that causes safety accidents. Prolonged work shifts and fatigue are some of the crucial issues that impact performance. Eye tracking technology can be used to distinguish the degree of awareness or alertness of participants that might be related to fatigue or onsite distractions. Oculomotor activity can be used to obtain visual cues that can quantify the drilling operators’ situational awareness that might enable us to develop warning alarms to alert the driller. Such systems can help reduce accidents and save non-productive time. In this paper, eye movement characteristics were investigated to differentiate the situational awareness between a representative expert and a group of novices using a scenario-based Virtual Reality Drilling Simulator. Significant visual oculomotor activity differences were identified between the expert and the novices that indicate an eye-tracking based system can detect the distraction and alertness exhibited by the workers. Results show promise on developing a framework which implements a real-time eye tracking technology in various drilling operations at drilling rigs and Real Time Operation Centers to improve process safety

Eye Tracking to Support eLearning

Online eLearning environments to support student learning are of growing importance. Students are increasingly turning to online resources for education; sometimes in place of face-to-face tuition. Online eLearning extends teaching and learning from the classroom to a wider audience with different needs, backgrounds, and motivations. The one-size-fits-all approach predominately used is not effective for catering to the needs of all students. An area of the increasing diversity is the linguistic background of readers. More students are reading in their non-native language. It has previously been established that first English language (L1) students read differently to second English language (L2) students. One way of analysing this difference is by tracking the eyes of readers, which is an effective way of investigating the reading process. In this thesis we investigate the question of whether eye tracking can be used to make learning via reading more effective in eLearning environments. This question is approached from two directions; first by investigating how eye tracking can be used to adapt to individual student’s understanding and perceptions of text. The second approach is analysing a cohort’s reading behaviour to provide information to the author of the text and any related comprehension questions regarding their suitability and difficulty. To investigate these questions, two user studies were carried out to collect eye gaze data from both L1 and L2 readers. The first user study focussed on how different presentation methods of text and related questions affected not only comprehension performance but also reading behaviour and student perceptions of performance. The data from this study was used to make predictions of reading comprehension that can be used to make eLearning environments adaptive, in addition to providing implicit feedback about the difficulty of text and questions. In the second study we investigate the effects of text readability and conceptual difficulty on eye gaze, prediction of reading comprehension, and perceptions. This study showed that readability affected the eye gaze of L1 readers and conceptual difficulty affected the eye gaze of L2 readers. The prediction accuracy of comprehension was consequently increased for the L1 group by increased difficulty in readability, whereas increased difficulty in conceptual level corresponded to increased accuracy for the L2 group. Analysis of participants’ perceptions of complexity revealed that readability and conceptual difficulty interact making the two variables hard for the reader to disentangle. Further analysis of participants’ eye gaze revealed that both the predefined and perceived text complexity affected eye gaze. We therefore propose using eye gaze measures to provide feedback about the implicit reading difficulty of texts read. The results from both studies indicate that there is enormous potential in using eye tracking to make learning via reading more effective in eLearning environments. We conclude with a discussion of how these findings can be applied to improve reading within eLearning environments. We propose an adaptive eLearning architecture that dynamically presents text to students and provides information to authors to improve the quality of texts and questions

Eye Tracking: A Perceptual Interface for Content Based Image Retrieval

In this thesis visual search experiments are devised to explore the feasibility of an eye gaze driven search mechanism. The thesis first explores gaze behaviour on images possessing different levels of saliency. Eye behaviour was predominantly attracted by salient locations, but appears to also require frequent reference to non-salient background regions which indicated that information from scan paths might prove useful for image search. The thesis then specifically investigates the benefits of eye tracking as an image retrieval interface in terms of speed relative to selection by mouse, and in terms of the efficiency of eye tracking mechanisms in the task of retrieving target images. Results are analysed using ANOVA and significant findings are discussed. Results show that eye selection was faster than a computer mouse and experience gained during visual tasks carried out using a mouse would benefit users if they were subsequently transferred to an eye tracking system. Results on the image retrieval experiments show that users are able to navigate to a target image within a database confirming the feasibility of an eye gaze driven search mechanism. Additional histogram analysis of the fixations, saccades and pupil diameters in the human eye movement data revealed a new method of extracting intentions from gaze behaviour for image search, of which the user was not aware and promises even quicker search performances. The research has two implications for Content Based Image Retrieval: (i) improvements in query formulation for visual search and (ii) new methods for visual search using attentional weighting. Futhermore it was demonstrated that users are able to find target images at sufficient speeds indicating that pre-attentive activity is playing a role in visual search. A current review of eye tracking technology, current applications, visual perception research, and models of visual attention is discussed. A review of the potential of the technology for commercial exploitation is also presented

What is the influence of genre during the perception of structured text for retrieval and search?

This thesis presents an investigation into the high value of structured text (or form) in the context of genre within Information Retrieval. In particular, how are these structured texts perceived and why are they not more heavily used within Information Retrieval & Search communities? The main motivation is to show the features in which people can exploit genre within Information Search & Retrieval, in particular, categorisation and search tasks. To do this, it was vital to record and analyse how and why this was done during typical tasks. The literature review highlighted two previous studies (Toms & Campbell 1999a; Watt 2009) which have reported pilot studies consisting of genre categorisation and information searching. Both studies and other findings within the literature review inspired the work contained within this thesis. Genre is notoriously hard to define, but a very useful framework of Purpose and Form, developed by Yates & Orlikowski (1992), was utilised to design two user studies for the research reported within the thesis. The two studies consisted of, first, a categorisation task (e-mails), and second, a set of six simulated situations in Wikipedia, both of which collected quantitative data from eye tracking experiments as well as qualitative user data. The results of both studies showed the extent to which the participants utilised the form features of the stimuli presented, in particular, how these were used, which ocular behaviours (skimming or scanning) and actual features were used, and which were the most important. The main contributions to research made by this thesis were, first of all, that the task-based user evaluations employing simulated search scenarios revealed how and why users make decisions while interacting with the textual features of structure and layout within a discourse community, and, secondly, an extensive evaluation of the quantitative data revealed the features that were used by the participants in the user studies and the effects of the interpretation of genre in the search and categorisation process as well as the perceptual processes used in the various communities. This will be of benefit for the re-development of information systems. As far as is known, this is the first detailed and systematic investigation into the types of features, value of form, perception of features, and layout of genre using eye tracking in online communities, such as Wikipedia

On the Influence of Representation Type and Gender on Recognition Tasks of Program Comprehension

RÉSUMÉ L’objectif de la maintenance logicielle est d’améliorer les logiciels existants en préservant leur intégrité. La maintenance peut représenter jusqu’à 60% du budget d’un logiciel. Ainsi, améliorer la maintenabilité des logiciels est bénéfique aussi bien pour les fournisseurs que les utilisateurs de logiciels. Les développeurs de logiciels consacrent un effort considérable à la compréhension des programmes, qui est une étape primordiale à la maintenance logicielle. Nous faisons l’hypothèque que le genre des développeurs de logiciels et le type de représentation peut affecter leur effort et leur efficacité. Ces facteurs doivent être considérés et minutieusement analysés dans la mesure où ils peuvent cacher certains effets significatifs pouvant être identifiés en analysant le processus de compréhension. Dans cette thèse, nous nous inspirons de l’utilisation de l’occulomètre pour l’étude du processus cognitif lors de la résolution des problèmes. Nous avons effectué une étude fonctionnelle pour évaluer tous les travaux de recherche faisant usage de l’occulomètre en génie logiciel. Les résultats obtenus nous ont motivé à utiliser l’occulomètre pour effectuer un ensemble d’études afin analyser l’effet de deux facteurs importants sur la compréhension des programmes : le type de représentation (textuelle ou graphique) et le genre du développeur. Afin de comprendre comment les différents types de représentations et le genre influencent les stratégies de visualisation, nous avons étudié la différence de stratégie entre développeurs. Les résultats obtenus montrent que, comparé à une représentation graphique, la représentation sous forme de texte structuré aide mieux le développeur dans son processus cognitif lors de la compréhension des programmes de petite taille. Ainsi, la représentation textuelle requiert moins de temps et d’effort aux participants. Par contre, la représentation graphique est celle préférée par les développeurs. Nos résultats montrent que la structure topologique de la représentation graphique aide les développeurs à mémoriser l’emplacement des éléments et à retrouver plus rapidement les éléments pertinents comparé à la représentation textuelle. En plus, la structure hiérarchique de la représentation graphique guide les développeurs à suivre une stratégie de visualisation spécifique. Nous avons observé que les femmes et les hommes ont des stratégies de visualisation différentes lors de la lecture du code ou de la mémorisation des noms des identificateurs. Les femmes ont tendance à inspecter minutieusement toutes les options afin de procéder à l’élimination de la mauvaise réponse. Au contraire, les hommes ont tendance à inspecter brièvement certaines réponses. Pendant que les femmes consacrent plus de temps à analyser chaque type d’entité l’un après l’autre, les hommes alternent leur attention entre différents type d’entité.----------ABSTRACT The purpose of software maintenance is to correct and enhance an existing software system while preserving its integrity. Software maintenance can cost more than 60% of the budget of a software system, thus improving the maintainability of software is important for both the software industry and its customers. Program comprehension is the initial step of software maintenance that requires the major amount of maintenance’s time and effort. We conjuncture that developers’ gender and the type of representations that developers utilize to perform program comprehension impact their efficiency and effectiveness. These factors must be considered and carefully studied, because they may hide some significant effects to be found by analyzing the comprehension process. In this dissertation, inspired by the literature on the usefulness of eye-trackers to study the cognitive process involved in problem solving activities, we perform a mapping study and evaluate all research relevant to the use of eye-tracking technique in software engineering. The results motivate us to perform a set of eye-tracking studies to analyze the impact of two important factors on program comprehension: representation type (textual vs. graphical) and developers’ gender. Moreover, we investigate and compare viewing strategies variability amongst developers to understand how the representation type and gender differences influence viewing strategies. Overall, our results indicate that structured text provides more cognitive support for developers while performing program comprehension with small systems compared to a graphical representation. Developers spend less time and effort working with textual representations. However, developers mostly preferred to use graphical representations and our results confirm that the topological structure of graphical representations helps developers to memorize the location of the elements and to find the relevant ones faster in comparison with textual representation. Moreover, the hierarchical structure of the representation guides developers to follow specific viewing strategies while working with representations. Regarding the impact of gender, our results emphasize that male and female developers exploit different viewing strategies while reading source code or recalling the names of identifiers. Female developers seem to carefully weigh all options and rule out wrong answers, while male developers seem to quickly set their minds on some answers and move forward. Moreover, female developers spend more time on each source code entity and analyze it before going to the next one. In contrast, male developers utilize several attention switching strategies between different source code entities

Workload-aware systems and interfaces for cognitive augmentation

In today's society, our cognition is constantly influenced by information intake, attention switching, and task interruptions. This increases the difficulty of a given task, adding to the existing workload and leading to compromised cognitive performances. The human body expresses the use of cognitive resources through physiological responses when confronted with a plethora of cognitive workload. This temporarily mobilizes additional resources to deal with the workload at the cost of accelerated mental exhaustion. We predict that recent developments in physiological sensing will increasingly create user interfaces that are aware of the user’s cognitive capacities, hence able to intervene when high or low states of cognitive workload are detected. In this thesis, we initially focus on determining opportune moments for cognitive assistance. Subsequently, we investigate suitable feedback modalities in a user-centric design process which are desirable for cognitive assistance. We present design requirements for how cognitive augmentation can be achieved using interfaces that sense cognitive workload. We then investigate different physiological sensing modalities to enable suitable real-time assessments of cognitive workload. We provide empirical evidence that the human brain is sensitive to fluctuations in cognitive resting states, hence making cognitive effort measurable. Firstly, we show that electroencephalography is a reliable modality to assess the mental workload generated during the user interface operation. Secondly, we use eye tracking to evaluate changes in eye movements and pupil dilation to quantify different workload states. The combination of machine learning and physiological sensing resulted in suitable real-time assessments of cognitive workload. The use of physiological sensing enables us to derive when cognitive augmentation is suitable. Based on our inquiries, we present applications that regulate cognitive workload in home and work settings. We deployed an assistive system in a field study to investigate the validity of our derived design requirements. Finding that workload is mitigated, we investigated how cognitive workload can be visualized to the user. We present an implementation of a biofeedback visualization that helps to improve the understanding of brain activity. A final study shows how cognitive workload measurements can be used to predict the efficiency of information intake through reading interfaces. Here, we conclude with use cases and applications which benefit from cognitive augmentation. This thesis investigates how assistive systems can be designed to implicitly sense and utilize cognitive workload for input and output. To do so, we measure cognitive workload in real-time by collecting behavioral and physiological data from users and analyze this data to support users through assistive systems that adapt their interface according to the currently measured workload. Our overall goal is to extend new and existing context-aware applications by the factor cognitive workload. We envision Workload-Aware Systems and Workload-Aware Interfaces as an extension in the context-aware paradigm. To this end, we conducted eight research inquiries during this thesis to investigate how to design and create workload-aware systems. Finally, we present our vision of future workload-aware systems and workload-aware interfaces. Due to the scarce availability of open physiological data sets, reference implementations, and methods, previous context-aware systems were limited in their ability to utilize cognitive workload for user interaction. Together with the collected data sets, we expect this thesis to pave the way for methodical and technical tools that integrate workload-awareness as a factor for context-aware systems.Tagtäglich werden unsere kognitiven Fähigkeiten durch die Verarbeitung von unzähligen Informationen in Anspruch genommen. Dies kann die Schwierigkeit einer Aufgabe durch mehr oder weniger Arbeitslast beeinflussen. Der menschliche Körper drückt die Nutzung kognitiver Ressourcen durch physiologische Reaktionen aus, wenn dieser mit kognitiver Arbeitsbelastung konfrontiert oder überfordert wird. Dadurch werden weitere Ressourcen mobilisiert, um die Arbeitsbelastung vorübergehend zu bewältigen. Wir prognostizieren, dass die derzeitige Entwicklung physiologischer Messverfahren kognitive Leistungsmessungen stets möglich machen wird, um die kognitive Arbeitslast des Nutzers jederzeit zu messen. Diese sind in der Lage, einzugreifen wenn eine zu hohe oder zu niedrige kognitive Belastung erkannt wird. Wir konzentrieren uns zunächst auf die Erkennung passender Momente für kognitive Unterstützung welche sich der gegenwärtigen kognitiven Arbeitslast bewusst sind. Anschließend untersuchen wir in einem nutzerzentrierten Designprozess geeignete Feedbackmechanismen, die zur kognitiven Assistenz beitragen. Wir präsentieren Designanforderungen, welche zeigen wie Schnittstellen eine kognitive Augmentierung durch die Messung kognitiver Arbeitslast erreichen können. Anschließend untersuchen wir verschiedene physiologische Messmodalitäten, welche Bewertungen der kognitiven Arbeitsbelastung in Realzeit ermöglichen. Zunächst validieren wir empirisch, dass das menschliche Gehirn auf kognitive Arbeitslast reagiert. Es zeigt sich, dass die Ableitung der kognitiven Arbeitsbelastung über Elektroenzephalographie eine geeignete Methode ist, um den kognitiven Anspruch neuartiger Assistenzsysteme zu evaluieren. Anschließend verwenden wir Eye-Tracking, um Veränderungen in den Augenbewegungen und dem Durchmesser der Pupille unter verschiedenen Intensitäten kognitiver Arbeitslast zu bewerten. Das Anwenden von maschinellem Lernen führt zu zuverlässigen Echtzeit-Bewertungen kognitiver Arbeitsbelastung. Auf der Grundlage der bisherigen Forschungsarbeiten stellen wir Anwendungen vor, welche die Kognition im häuslichen und beruflichen Umfeld unterstützen. Die physiologischen Messungen stellen fest, wann eine kognitive Augmentierung sich als günstig erweist. In einer Feldstudie setzen wir ein Assistenzsystem ein, um die erhobenen Designanforderungen zur Reduktion kognitiver Arbeitslast zu validieren. Unsere Ergebnisse zeigen, dass die Arbeitsbelastung durch den Einsatz von Assistenzsystemen reduziert wird. Im Anschluss untersuchen wir, wie kognitive Arbeitsbelastung visualisiert werden kann. Wir stellen eine Implementierung einer Biofeedback-Visualisierung vor, die das Nutzerverständnis zum Verlauf und zur Entstehung von kognitiver Arbeitslast unterstützt. Eine abschließende Studie zeigt, wie Messungen kognitiver Arbeitslast zur Vorhersage der aktuellen Leseeffizienz benutzt werden können. Wir schließen hierbei mit einer Reihe von Applikationen ab, welche sich kognitive Arbeitslast als Eingabe zunutze machen. Die vorliegende wissenschaftliche Arbeit befasst sich mit dem Design von Assistenzsystemen, welche die kognitive Arbeitslast der Nutzer implizit erfasst und diese bei der Durchführung alltäglicher Aufgaben unterstützt. Dabei werden physiologische Daten erfasst, um Rückschlüsse in Realzeit auf die derzeitige kognitive Arbeitsbelastung zu erlauben. Anschließend werden diese Daten analysiert, um dem Nutzer strategisch zu assistieren. Das Ziel dieser Arbeit ist die Erweiterung neuartiger und bestehender kontextbewusster Benutzerschnittstellen um den Faktor kognitive Arbeitslast. Daher werden in dieser Arbeit arbeitslastbewusste Systeme und arbeitslastbewusste Benutzerschnittstellen als eine zusätzliche Dimension innerhalb des Paradigmas kontextbewusster Systeme präsentiert. Wir stellen acht Forschungsstudien vor, um die Designanforderungen und die Implementierung von kognitiv arbeitslastbewussten Systemen zu untersuchen. Schließlich stellen wir unsere Vision von zukünftigen kognitiven arbeitslastbewussten Systemen und Benutzerschnittstellen vor. Durch die knappe Verfügbarkeit öffentlich zugänglicher Datensätze, Referenzimplementierungen, und Methoden, waren Kontextbewusste Systeme in der Auswertung kognitiver Arbeitslast bezüglich der Nutzerinteraktion limitiert. Ergänzt durch die in dieser Arbeit gesammelten Datensätze erwarten wir, dass diese Arbeit den Weg für methodische und technische Werkzeuge ebnet, welche kognitive Arbeitslast als Faktor in das Kontextbewusstsein von Computersystemen integriert